Tell HN: Cloudflare and Scaleway have a peering issue

6 pointsby zhan_eghace 4 años6 comments

For close to ~35 hours, Scaleway have this status update [1] with no meaningful change. Issues is some requests coming from Cloudflare to origins in the Scaleway PAR region, are either timing out (520s, or taking very long to finish. Outgoing traffic looks affected too.

This affects even Scaleway own website/docs and control panel.

We have opened a ticket, to which the answer was - we can't give move information than in the ticket, the issues is on the Cloudflare side.

Scaleway have a Slack community [2] where more details are present, and many people are having the same issues.

We opened a ticket with Cloudflare where we are also a paying customer (a Pro one), again no answer for 15 hours.

Anyone having ideas, what can we do as a customer (except activating DR plans and moving altogether), to solve such an issue? I tried checking the Scaleway netmap to see who is really to blame [3], but I can't see a route from Cloudflare (I assume it is just under some IX).

The whole thing is even stranger, having in mind Scaleway and Cloudflare are partners, which I assume should make solving such issues faster. [4]

[1] https://status.scaleway.com/incidents/y19hcz28rkwp [2] https://scaleway-community.slack.com/archives/C7Z76CCUE/p1670226716954159 [3] https://netmap.scaleway.com [4] https://www.cloudflare.com/en-gb/partners/technology-partners/scaleway/

6 comments

zhan_eghace 4 años

Issue is solved. The context for anyone interested from one of Scaleway Engineers [1]:

> To give some context, there are a lot of possible path between Cloudflare and Scaleway : France IX in Paris, NL IX in Paris and in Amsterdam, AMS IX in Amsterdam, and the transit in both region (Cogent, Lumen, Arelion). Cloudflare is announcing some prefixes only on some of these links.

> We were trying to find the root cause since Monday morning, but as we did not have any errors visible on our side, it was pretty hard to identify the faulty path.

Another fun fact from Scaleway CTO is that Cloudflare started answering ... just ~30 minutes ago. [2]

[1] https://scaleway-community.slack.com/archives/C7Z76CCUE/p167...

[2] https://scaleway-community.slack.com/archives/C7Z76CCUE/p167...

zhan_eghace 4 años

Hopefully not because of this submission, some more updates started flowing on the Scaleway side [1], but nothing solving the issue.

The are some workarounds (from Marco@ [2])

> moving workload out of scaleway to other datacenters and keeping all cloudflare CDN and features working perfectly (seems to be affecting exclusively scaleway/online.net)

> disabling all cloudflare features (DNS only) (some people can’t do it as cloudflare could be integral part of webapps flow and servings)

But applying those can't be done in all cases (i.e. disables Cloudflare WAF, CDN, etc.).

[1] https://status.scaleway.com/incidents/y19hcz28rkwp

[2] https://scaleway-community.slack.com/archives/C7Z76CCUE/p167...

0xbkthace 4 años

I had to resolve a similar connectivity issue for a client a couple months ago. The server was in Canada, but for some reason, Cloudflare's Asia-Pacific PoPs didn't want to reach this server and returned 525 SSL Handshake Failed error instead. So, I went ahead to install Cloudflare Tunnel, because I know that instead of waiting for Cloudflare to move through the Internet up to the last mile to the origin, I could establish long-lived connections to them over multiple PoPs so that the traffic from entry PoP(s) would stay in Cloudflare's network perimeter. This is assuming that your Tunnel connections are healthy no matter what. And the problem was gone.

The path basically transitioned from:

Entry PoP <----> Origin

to:

Entry PoP <----> Tunnel PoP <----> Origin

zhan_eghace 4 años

Thanks for the suggestion, that sounds like a workaround too.

ajimixhace 4 años

We are also affected by this since Sunday. All our sites hosted in scaleway with Cloudflare are very slow to load (sometimes up to 30 seconds) and we get back 520 errors every now and then.

evczhace 4 años

Traffic to cloudflare seems to be going thru NL-IX for instance...

Even on zones where we have tiered caching and cloudflare report CDG as both main and backup pops