Debunking Cloudflare’s recent performance tests(fastly.com)
fastly.com
Debunking Cloudflare’s recent performance tests
https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests
312 comments
> Fastly has been trumpeting for years that Compute@Edge
This is the main problem with Compute@Edge, "for years".
They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.
After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime.
Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed.
Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
This is the main problem with Compute@Edge, "for years".
They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.
After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime.
Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed.
Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
> Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
I greatly respect Fastly and their technology, but they (and Varnish) are biased against small companies and make it impossible for me to sign up customers who would benefit from their services and one day might grow into big spending customers.
AWS has shown that the days of "min $5000/year" (as both these services have quoted me for individual products) can be over and you can be profitable. Companies who don't catch up with that are cutting off their future.
I greatly respect Fastly and their technology, but they (and Varnish) are biased against small companies and make it impossible for me to sign up customers who would benefit from their services and one day might grow into big spending customers.
AWS has shown that the days of "min $5000/year" (as both these services have quoted me for individual products) can be over and you can be profitable. Companies who don't catch up with that are cutting off their future.
> Companies who don't catch up with that are cutting off their future.
I wish someone would smack Okta over the head with this and get them to figure that part out.
Yes, they're expensive, but the per-use pricing was acceptable for what we wanted. I didn't realise there were (non-public) minimum spend amounts until we hit the end of our trial period and couldn't find any way to pay them other than to contact sales.
Sales wouldn't take the order for just a handful of users.
I wish someone would smack Okta over the head with this and get them to figure that part out.
Yes, they're expensive, but the per-use pricing was acceptable for what we wanted. I didn't realise there were (non-public) minimum spend amounts until we hit the end of our trial period and couldn't find any way to pay them other than to contact sales.
Sales wouldn't take the order for just a handful of users.
Okta has an extremely generous free plan...
Where? All I can find is https://www.okta.com/uk/free-trial/ and https://www.okta.com/pricing/
I didn't see anything about a free plan when we were doing this, and Sales only offered to extend our trial for another 30/60 days.
I totally agree with this, Fastly makes it pretty obvious from their website they do not want small customers. There is no price list and the prices on domains and such is 10x Cloudfare. It is also difficult to se what type of services they provide, everything ends up in a contact us. They seem to have great product with Compute Edge but no free thier and fuzzy pricing.
If you read the discussions in the financial forums Fastly might be in a tricky place since their biggest customer is assumed to be Amazon and if one goes to their website https://www.fastly.com/partners/featured/ they show off all the biggest cloud providers and customers. Seems like Fastly are afraid to steal customers from their partners.
If you read the discussions in the financial forums Fastly might be in a tricky place since their biggest customer is assumed to be Amazon and if one goes to their website https://www.fastly.com/partners/featured/ they show off all the biggest cloud providers and customers. Seems like Fastly are afraid to steal customers from their partners.
Actually the CDN pricing is in the pricing page and you can start a trial without contacting sales.
https://www.fastly.com/pricing/
https://www.fastly.com/pricing/
As someone who chose Fastly over Cloudflare for a small-medium business after a long, long deliberation I disagree.
Cloudflare's pricing is sneaky. They are good for small and personal projects with their free tier, but as soon as you need anything non trivial you get hit with their $xxxx/mo minimum commitment, "contact us for pricing" Enterprise plan. On Fastly you pay $50/mo (peanuts for a business) and that's it for basically anything you might want from a CDN, VCL is endlessly flexible and free. That $50 buys you access to a featureset that is arguably beyond Cloudflare's Enterprise plan. You will need quite a lot of traffic to go over $50, so overall it works out significantly cheaper for an SME that wants to use anything but a very basic CDN.
It's also important to remember that a lot of cases are perfectly served by VCL, I believe most businesses won't even need C@E. Sure, most people would need to learn a bit about it, but it's amazing to have it on your disposal for free compared to a usual rigid configuration.
Besides, something they don't advertise but is now the main selling point for me is support. Fastly was the only company I had pleasure dealing with where after shooting a technical question to their free tier support I got not just a technical answer, but a piece of code that solved the problem I had. In an hour. With a link to their Fastly Fiddle demonstrating how it works. When I encountered what looked like a bug, a few hours later their support person filed a PR to their TF provider fixing the docs. It was and is a breath of fresh air compared to my experience with any other SaaS, Cloudflare included.
I think the only thing that could've swayed me is Cloudflare's Chinese network. It's gated behind their Enterprise plan and they won't hold your hands much, but if you need it it's there.
Cloudflare's pricing is sneaky. They are good for small and personal projects with their free tier, but as soon as you need anything non trivial you get hit with their $xxxx/mo minimum commitment, "contact us for pricing" Enterprise plan. On Fastly you pay $50/mo (peanuts for a business) and that's it for basically anything you might want from a CDN, VCL is endlessly flexible and free. That $50 buys you access to a featureset that is arguably beyond Cloudflare's Enterprise plan. You will need quite a lot of traffic to go over $50, so overall it works out significantly cheaper for an SME that wants to use anything but a very basic CDN.
It's also important to remember that a lot of cases are perfectly served by VCL, I believe most businesses won't even need C@E. Sure, most people would need to learn a bit about it, but it's amazing to have it on your disposal for free compared to a usual rigid configuration.
Besides, something they don't advertise but is now the main selling point for me is support. Fastly was the only company I had pleasure dealing with where after shooting a technical question to their free tier support I got not just a technical answer, but a piece of code that solved the problem I had. In an hour. With a link to their Fastly Fiddle demonstrating how it works. When I encountered what looked like a bug, a few hours later their support person filed a PR to their TF provider fixing the docs. It was and is a breath of fresh air compared to my experience with any other SaaS, Cloudflare included.
I think the only thing that could've swayed me is Cloudflare's Chinese network. It's gated behind their Enterprise plan and they won't hold your hands much, but if you need it it's there.
> They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity.
This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on.
(I'll note that this is separate from the problem of AI-based fraud detection run amok.)
This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on.
(I'll note that this is separate from the problem of AI-based fraud detection run amok.)
I strongly believe that there will be new laws that adapt to the SaaS like business model.
Most SaaS are structured in an effectively "non competitive market" kind of way. Lots of information hiding (don't post prices), high costs of switching (vendor lockin), and so on. These type of things stifle competition and lead to a more pseudo Monopoly kind of state rather than a truly competitive market.
Anyway, transparent pricing should be required for all businesses IMO. The problem is particularly bad in healthcare, and big contributor into high costs.
Of course, it will take a few decades for laws to catch up
Most SaaS are structured in an effectively "non competitive market" kind of way. Lots of information hiding (don't post prices), high costs of switching (vendor lockin), and so on. These type of things stifle competition and lead to a more pseudo Monopoly kind of state rather than a truly competitive market.
Anyway, transparent pricing should be required for all businesses IMO. The problem is particularly bad in healthcare, and big contributor into high costs.
Of course, it will take a few decades for laws to catch up
Fastly is a public traded company.
In their annual reports Fastly have previously said C@E was available. They regularly released PR pieces of new functionality on C@E.
The reality was it was in private beta and deep in active development. Their official annual reports and PR pieces used by investors for investment decisions where misleading insinuating a product was available that was not. Fastly annual reports where painting a misleading picture. What's worse is the honest answer, we are deep in development and now have several large existing customers onboard for private beta is nothing to be ashamed of putting out to your share holders. Fastly where probably trying to hide the amount of time it's taken them to roll C@E out since first announced early 2019.
I'm sure every piece of public information went through their legal department and vetted but at the same time every piece of mandatory training I've done at publicly traded companies have me believe I could suffer huge legal consequences with the SEC for doing similar.
In their annual reports Fastly have previously said C@E was available. They regularly released PR pieces of new functionality on C@E.
The reality was it was in private beta and deep in active development. Their official annual reports and PR pieces used by investors for investment decisions where misleading insinuating a product was available that was not. Fastly annual reports where painting a misleading picture. What's worse is the honest answer, we are deep in development and now have several large existing customers onboard for private beta is nothing to be ashamed of putting out to your share holders. Fastly where probably trying to hide the amount of time it's taken them to roll C@E out since first announced early 2019.
I'm sure every piece of public information went through their legal department and vetted but at the same time every piece of mandatory training I've done at publicly traded companies have me believe I could suffer huge legal consequences with the SEC for doing similar.
It has been two weeks since Fastly's VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly's claims.
Are you going to remove the ToS clause or issue a correction on the blog post?
[1] https://twitter.com/lxt/status/1462896850055352320
Are you going to remove the ToS clause or issue a correction on the blog post?
[1] https://twitter.com/lxt/status/1462896850055352320
To be clear my comment here is not in any way, shape, or form, Cloudflare's response. I am here representing only myself.
Sure, but you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks. I think the community here calling out the double standards and asking for an update/response is entirely fair. If you didn't want the flak then leave off the "I'm the tech lead on CF workers" intro. Seems to me like the ball is in your court to at least try and make your advice actionable.
I do not believe I suggested doing anything against the ToS. I think you're misinterpreting the clause. But not being a lawyer, I don't really want to get into that discussion.
If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
> I think you're misinterpreting the clause. But not being a lawyer, I don't really want to get into that discussion.
The clause says "Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"[1]
IANAL, but this seems to very unambiguously prohibit benchmarking Cloudflare's services unless you have written permission. I know you don't want to get into an argument on HN, but could you like... bring it up to someone inside of CloudFlare who would be capable of changing it? You can point to this thread about how this clause is generating negative publicity.
[1] https://www.cloudflare.com/terms/ section 2.2
The clause says "Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"[1]
IANAL, but this seems to very unambiguously prohibit benchmarking Cloudflare's services unless you have written permission. I know you don't want to get into an argument on HN, but could you like... bring it up to someone inside of CloudFlare who would be capable of changing it? You can point to this thread about how this clause is generating negative publicity.
[1] https://www.cloudflare.com/terms/ section 2.2
I am a lawyer and also the CEO of Cloudflare. I have no idea why that clause is in our ToS. It was a surprise to me when it was pointed out recently. Not sure when or why it got included. Best guess is it was when some “stress tester” services decided to “benchmark” us by performing DDoS attacks and we thought we needed another justification to shut them off. Has been a loooong time since we worried about such things. Regardless, we decided weeks ago we’re removing the clause during the next ToS refresh. That’s scheduled for the coming weeks. And, in the meantime, have no issue with anyone benchmarking our performance. And seems we should do a more thorough and unimpeachable set of comparisons ourselves. Stay tuned.
I appreciate this reply, and hope the initial engineer gets no flak for his personal opinions attempting to defend your company. It's nice to see a tech company with employees defending it and leadership making such public statements as this.
I upvoted Kenton’s post. He’s the reason Workers exists. Surprised anyone worried about him commenting. I’d only be worried if I were a competitor who pissed him off by publishing BS stats. I’d imagine there’ll be an incredibly thorough and totally unimpeachable benchmarking study that comes out of this. And, anywhere we’re not the fastest, we soon will be. Game on.
> I’d only be worried if I were a competitor who pissed him off by publishing BS stats
An unusual and excellent CEO stance. Now I like Cloudflare even more.
An unusual and excellent CEO stance. Now I like Cloudflare even more.
Cloudlfare and other CDNs like it are a scourge on the Internet destroying privacy. At this point they have better tracking hooks than Google.
No JS required, just feed all web requests directly through them where they can see all first party cookies, encrypted contents, etc.
No JS required, just feed all web requests directly through them where they can see all first party cookies, encrypted contents, etc.
Why would you send a CDN any cookies?
[deleted]
Kenton is a star and they're lucky to have him. But I don't think he was defending the company, rather defending his own work.
They seem to be one in the same.
Yep. When you're the tech lead for a significant feature in a very significant company, indeed they are one and the same. Whether you want them to be or not.
Not quite at the level of a public figure (Pres. Biden can't go around making flippant comments) but more than being just a private citizen or "I just work here". No amount of disclaimer can remove that, and for better or worse it's part and parcel with the job.
Not quite at the level of a public figure (Pres. Biden can't go around making flippant comments) but more than being just a private citizen or "I just work here". No amount of disclaimer can remove that, and for better or worse it's part and parcel with the job.
It's not the same. The lead dev is not responsible for all marketing around his work. That responsibility lies with the author and any editors of the blog post. Ultimately the CEO can be held responsible for both dev and marketing and they already did own the TOS issue here.
At the same time can you change 2.2(a) and remove the "or sign up on behalf of a third party"? My clients are not technical, and when they do manage to sign up they then email me their login details in plain text...
Without this in your T&Cs I could create the account for them in a couple of minutes. And avoid doing a screenshare to walk those who fail through the sign up process.
Without this in your T&Cs I could create the account for them in a couple of minutes. And avoid doing a screenshare to walk those who fail through the sign up process.
If you're employed by your clients and do it in their name, doesn't that make you the first party? I'm no lawyer but I can imagine it's to legally be able to close off bots and other shady services.
> If you're employed by your clients and do it in their name, doesn't that make you the first party?
If you're an employee, yes. If you're a consultant, contractor, freelancer or similar then you are a third party doing as you do it on behalf of your client (the first party). This is for UK law, and the distinction of first/thrid party is important when it comes to tax (see IR35 for the mess created).
If you're an employee, yes. If you're a consultant, contractor, freelancer or similar then you are a third party doing as you do it on behalf of your client (the first party). This is for UK law, and the distinction of first/thrid party is important when it comes to tax (see IR35 for the mess created).
Perhaps you can have them provide an access key instead? I vaguely recall seeing a button on a 3rd party platform that let me configure my DNS in Cloudflare to route to the 3rd party. Not sure how that flow worked to be honest, but I believe there is some programmatic way to delegate.
Nice. It was a silly jab anyhow, seeing as you need to contact a rep at Fastly to even sign up for their product or get a quote on its cost.
Fastly made it sound like contacting Cloudflare is "impossible", yet here you and one of your top devs are.
Fastly made it sound like contacting Cloudflare is "impossible", yet here you and one of your top devs are.
I’m just reading, but I appreciate coming to the comment section of a post here and seeing I’m this interaction, the CEO of a company like CloudFlare posting and, to top it off, posting something like this.
> Regardless, we decided weeks ago we’re removing the clause during the next ToS refresh
I've seen you reply about ToS issues before (specifically over caching of non-html assets): https://news.ycombinator.com/item?id=20791605
You verbally allowed it in that thread but have you considered officially adding that into the next revision of your ToS too?
I've seen you reply about ToS issues before (specifically over caching of non-html assets): https://news.ycombinator.com/item?id=20791605
You verbally allowed it in that thread but have you considered officially adding that into the next revision of your ToS too?
So you haven't read the agreement which you ask all the customers to agree to? Lame.
Good move.
As for "stand-up" with Fastly, I believe the whole situation brings only negative consequences on both parties. I always become wary towards any service that posts comparisons with its competitors (or simply with services of similar nature).
Good luck and wise decisions to you.
As for "stand-up" with Fastly, I believe the whole situation brings only negative consequences on both parties. I always become wary towards any service that posts comparisons with its competitors (or simply with services of similar nature).
Good luck and wise decisions to you.
Love these direct founder responses. Makes me trust the service more than any amount of benchmarks
> Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent
This - I mean even the conversation prior to this message - surely constitutes "expressly permitted in writing by Cloudflare"?
This - I mean even the conversation prior to this message - surely constitutes "expressly permitted in writing by Cloudflare"?
I would not be surprised if the person on the other end of that conversation would take poorly to the existing contributions made to it by people who have publicly identified themselves as Cloudflare employees.
First off, just want to say thanks for your posts, I found they give useful context and I really appreciate them.
I don't want the following to come off as unnecessarily argumentative, but regarding the ToS, I'm not a lawyer either, but my "ability to read English" interpretation of the section on "perform or publish benchmarks..." certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn't publish it.
Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it's fair to call out CloudFlare for this.
I don't want the following to come off as unnecessarily argumentative, but regarding the ToS, I'm not a lawyer either, but my "ability to read English" interpretation of the section on "perform or publish benchmarks..." certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn't publish it.
Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it's fair to call out CloudFlare for this.
Note that the CEO had replied too this parent thread and said he is removing that language
Plus it was there to prevent DoS attacks which is understandable.
Even without the ToS language, if you're really going to stress test a service, it's probably a good idea to give them a heads up, lest you get marked a bad actor.
Even without the ToS language, if you're really going to stress test a service, it's probably a good idea to give them a heads up, lest you get marked a bad actor.
He still should take some responsibility for it being there in the first place. "Our legal department insists on adding user-hostile clauses everywhere they get the chance" is an OK excuse for a Cloudflare sales rep or engineer, but it's disingenuous coming from the guy who is in a position to tell them to take a friendlier approach by default.
I'm assuming this isn't the only overly restrictive clause in the contract. Maybe it's an anomaly in an otherwise respectful ToS.
I'm assuming this isn't the only overly restrictive clause in the contract. Maybe it's an anomaly in an otherwise respectful ToS.
He’s the CEO. He acknowledged it and is fixing it. There is no more responsibility left unclaimed.
We aren’t owed a historical explanation, and yet we’ll likely receive one with what I presume will be a TOS update blog post in a few weeks.
I feel like this is that moment where someone lays on the car horn because they want to be sure the other driver understands that they’re a bad person, and should feel bad about themselves. It’s not about making you right by their actions, it’s about making sure they know the depth of your anger at them.
That has little value here. It’s socially valuable in interpersonal interactions, but it’s a tire fire when left uncurbed at Internet scale, and becomes vitriolic and harmful to discourse.
I may have misunderstood your specific intentions and desires from the CEO, and if so, I apologize; but I stand by my point in the general sense for all of us.
We aren’t owed a historical explanation, and yet we’ll likely receive one with what I presume will be a TOS update blog post in a few weeks.
I feel like this is that moment where someone lays on the car horn because they want to be sure the other driver understands that they’re a bad person, and should feel bad about themselves. It’s not about making you right by their actions, it’s about making sure they know the depth of your anger at them.
That has little value here. It’s socially valuable in interpersonal interactions, but it’s a tire fire when left uncurbed at Internet scale, and becomes vitriolic and harmful to discourse.
I may have misunderstood your specific intentions and desires from the CEO, and if so, I apologize; but I stand by my point in the general sense for all of us.
You suggest performing one's own tests to evaluate performance of the services, which fits the definition of benchmarking.
The ToS explicitly say that performing any benchmark analyses or test relating to the services is not allowed without written permission from CloudFlare.
99% of the people reading and attempting to abide by the CloudFlare ToS are not lawyers, but as a rule contracts mean what they say when they say it clearly and unambiguously as this seems to.
The ToS explicitly say that performing any benchmark analyses or test relating to the services is not allowed without written permission from CloudFlare.
99% of the people reading and attempting to abide by the CloudFlare ToS are not lawyers, but as a rule contracts mean what they say when they say it clearly and unambiguously as this seems to.
> If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
The way I usually see people handle this, and what I do myself, is to both state your relation to the company and clarify whether you're speaking for the company. Ex: "I'm the tech lead for Cloudflare Workers, but speaking in my personal capacity..." or "I'm the tech lead for Cloudflare Workers (speaking only for myself) ..."
The way I usually see people handle this, and what I do myself, is to both state your relation to the company and clarify whether you're speaking for the company. Ex: "I'm the tech lead for Cloudflare Workers, but speaking in my personal capacity..." or "I'm the tech lead for Cloudflare Workers (speaking only for myself) ..."
It's generally understood on HN that when someone says "I work at X" that they are not speaking on behalf of the company.
On top of that, Kenton is a frequent commenter here, and I've never gotten the "air of superiority" vibe from him where such verbosity would be necessary.
On top of that, Kenton is a frequent commenter here, and I've never gotten the "air of superiority" vibe from him where such verbosity would be necessary.
This is true, but it's better to just assume people aren't speaking in an official capacity unless they say they are. This would also shave 30% off most Twitter bios.
> If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
I think the nuance is that you are presenting yourself as someone with responsibility/authority/control over the subject of these benchmarks. As a comparison, consider wording that skirts taking up that mantle:
Full disclosure, I work for Cloudflare, but ...
Not trying to be argumentative and really don't have any hostile feelings or intent. I understand where you're coming from.. just providing my outside take on how the interaction appears. You're within your right to defend your product. Nobody seems to have a problem with that. But you also decided to throw mud on the pile, metaphorically. You admitted you aren't plugged into the back and forth, that's fine. But this isn't news for Fastly and it's hard to take your side in this discussion when your solution is "go do your own tests" which is exactly what Fastly is would like to do. They clearly call attention to your ToS preventing them from doing that in the piece we're discussing here.
I think the nuance is that you are presenting yourself as someone with responsibility/authority/control over the subject of these benchmarks. As a comparison, consider wording that skirts taking up that mantle:
Full disclosure, I work for Cloudflare, but ...
Not trying to be argumentative and really don't have any hostile feelings or intent. I understand where you're coming from.. just providing my outside take on how the interaction appears. You're within your right to defend your product. Nobody seems to have a problem with that. But you also decided to throw mud on the pile, metaphorically. You admitted you aren't plugged into the back and forth, that's fine. But this isn't news for Fastly and it's hard to take your side in this discussion when your solution is "go do your own tests" which is exactly what Fastly is would like to do. They clearly call attention to your ToS preventing them from doing that in the piece we're discussing here.
I think a lot of us have run into this exact clause because of Oracle and then other database vendors. We are well aware of what happens when you violate one of these clauses.
i think it's good there was full disclosure about your employer, but it also set the post up for me as "i am wearing my cloudflare tech lead hat", as nowhere in your post you state you are speaking for yourself and not as an employee... very confusing.
> you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks.
He did not. Profiling how your particular app and use case runs on a given serverless provider is not benchmarking.
He did not. Profiling how your particular app and use case runs on a given serverless provider is not benchmarking.
FWIW that semantic difference is meaningless to me. A senior member of CF just commented in detail about CF and a competitor.
Sounds like you have it right.
It seems weird to want to claim authority (eg. that you are the tech lead) but not be willing to also be accountable for your statements.
In general, the situation is the exact opposite: companies want to claim authority, but when an employee makes a statement they don’t like they want to have the deniability of “oh they weren’t representing us, if you want our real opinion please talk to our spokesperson”. Corporate PR is a strange mix of wanting engagement but also being incredibly risk-averse, and it’s very different from how people typically communicate.
I propose changing these disclaimers about "not representing the company" yada yada, to ICOG, which stands for I am a cog in the machine.
What statements do you feel he should be held accountable for that he is not? Please quote.
I’m not sure how much authority being tech lead confers? It’s the lowest possible line management position. I wouldn’t expect a tech lead to have any influence whatsoever on legal, contractual or communication issues
If you know a little about how corporations work, than a tech lead is not responsible for a ToS, probably doesn't know anything about it and that's fine. Since it's not expected either.
It's the legal department... And the CEO already mentioned that they are removing it and he gave a valid response/reason.
It seems that they will actually benchmark Fastly in detail now ( could be after another improvement week), which probably isn't what Fastly wanted.
Something definitely seems to be happening if you read their response and i'm awaiting it with actual stats!
https://news.ycombinator.com/item?id=29468771
@dwwoelfel that's what you wanted? :)
It's the legal department... And the CEO already mentioned that they are removing it and he gave a valid response/reason.
It seems that they will actually benchmark Fastly in detail now ( could be after another improvement week), which probably isn't what Fastly wanted.
Something definitely seems to be happening if you read their response and i'm awaiting it with actual stats!
https://news.ycombinator.com/item?id=29468771
@dwwoelfel that's what you wanted? :)
My experience working at tech companies is that the tech lead, or anybody at the company, can post in an internal message board or slack to ask "what's up with this weird clause in our ToS" and expect an explanation.
It's nice that eastdakota responded here, but he had two weeks since the original tweet thread from Fastly's VP of Eng calling out the problems with their benchmarking. They didn't respond or retract the blog post in those two weeks.
As a cloudflare shareholder (and a fastly shareholder), I want Cloudflare to act ethically and either retract the blog post or issue a correction.
It's nice that eastdakota responded here, but he had two weeks since the original tweet thread from Fastly's VP of Eng calling out the problems with their benchmarking. They didn't respond or retract the blog post in those two weeks.
As a cloudflare shareholder (and a fastly shareholder), I want Cloudflare to act ethically and either retract the blog post or issue a correction.
And what i read from his comments is that a follow-up post will come.
You are insinuating bad will/faith and that's not the impression i observed.
You are insinuating bad will/faith and that's not the impression i observed.
Cloudflare's blog post still says, in bold, "Cloudflare Workers is 196% faster than Fastly’s Compute@Edge based on the time to first byte from the tests we ran on 50 nodes using Catchpoint’s data from across the world".
It is unethical to leave that up after Fastly pointed out core issues with the benchmarking, like using a free tier that was rate-limited.
It is unethical to leave that up after Fastly pointed out core issues with the benchmarking, like using a free tier that was rate-limited.
Cloudflare's test compared the free tier of both services. The post was explicit about this. Workers free tier has limits too, and we would certainly have preferred to use the paid version of Workers in our test, but as the paid version of C@E is only available with an enterprise contract, the only fair test we could run was between free tiers.
Incidentally, this means Fastly's blog post is currently displaying test results that compare the enterprise version of Compute@Edge against the free version of Workers. Granted, our bad for the ToS clause, but still.
Despite the strong language in their post, Fastly has not actually demonstrated that anything was intentionally biased or unfair in Cloudflare's test. They've only laid out their opinions as to what would make a more representative benchmark. That's a debate you can have about any benchmark, but that doesn't somehow make the original benchmark "unethical".
Incidentally, this means Fastly's blog post is currently displaying test results that compare the enterprise version of Compute@Edge against the free version of Workers. Granted, our bad for the ToS clause, but still.
Despite the strong language in their post, Fastly has not actually demonstrated that anything was intentionally biased or unfair in Cloudflare's test. They've only laid out their opinions as to what would make a more representative benchmark. That's a debate you can have about any benchmark, but that doesn't somehow make the original benchmark "unethical".
It's not the benchmark that's unethical. The unethical part is leaving up the original claim without adding a correction or a note that addresses the problems Fastly found with the benchmark.
But it still compares both free tiers as he mentioned. It seems dubious that you demand to compare Fastly's paying tier with Cloudflare's free one ¯\_(ツ)_/¯
Cloudflare's free tier does not optimize for speed/performance, but for available ( = unused ) datacenter capacity based on location. Which makes it less fast than their paying tier.
Additionally, there will be an update soon as mentioned before, based on past comments.
There are other things that you ignore/are unaware of that are not even mentioned in Fastly's post. Eg. That cloudflare also optimizes their network for routing to denser cities instead of rural areas. That metric is not even mentioned by Fastly...
Note: i don't know any inner workings of them as I don't work there. It's based on what I remember from their blog about their SDN and performance weeks. I suppose it's applicable to this scenario, if i got the details right.
Cloudflare's free tier does not optimize for speed/performance, but for available ( = unused ) datacenter capacity based on location. Which makes it less fast than their paying tier.
Additionally, there will be an update soon as mentioned before, based on past comments.
There are other things that you ignore/are unaware of that are not even mentioned in Fastly's post. Eg. That cloudflare also optimizes their network for routing to denser cities instead of rural areas. That metric is not even mentioned by Fastly...
Note: i don't know any inner workings of them as I don't work there. It's based on what I remember from their blog about their SDN and performance weeks. I suppose it's applicable to this scenario, if i got the details right.
I wouldn't expect everyone at Cloudflare to be intimately following every competitor's blog posts tbh
But I'd bet kentonv was following this one, since he said "until Fatly complained about it" and not "until I read the blog post".
[deleted]
Personal Opinion only.
I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS.
Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different.
Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare.
But I agree everyone should run their own benchmarks.
I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS.
Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different.
Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare.
But I agree everyone should run their own benchmarks.
The material commonly mentions "isolates" or V8 taking 5ms to cold start, which seems clearly aimed at Workers.
But the same point applies when comparing against Lambda. It's not a cold start if the app code is already resident in memory in advance. Workers and Lambda also proactively load some apps in advance, and we don't call it a "cold start" in those cases.
But the same point applies when comparing against Lambda. It's not a cold start if the app code is already resident in memory in advance. Workers and Lambda also proactively load some apps in advance, and we don't call it a "cold start" in those cases.
[deleted]
>> So, I would encourage everyone to run your own test and don't just go on other people's numbers.
> But I agree everyone should run their own benchmarks.
But they literally can't because the ToS forbids it.
> But I agree everyone should run their own benchmarks.
But they literally can't because the ToS forbids it.
They don't forbid it. They request you get permission first. No one seems to have bothered asking to find out if CF say no.
Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future.
This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.
Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future.
This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.
It would be nice for the first person to actually own it and not just say "I have no idea who put it into our TOS". Besides, it's either disingenuous or something is seriously broken in Cloudflare if their CEO is more attentive to HN comments than their own SMM people, as Fastly called that clause out on Twitter weeks ago.
Anti benchmark clauses usually prohibit publication. This wouldn’t exclude running two systems with your workload and documenting the results for internal use.
Except the CloudFlare ToS (posted as an image from the tweet thread above) says "perform or publish benchmark tests or analyses..."
[deleted]
> So, I would encourage everyone to run your own test and don't believe what either side posts.
I don't have the full TOS in front of me but wouldn't this violate it?
I don't have the full TOS in front of me but wouldn't this violate it?
I was about to disagree, but I just re-read Cloudflare's language, and you're right. Cloudflare forbids performing any tests or analyses, whether or not you publish the results. Even running something like `curl http://my-workers-site | time`, for your own personal curiosity, is in violation of their ToS. That's even more egregious than Oracle DB's infamous DeWitt clause.
Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)
Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)
Interesting, AWS terms of service seem more reasonable, allowing you to perform benchmarks at will, but requiring that you disclose to them the methodology you used to perform the benchmark. Seems like a more reasonable clause while still preserving the ability to respond to hostile or inaccurate benchmarks.
1.8 in https://aws.amazon.com/service-terms/
Here's some background from 2008 on benchmarking clauses and their precariousness: https://corporate.findlaw.com/business-operations/n-y-case-c...
1.8 in https://aws.amazon.com/service-terms/
Here's some background from 2008 on benchmarking clauses and their precariousness: https://corporate.findlaw.com/business-operations/n-y-case-c...
I think the way AWS does it are great. I understand why some companies doesn't like benchmarks, mainly because it is subject to marketing spin and picking specifics without context. But giving out exact methodology so others could repeat the test seems fair. Basically the whole benchmarks should be open source if you do intend to publish it.
I any civilised jurisdiction such ToS are unenforceable.
Why do firms continue to make unenforceable claims? (My least favourite are restriction of trade clauses in employment contracts that are simply illegal, unless paid for, where I come from. But they appear all the time in contracts)
Why do firms continue to make unenforceable claims? (My least favourite are restriction of trade clauses in employment contracts that are simply illegal, unless paid for, where I come from. But they appear all the time in contracts)
I just skimmed the ToS, the only remedy they offer to a violation of the terms is a denial of access to their service. Which is something they definitely can enforce.
Because it requires lawyers to prove that the claims are unenforceable, and most smaller companies aren't going to have the pockets needed to sustain the legal back-and-forth, which forces them to settle even if the big company is in the wrong.
It's legal 'chicken'.
It's legal 'chicken'.
threat of lawyers and settling out of court is a powerful extralegal deterrent all on its own.
What do you mean unenforceable? In many jurisdictions it is a criminal offense to violate the terms of service of a web site. In Illinois the first violation is a misdemeanour punishable by a year in the county jail. Further violations carry 2-5 years in prison.
A ToS is limited in what it can legally contain by law.
The EU e.g. has 93/13/EEC aka "Unfair contract terms directive" which bans a lot of shenanigans in ToS (e.g. mandatory location for law disputes). Or just writing in your ToS that "you agree that contract parties are not bound by the EU GDPR" for another example, like some companies do write, doesn't mean this is a enforceable clause. Another example, Germany recently made a law that mandates automatic subscription renewals - such as mobile phone contracts, gym memberships or online services - can be cancelled every month (there already was a law before that limited the initial subscription contract to a duration of max 2 years).
If there are some laws specifically that disallow general "benchmarking" I don't know, but I wouldn't be surprised if at least in the EU such a clause would be unenforcable. That sounds like a clear unfair one-sided advantage (the customer cannot check the services actually provided match what was promised). Publication of such results is another matter.
Also, I am curious about what you said about Illinois? Sounds bonkers. I mean I could imagine criminal prosecution for a set of defined, deemed specially bad violations that constitute crimes, like "hacking" and "sabotage", but not for things like e.g. "failure to pay membership dues in time" and other civil matters. Then again, I remember that case of some guy in Florida who couldn't keep his lawn nice and green, and didn't have the money to replace the lawn again and again, so the HOA took him to court, he was ordered to replace the lawn (which he still couldn't afford) and ended up in prison for a couple of days for "contempt of the court" until some friends and neighbors replaced the lawn.
The EU e.g. has 93/13/EEC aka "Unfair contract terms directive" which bans a lot of shenanigans in ToS (e.g. mandatory location for law disputes). Or just writing in your ToS that "you agree that contract parties are not bound by the EU GDPR" for another example, like some companies do write, doesn't mean this is a enforceable clause. Another example, Germany recently made a law that mandates automatic subscription renewals - such as mobile phone contracts, gym memberships or online services - can be cancelled every month (there already was a law before that limited the initial subscription contract to a duration of max 2 years).
If there are some laws specifically that disallow general "benchmarking" I don't know, but I wouldn't be surprised if at least in the EU such a clause would be unenforcable. That sounds like a clear unfair one-sided advantage (the customer cannot check the services actually provided match what was promised). Publication of such results is another matter.
Also, I am curious about what you said about Illinois? Sounds bonkers. I mean I could imagine criminal prosecution for a set of defined, deemed specially bad violations that constitute crimes, like "hacking" and "sabotage", but not for things like e.g. "failure to pay membership dues in time" and other civil matters. Then again, I remember that case of some guy in Florida who couldn't keep his lawn nice and green, and didn't have the money to replace the lawn again and again, so the HOA took him to court, he was ordered to replace the lawn (which he still couldn't afford) and ended up in prison for a couple of days for "contempt of the court" until some friends and neighbors replaced the lawn.
> it is a criminal offense to violate the terms of service of a web site
Sorry, what? Where? How's that even possible?
Sorry, what? Where? How's that even possible?
Among other places, in the United States. The CFAA [0] prohibits "access[ing] a computer without authorization or exceed[ing] authorized access". It's interpreted extremely broadly by the courts.
[0] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
[0] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
Here in Illinois they have a specific law which makes violating ToS a criminal offense punishable by up to 5 years in prison.
https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
This means I could set up watchcutecatpictures.org with some fine print that prohibits watching more than 10 cats a day and call the cops on random people.
What legal reasoning exists to give companies this kind of unchecked power?
What legal reasoning exists to give companies this kind of unchecked power?
Just to make certain put a button that says "I agree" next to a link to the T+Cs.
>According to the court, DeJohn's claim that he did not read those terms was >irrelevant because, absent fraud (which was not alleged), "failure to read a >contract is not a get out of jail free card."
https://www.wilmerhale.com/en/insights/publications/click-wr...
>According to the court, DeJohn's claim that he did not read those terms was >irrelevant because, absent fraud (which was not alleged), "failure to read a >contract is not a get out of jail free card."
https://www.wilmerhale.com/en/insights/publications/click-wr...
It doesn't need to be legally enforecable to be useful.
At the very least, it is an indication of where they stand. They are free to take any customers they want, and if they say they don't want customers with a certain use case, that's good enough for me.
At the very least, it is an indication of where they stand. They are free to take any customers they want, and if they say they don't want customers with a certain use case, that's good enough for me.
> so maybe we're all off the hook! :)
Or maybe its entrapment!
Same as when accepting a job, never trust someone who says a section of their contract/terms won't be enforced.
I might need to start looking for another provider...
Or maybe its entrapment!
Same as when accepting a job, never trust someone who says a section of their contract/terms won't be enforced.
I might need to start looking for another provider...
[deleted]
> I've been saying for decades that benchmarks are almost always meaningless
I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred.
I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.
I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred.
I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.
Oh absolutely, benchmarks are great for measuring your own performance and improving it.
I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
I absolutely agree with this sentiment.
Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks.
The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks.
The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
It is extremely helpful to be able to estimate performance without building out a whole system. For instance 10ms vs 100ms vs 1s latency makes a world of difference as far as what is possible and what component might be the bottleneck. Though, I agree that the “horse race” between several highly optimized and broadly similar libraries may be far less interesting
Unfortunately not all companies follow your great obvious recommendation and listen to snake oil salesman dressed up as consultants and industry analysts:
So, I would encourage everyone to run your own test and don't believe what either side posts.
---
Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.
So, I would encourage everyone to run your own test and don't believe what either side posts.
---
Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.
>> But as long as we're already mud slinging
Strange statement. Didn't Cloudflare start the mud slinging?
I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.
Strange statement. Didn't Cloudflare start the mud slinging?
I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.
Sounds like you might be in a position to get some of these things changed and pave the way for fair benchmarks to take place.
Sounds like there's a lack of communication if Cloudflare published a benchmark blog post without the tech lead even knowing (better yet, finding out about the competitor's post first!).
I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post.
Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not.
Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.
I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post.
Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not.
Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.
I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.
Thanks for posting this. I think a lot of people are hyperfocusing on some of your wording, but this post was very informative.
Agree with the other comment. Unless you have explicit permission I'd be careful what you post here about your employer
Did you notice his username? I’m pretty sure he has permission from himself lol.
Believe it or not, I don't have any clue who he is. And the post was edited btw.
I made a very small edit to one sentence to change "don't believe either side's benchmarks" (or something like that) to "don't just go on other people's numbers", since the latter better reflects what I meant -- not that the numbers are false or intentionally misleading, but that they probably aren't reflective of your particular use case.
Every benchmark should start with:
For us/me XX works better because of...
For us/me XX works better because of...
Thank you for your hard work on workers and for your openness in these forums.
As a dev, it sounds to me like Fastly's whole purpose is wrapped up in attacking Cloudflare whereas Cloudflare is focused on innovating.
As a dev, it sounds to me like Fastly's whole purpose is wrapped up in attacking Cloudflare whereas Cloudflare is focused on innovating.
I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all.
This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.*
A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened.
* intentionally vague here, in order to give you space to delete and leave no record
This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.*
A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened.
* intentionally vague here, in order to give you space to delete and leave no record
> their post doesn't come off like mud-slinging, at all.
The post's current title is Lies, damned lies... and the article's synopsis is,
> This nonsensical conclusion provides a great example of how statistics can be used to mislead. Read on for an analysis of Cloudflare’s testing methodology, and the results of a more scientific and useful comparison.
That is definitely mud-slinging.
The post's current title is Lies, damned lies... and the article's synopsis is,
> This nonsensical conclusion provides a great example of how statistics can be used to mislead. Read on for an analysis of Cloudflare’s testing methodology, and the results of a more scientific and useful comparison.
That is definitely mud-slinging.
No it isn't. It's their response to CF's crap test which CF then posted on their blog <---THAT is mud slinging.
And Kenton himself agrees: "On a serious note, I've been saying for decades that benchmarks are almost always meaningless..."
Benchmarks can be used to mislead which CF was clearly doing and which Fastly pointed out with their "Lies, damned lies..."
And Kenton himself agrees: "On a serious note, I've been saying for decades that benchmarks are almost always meaningless..."
Benchmarks can be used to mislead which CF was clearly doing and which Fastly pointed out with their "Lies, damned lies..."
I think you misinterpreted me a bit. I don't believe CF's benchmark was designed to be misleading or biased. I think the benchmark had a perfectly reasonable setup and was actually representative of one relevant use case -- the case of an app that just returns a response without a lot of processing. I think Fastly's criticisms of the benchmark are surprisingly weak compared to the strong language they are couched in, and the strong language itself is what I refer to as "mud-slinging".
My point about benchmarks in general was that unless your application is extremely similar to the one tested -- which is unlikely -- then these numbers don't really tell you much about how your own app will perform. This is a criticism I have of every benchmark that attempts to compare performance between very different technologies. I don't think it makes such benchmarks unethical, just not very useful.
My point about benchmarks in general was that unless your application is extremely similar to the one tested -- which is unlikely -- then these numbers don't really tell you much about how your own app will perform. This is a criticism I have of every benchmark that attempts to compare performance between very different technologies. I don't think it makes such benchmarks unethical, just not very useful.
Sorry and thanks for the clarification.
I don't understand why you're arguing for the parent comment to be deleted, it's useful to hear from someone, (especially an engineer and the tech lead) who is working on the product. I hope that the comment is not deleted.
I found parent interesting and relevant.
But mostly importantly, kentonv's last paragraph.
At the end of the day, benchmarking for your use case is the only true north. Everything else, on all sides, is marketing fluff.
But mostly importantly, kentonv's last paragraph.
At the end of the day, benchmarking for your use case is the only true north. Everything else, on all sides, is marketing fluff.
refulgentis(5)
You're implying that he did something wrong.
But the title of the post validates his opinion and his claim about fastly and cold starts can literally be found anywhere in their blog and the compute@edge landing page.
I think it's a valid and valuable comment.
But the title of the post validates his opinion and his claim about fastly and cold starts can literally be found anywhere in their blog and the compute@edge landing page.
I think it's a valid and valuable comment.
Fwiw, I don’t have an issue with it. It’s good information.
I think the comment you were replying to was just fine. They have opinions about another service, I think that’s ok.
> their post doesn't come off like mud-slinging, at all.
Sure felt like mud slinging to me. Just the whole list of complaints they had at the start of the post was hard to get through.
They could just tell us what they did differently and show the result.
Sure felt like mud slinging to me. Just the whole list of complaints they had at the start of the post was hard to get through.
They could just tell us what they did differently and show the result.
[deleted]
[deleted]
I suspect you're mixing up microseconds with milliseconds.
5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...
5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...
No, I didn't mix them up.
> In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.
Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor.
You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product.
Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.
Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor.
You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product.
Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.
My manager's manager is the CEO. Yes, we all discussed it and everyone agreed the clause should be removed, but at the time of my comment I wasn't sure if I was authorized to announce that publicly. He has since posted in this thread.
If you bother to read the thread the CEO has already commented to say they're changing that ToS. Why give advice that no-one asked for and no-one needed? Isn't that pretty much the worst?
I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points:
* Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)
* Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing).
They do have two good points:
* Benchmarks between CDNs should be done from the same client locations (fine, but why did you then use your own set)
* Cloudflare should not prohibit benchmarking of their products (fair)
But this whole post read like a feeble attempt at misdirection, and made me distrust Fastly as a company. The complaints about their methodology aren't solved by using your own, even more flawed, methodology that doesn't even use the same language. You could've just emailed them and asked to work together on an apple-to-apples test instead of creating an even more flawed benchmark.
* Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)
* Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing).
They do have two good points:
* Benchmarks between CDNs should be done from the same client locations (fine, but why did you then use your own set)
* Cloudflare should not prohibit benchmarking of their products (fair)
But this whole post read like a feeble attempt at misdirection, and made me distrust Fastly as a company. The complaints about their methodology aren't solved by using your own, even more flawed, methodology that doesn't even use the same language. You could've just emailed them and asked to work together on an apple-to-apples test instead of creating an even more flawed benchmark.
> * Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)
First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript.
Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.
First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript.
Second, when you’re measuring milliseconds for response latencies, the difference between JS taking 10us CPU and Rust-wasm taking 1us is negligible. And system overheads far outweigh the actual computation time.
> First, the JavaScript is compiled to hot bytecode (WASM-equivalent) by the VM, so the performance difference is smaller than you would expect than the normal performance difference between Rust and JavaScript.
Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs
Can you explain this? JS intermediate is wildly different from WASM, not to mention WASM requires a max memory album and a lot of other things resulting from the different VMs
Assuming that we're using V8 and the function is hot, both the JS code and the WASM will be compiled to optimized bytecode by V8's TurboFan optimizing compiler. The two big reasons why JS is inefficient vs Rust+WASM is because (1) everything is untyped -- without being able to statically resolve properties, the compiler is forced to resort to string lookups and (2) garbage collection. Compiling Rust to WASM will emit code that uses Rust's type system and lifetime analysis to create direct accesses and deallocate objects when necessary.
For such a small program (write the current time to a socket), it's trivial for TurboFan to examine the types of the JS code at runtime [1]. Also, escape analysis should also be trivial [2]. So, at runtime, the optimizing compiler can (1) replace string lookups with very fast address dereferences, and (2) insert appropriate deallocations. My guess is that with these two optimizations the performance difference can be reduced significantly. (Of course there is still some overhead -- deoptimization checks, etc. -- but it's not nearly as big as between a 5 MB JS file vs the same amount of code in Rust.)
[1] E.g. https://v8.dev/blog/fast-properties
[2] https://www.jfokus.se/jfokus18/preso/Escape-Analysis-in-V8.p...
For such a small program (write the current time to a socket), it's trivial for TurboFan to examine the types of the JS code at runtime [1]. Also, escape analysis should also be trivial [2]. So, at runtime, the optimizing compiler can (1) replace string lookups with very fast address dereferences, and (2) insert appropriate deallocations. My guess is that with these two optimizations the performance difference can be reduced significantly. (Of course there is still some overhead -- deoptimization checks, etc. -- but it's not nearly as big as between a 5 MB JS file vs the same amount of code in Rust.)
[1] E.g. https://v8.dev/blog/fast-properties
[2] https://www.jfokus.se/jfokus18/preso/Escape-Analysis-in-V8.p...
> For such a small program ... the performance difference can be reduced significantly.
I think you're wrong. Small programs don't necessarily minimize the difference between languages. Small programs often exacerbate them. Often for weird, idiosyncratic reasons. Especially if you're measuring time-to-first-byte, and not letting the VM warm up.
For example, in this case I wouldn't be surprised if most of the CPU time was going into node + v8's startup, rather than executing the user's javascript at all.
Look at the plaintext techempower benchmarks[1]. These benchmarks test tiny amounts of code - the programs only have to respond with "hello world" HTTP responses. The best performers are hitting hardware limits. If your theory that small program = small relative cost of using javascript was true, nodejs's performance should be similar to rust / C++. It is not - node is only 13% of the performance of the top performers. Weirdly "justjs" (another v8 based runtime) is ~5x faster than nodejs on this test. The reason probably has nothing to do with javascript at all, and is because justjs has less overhead and talks to the OS in a more efficient way. (It probably uses io_ring.)
But maybe this is evidence you're technically correct. The performance differences can be reduced. But we have every reason to assume nodejs will have way more overhead than rust at executing the same code.
In all, I agree with other posters. A benchmark showing rust on fastly is faster than javascript on cloudflare tells us nothing about the underlying hosting providers.
https://www.techempower.com/benchmarks/#section=data-r20&hw=...
I think you're wrong. Small programs don't necessarily minimize the difference between languages. Small programs often exacerbate them. Often for weird, idiosyncratic reasons. Especially if you're measuring time-to-first-byte, and not letting the VM warm up.
For example, in this case I wouldn't be surprised if most of the CPU time was going into node + v8's startup, rather than executing the user's javascript at all.
Look at the plaintext techempower benchmarks[1]. These benchmarks test tiny amounts of code - the programs only have to respond with "hello world" HTTP responses. The best performers are hitting hardware limits. If your theory that small program = small relative cost of using javascript was true, nodejs's performance should be similar to rust / C++. It is not - node is only 13% of the performance of the top performers. Weirdly "justjs" (another v8 based runtime) is ~5x faster than nodejs on this test. The reason probably has nothing to do with javascript at all, and is because justjs has less overhead and talks to the OS in a more efficient way. (It probably uses io_ring.)
But maybe this is evidence you're technically correct. The performance differences can be reduced. But we have every reason to assume nodejs will have way more overhead than rust at executing the same code.
In all, I agree with other posters. A benchmark showing rust on fastly is faster than javascript on cloudflare tells us nothing about the underlying hosting providers.
https://www.techempower.com/benchmarks/#section=data-r20&hw=...
Of course you're right in cases of a cold start. But if I understand the benchmark in the linked post correctly, the TTFB was measured assuming that the VM was already hot & and the code optimized through TurboFan already. (Hence "assuming the function is hot.")
Also, I don't see Rust + WASM on the linked benchmarks. WASM code still has to run in a VM and probably uses the Node.js runtime for syscalls. (Based on a quick search, it seems like most people compile to WASM and create Node bindings with wasm-bindgen.)
Third, my claim isn't that JS can reach Rust+WASM speeds; just that it's not as slow as you might expect because you do have type information.
Anyway, the whole point is moot because we're arguing over microsecond differences.
Also, I don't see Rust + WASM on the linked benchmarks. WASM code still has to run in a VM and probably uses the Node.js runtime for syscalls. (Based on a quick search, it seems like most people compile to WASM and create Node bindings with wasm-bindgen.)
Third, my claim isn't that JS can reach Rust+WASM speeds; just that it's not as slow as you might expect because you do have type information.
Anyway, the whole point is moot because we're arguing over microsecond differences.
In many cases those TechEmpower benchmarks show JavaScript beating both C++ and Rust frameworks. Are those tests anywhere close to being representative of real world use? V8 really is an incredible piece of engineering, but then so is LLVM. For context I don't do much if any web stuff.
I think whats going on is that the code involved really isn't spending much time in JS / C++ / whatever at all. The winners are probably libraries tuned to make good use of threading, io_uring and stuff like that.
> For such a small program (write the current time to a socket), it's trivial for TurboFan
On the contrary, for short programs the JavaScript engine may not choose to do anything at all beyond run it through the baseline interpreter.
On the contrary, for short programs the JavaScript engine may not choose to do anything at all beyond run it through the baseline interpreter.
>I was expecting a point-by-point comparison from Fastly. Instead, this comes off incredibly defensive and evasive, basically boiling down to two irrelevant points:
Because it is not meant as a comparison but they are the one defending a benchmarks ran by their competitors?
Because it is not meant as a comparison but they are the one defending a benchmarks ran by their competitors?
> * Fastly performance throttles their free accounts (ok, but not Cloudflare's fault, and I definitely don't want to get into your pricing game if you're going to start offering different tiers of serverless workers... part of Cloudflare's allure is its simplicity in pricing).
TBF, people who care about benchmarks usually expect high loads and value performance enough to put sufficient money on the table. I'd see the free tier people having different priorities.
Sure, it's still valuable info to know what's the free account limits are, but it shouldn't be headline news (except if it's exceptionally fast...)
TBF, people who care about benchmarks usually expect high loads and value performance enough to put sufficient money on the table. I'd see the free tier people having different priorities.
Sure, it's still valuable info to know what's the free account limits are, but it shouldn't be headline news (except if it's exceptionally fast...)
> Rust compiled to WASM is faster than Javascript (ok, but I'm not going to switch my entire stack just to use your CDN)
This was also addressed in the post: they don’t consider the JavaScript flavor production ready in part due to performance. Even if you’re an all JavaScript shop which will never consider something else, comparing a mature released service to one under development makes the comparison harder.
It would have been interesting and more honest to compare other language runtimes to see whether there’s a trend for the entire platform or simply one language runtime.
This was also addressed in the post: they don’t consider the JavaScript flavor production ready in part due to performance. Even if you’re an all JavaScript shop which will never consider something else, comparing a mature released service to one under development makes the comparison harder.
It would have been interesting and more honest to compare other language runtimes to see whether there’s a trend for the entire platform or simply one language runtime.
As mentioned in the article, Cloudflare expressly prohibits benchmarking their services in the tos.
I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.
I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.
Note the CloudFlare CEO subsequently commented elsewhere on this post that they would remove the problematic language from their ToS: https://news.ycombinator.com/item?id=29468771
I'm not a lawyer but could someone explain to me could you be legally persecuted for this? It's "Terms of Service" - I broke the terms by performing the benchmarks and Cloudflare no longer provides service to me. Lets say I'm a scientist and I'm fine with never using cloudflare services but this benchmark is important for my field - would cloudflare have a case here? I'd imagine that would be quite absurd but corporate law very much already is.
The feds ruined Aaron Swartz's life for violating "Terms of Service", so people are right to be fearful.
https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#A...
https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#A...
Yes. Here in Illinois violating terms of service carries up to 5 years in prison.
https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
Anti-benchmarking clauses are sometimes called DeWitt clauses after Oracle famously tried to get a professor fired for testing their offerings. https://danluu.com/anon-benchmark/
I never knew how true that anecdote was but that’s typically the boogeyman story that gets passed around for things like this
I never knew how true that anecdote was but that’s typically the boogeyman story that gets passed around for things like this
Depends on the country you are in, in the US, thinks seem bad. See the other posts.
In the EU there are various ways of ToS are limited in their power, if that applies here I'm not sure.
But they are basically forbidding you to evaluate how well a product you (might be) paying for works, depending on what they advertised they might even prevent you from evaluation if the advertised product characteristics uphold. Especially in the later case I guess that the ToS clause would be void.
In the EU there are various ways of ToS are limited in their power, if that applies here I'm not sure.
But they are basically forbidding you to evaluate how well a product you (might be) paying for works, depending on what they advertised they might even prevent you from evaluation if the advertised product characteristics uphold. Especially in the later case I guess that the ToS clause would be void.
They don't prohibit it outright. They just require you to ask first. The article made no mention of if they asked or not.
I feel like that's splitting hairs. if you say something is "prohibited without express written permission," to me that reads as "prohibited." Especially when it comes to benchmarking, because it effectively means that you can never do an impartial benchmark of the service.
[deleted]
to me that reads as "prohibited."
That's because you don't have a good enough reason to want to ask for permission though. If you were spending $xx thousands on edge workers every year you'd just ask, and you'd rule CF out if they said no. But, very likely, they'd say yes.
That's because you don't have a good enough reason to want to ask for permission though. If you were spending $xx thousands on edge workers every year you'd just ask, and you'd rule CF out if they said no. But, very likely, they'd say yes.
You have to ask for a written exemption that may or may not be granted.
With enough money you can discuss any ToS.
That's a fake opening with the same effect as prohibiting something by ToS.
That's a fake opening with the same effect as prohibiting something by ToS.
Asking before benchmarking seems to be a sure way to poison your results.
Who's to say that the performance of the services won't coincidentally perform better after you've given your request to test?
I am sure they might mention some legalese of "avoiding putting arduous strain on the services which may cause disruption to our other users" as a justification, but that PR speak won't fly with me since there is a separate section that already references that.
I say this as a big fan of Cloudflare and I am sure they don't need to use such underhanded tactics to make their services look good.
Who's to say that the performance of the services won't coincidentally perform better after you've given your request to test?
I am sure they might mention some legalese of "avoiding putting arduous strain on the services which may cause disruption to our other users" as a justification, but that PR speak won't fly with me since there is a separate section that already references that.
I say this as a big fan of Cloudflare and I am sure they don't need to use such underhanded tactics to make their services look good.
[deleted]
> They don't prohibit it outright. They just require you to ask first
Frankly, that's a distinction without a difference in this context, at least in my opinion.
Frankly, that's a distinction without a difference in this context, at least in my opinion.
"If you get permission to bypass this, you can bypass it" is implicitly part of all agreements. So explicitly including that doesn't change it. It's the same as an outright prohibition.
In contrast to the other comments, this is a valid concern regarding whether or not they asked first. I assume legal cleared this blog post, which would likely infer that legal either (A) got permission to publish benchmarks, or (B) doesn't think that part of the TOS is enforceable/doesn't expect CF to actually do anything about it.
The TOS clause, for reference: https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or... (ctrl-f benchmark )
The TOS clause, for reference: https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or... (ctrl-f benchmark )
To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark, comparable to the one that Cloudflare ran on our platform, and compared those results to the ones that Cloudflare published as part of the original blog post.
The Cloudflare terms prohibit benchmarking without their explicit permission.
The Cloudflare terms prohibit benchmarking without their explicit permission.
> To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark…
Am I the only having cognitive dissonance here? I was confused how they complained about not being able to perform benchmarks but then… ran a benchmark? There must be some distinction here that I’m missing
Am I the only having cognitive dissonance here? I was confused how they complained about not being able to perform benchmarks but then… ran a benchmark? There must be some distinction here that I’m missing
Fastly is unable to reproduce the benchmark against Cloudflare. They reproduced a modified (and theoretically improved) benchmark against their own service, but then had to compare it to the original values provided by Cloudflare.
CloudFlare ran a benchmark on both services and published their results. The contention is that the benchmark did not represent fastly performance fairly, so fastly reran the benchmark in what they consider to be a fair way, on their service only. They did not retest CloudFlare in their benchmark. The results in the post compare CloudFlare's results for CloudFlare and fastly's results for fastly
Yeah, ok that must be the distinction.
However, I see that the TOS says you may not "perform or publish" benchmarks. Seems this still violates the "or publish" part, even if Fastly avoided doing the performing. What would be the point of forbidding publishing if it is already forbidden to perform said benchmarks in the first place? The TOS seems to also forbid publishing on the benchmark even when someone else performs the Fastly part of the benchmark.
Edit: It must be the TOS doesn't apply if you don't use the service.
However, I see that the TOS says you may not "perform or publish" benchmarks. Seems this still violates the "or publish" part, even if Fastly avoided doing the performing. What would be the point of forbidding publishing if it is already forbidden to perform said benchmarks in the first place? The TOS seems to also forbid publishing on the benchmark even when someone else performs the Fastly part of the benchmark.
Edit: It must be the TOS doesn't apply if you don't use the service.
Fortunately, no one needed to agree to the CloudFlare ToS to read their blog post.
Haha. Actually, now that I think about it, I guess the TOS doesn't apply if you don't use the service!
Sorry, I was trying to clarify in my comment but failed to put it through clearly. When I say "We ran our own benchmark" I mean Fastly ran a benchmark on a Fastly Compute@Edge service.
Notably, the self-service and enterprise TOS differ a bit here.
From "Cloudflare Self-Serve Subscription Agreement" https://www.cloudflare.com/terms/ , "2.2 Restrictions. Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"
From "ENTERPRISE SUBSCRIPTION TERMS OF SERVICE" https://www.cloudflare.com/enterpriseterms/ , "2.3. Restrictions and Acceptable Use. Customer must not: [...] (i) perform or publish any performance or benchmark tests or analyses relating to the Service, other than solely for Customer’s internal use;"
Generally, it sounds like the kind of thing that would be obvious to a lawyer (we want to prohibit random people badmouth our product, right?) but stupid to engineers.
The best outcome would be if Fastly and Cloudflare both dropped any language limiting benchmarking and sharing of results.
If they want to protect their brands, the most common sense requirement would be "We require you to tell us, within X hours after publishing, of benchmarks you're publishing and send us a copy of the article in which you published them (or link if publicly accessible)."
From "Cloudflare Self-Serve Subscription Agreement" https://www.cloudflare.com/terms/ , "2.2 Restrictions. Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"
From "ENTERPRISE SUBSCRIPTION TERMS OF SERVICE" https://www.cloudflare.com/enterpriseterms/ , "2.3. Restrictions and Acceptable Use. Customer must not: [...] (i) perform or publish any performance or benchmark tests or analyses relating to the Service, other than solely for Customer’s internal use;"
Generally, it sounds like the kind of thing that would be obvious to a lawyer (we want to prohibit random people badmouth our product, right?) but stupid to engineers.
The best outcome would be if Fastly and Cloudflare both dropped any language limiting benchmarking and sharing of results.
If they want to protect their brands, the most common sense requirement would be "We require you to tell us, within X hours after publishing, of benchmarks you're publishing and send us a copy of the article in which you published them (or link if publicly accessible)."
They stated in the blogpost that they only performed the benchmarks against their own services
> So instead, we ran the same tests (echoing headers and measuring TTFB via Catchpoint) against our own platform
> So instead, we ran the same tests (echoing headers and measuring TTFB via Catchpoint) against our own platform
In a totally different scenario - if someone requires you to ask before undergoing a drug test, for example, if an Olympic athlete made drug testing conditional upon having notice, it wouldn't be a real test.
It's not quite the same. Cloudflare aren't going to rejig their global service every time someone wants to benchmark their code.
Every prohibited thing is not prohibited if you get an exception. Nothing is prohibited huzzah!
Why tho?
Probably for the same reason that I've seen processes put in place for performance testing a service (internally or on production) - because it puts a significant load on the service and automatic rate limiting, flagging, etc can be triggered.
if you benchmark your competitor, your competitor should have the ability to provide tuning so their numbers look as good as possible, before you run to the press.
Unless your competitor was offering a complimentary service to do that for all customers (tuning) then perhaps it shouldn't be done. I believe the idea of benchmarking is to get a sense of how the service would perform for a customer.
Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.
Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.
No, benchmarking aims to identify the peak performance of a service, not the average customer experience. Real Benchmarking is done for people who are serious about performance. marketing benchmarking is designed to attract naive customers.
As for the latter, that benchmark wouldn't reproduce and would require extensive configuration (which in a sense is just a part of negotiation between the customer and the client). If a company got caught faking benchmarks run by competitors and it could be shown publicly they did, that's pretty much the end of that company's public reputation.
As for the latter, that benchmark wouldn't reproduce and would require extensive configuration (which in a sense is just a part of negotiation between the customer and the client). If a company got caught faking benchmarks run by competitors and it could be shown publicly they did, that's pretty much the end of that company's public reputation.
Cloudflare not allowing benchmarks in their TOS is very sketchy, that puts them in the same tier as Oracle.
Cloudflare have pulled enough shady stuff now that they've fallen out of my favor. Their generous free product bought them a lot of community goodwill but their real face has been showing the past few years.
Cloudflare have pulled enough shady stuff now that they've fallen out of my favor. Their generous free product bought them a lot of community goodwill but their real face has been showing the past few years.
> Cloudflare not allowing benchmarks in their TOS is very sketchy, that puts them in the same tier as Oracle.
There's a non-sketchy way for a company to do that kind of thing which avoids putting them in Oracle territory. Simply require that anyone publishing benchmarks (1) publish complete configuration information sufficient to allow the company and other interested third parties to reproduce the benchmarks, and (2) allows benchmarking of their products that they are benchmarking against yours under similar terms.
It is nicer of course to not have any benchmarking restrictions, but if you do that you are putting yourself at a disadvantage to companies like Oracle. They can benchmark against you put you can't do the same against them.
Once one important player in a market goes down the Oracle route others tend to follow. But if they aren't asses they will follow with the kind of restriction with reciprocity I outlined above instead of an Oracle-like restriction.
There's a non-sketchy way for a company to do that kind of thing which avoids putting them in Oracle territory. Simply require that anyone publishing benchmarks (1) publish complete configuration information sufficient to allow the company and other interested third parties to reproduce the benchmarks, and (2) allows benchmarking of their products that they are benchmarking against yours under similar terms.
It is nicer of course to not have any benchmarking restrictions, but if you do that you are putting yourself at a disadvantage to companies like Oracle. They can benchmark against you put you can't do the same against them.
Once one important player in a market goes down the Oracle route others tend to follow. But if they aren't asses they will follow with the kind of restriction with reciprocity I outlined above instead of an Oracle-like restriction.
I like your ideas, but they seem difficult to enforce. It assumes good faith on all sides. One of the biggest complaints about AI/ML research results: It is frequently hard/impossible to replicate the results.
One idea: The edge competitors can create a public (SourceHut?) project that runs various daily tests against themselves. This would similar to JSON library benchmarks. [1] Then allow each competitors to continuously tweak there settings to accomplish the task in the shortest amount of time.
Also: It would be nice to see a cost analysis. For years, IBM's DB2 was insanely fast if you could afford to pay outrageous hardware, software license, and consulting costs. I'm not in the edge business, but I guess there are some operators where you can just pay a lot more and get better performance -- if you really need it.
[1] https://github.com/miloyip/nativejson-benchmark
One idea: The edge competitors can create a public (SourceHut?) project that runs various daily tests against themselves. This would similar to JSON library benchmarks. [1] Then allow each competitors to continuously tweak there settings to accomplish the task in the shortest amount of time.
Also: It would be nice to see a cost analysis. For years, IBM's DB2 was insanely fast if you could afford to pay outrageous hardware, software license, and consulting costs. I'm not in the edge business, but I guess there are some operators where you can just pay a lot more and get better performance -- if you really need it.
[1] https://github.com/miloyip/nativejson-benchmark
> One of the biggest complaints about *CS* research results:
FTFY.
FTFY.
I would give them a bit more wiggle room before giving up. We’re currently living under a monopoly of AWS (GCP is the most serious “competitor”, still far far behind in market share) and I would love to see more competition in this space.
That said: this is definitely not the kind of behavior I expect from them and I’m hoping there’s a better reason than them not willing to submit their products to an impartial evaluation.
That said: this is definitely not the kind of behavior I expect from them and I’m hoping there’s a better reason than them not willing to submit their products to an impartial evaluation.
AWS is a monopoly if you're living in a HN bubble. The vast majority of "hosting" (internet or otherwise) doesn't use AWS. Large telco's (Telefonica, AT&T, China Unicom, ...) have numerous data centers. Conglomerates (Tata, ...) have numerous data centers. You have wholesalers (Equinix, Digital Realty, ...). You have dedicated providers (OVH, Hetzner, PhoenixNap, ...). VPS' (DO, Linode, Vultr,...) and shared hosting (GoDaddy, ...)
Then you have a lot of private data centers, e.g. I worked at a major bank that had its own datacenters (multiple around the world). Admittedly, these are super small compared to an AWS DC, but there's literally thousands of these around the world (I know for banking and government, but I assume other industries do this too).
Then you have a lot of private data centers, e.g. I worked at a major bank that had its own datacenters (multiple around the world). Admittedly, these are super small compared to an AWS DC, but there's literally thousands of these around the world (I know for banking and government, but I assume other industries do this too).
You are forgetting Azure, which is much bigger than GCP.
I’m not forgetting Azure, it’s a deliberate omission because it is not a Tier 1 cloud product. It might be better than OCI or IBM Cloud but it simply does not have the maturity of either AWS or GCP.
Interesting. As someone who's only really used Azure (other than a little GCP) - what's missing?
Wrong. Cloudflare introduce another type of monopoly on top of the existing cloud services.
A very harmful one - they are trying to centralize the web.
And kill Tor, while at it.
A very harmful one - they are trying to centralize the web.
And kill Tor, while at it.
> Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two.
https://www.fastly.com/pricing/ does not make a distinction it says that "Any size company looking to give Fastly a try." They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.
https://www.fastly.com/pricing/ does not make a distinction it says that "Any size company looking to give Fastly a try." They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.
> They make distinctions on paying for more bandwidth, not that they are giving you an inferior product performance wise.
I think this is a non-sequitur. Bandwidth is a fundamental property of the performance of a system, if you can "pay for more bandwidth" then you are literally getting an inferior product when you do not pay.
Consider the implementation for "paying for more bandwidth". I assume that it is the same copper cabling hitting the same switches for fee paying and non-fee paying customers so the only way to "offer more bandwidth" is to implement some kind of quality-of-service metric on the network. This is explicitly giving the non-fee paying customer an inferior product.
I think this is a non-sequitur. Bandwidth is a fundamental property of the performance of a system, if you can "pay for more bandwidth" then you are literally getting an inferior product when you do not pay.
Consider the implementation for "paying for more bandwidth". I assume that it is the same copper cabling hitting the same switches for fee paying and non-fee paying customers so the only way to "offer more bandwidth" is to implement some kind of quality-of-service metric on the network. This is explicitly giving the non-fee paying customer an inferior product.
> Test up to $50 of traffic for free. Pay as you go from there with bandwidth pricing.
Looks like it's just post-use monitoring that'll cut off the account after it exceeds the trial limit or start billing per-GB; it doesn't say 'slower connection', so CF didn't realize that a trial account would affect their benchmarks regarding Compute@Edge
Looks like it's just post-use monitoring that'll cut off the account after it exceeds the trial limit or start billing per-GB; it doesn't say 'slower connection', so CF didn't realize that a trial account would affect their benchmarks regarding Compute@Edge
People have stopped using "bandwidth" as we used to. It now means a total sum of the amount transferred in a lot of cases, not the "width" of the pipe.
It is more evident here: https://www.fastly.com/pricing/#bandwidthpricing
(To be clear, I agree with your definition, but it is not used)
It is more evident here: https://www.fastly.com/pricing/#bandwidthpricing
(To be clear, I agree with your definition, but it is not used)
It's been hilarious watching competing companies publish warring blog posts declaring each other full of shit. I mean, I knew, but thanks for airing each others' dirty underpants.
It's just Fastly's laundry. Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. Yet they claim contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison,
> [FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services
>> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;
That is not so different from Fastly's own sign-up process.
[1] https://www.fastly.com/pricing/
[2] https://news.ycombinator.com/item?id=29468455
> [FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services
>> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;
That is not so different from Fastly's own sign-up process.
[1] https://www.fastly.com/pricing/
[2] https://news.ycombinator.com/item?id=29468455
How is a sales deal same as a tech stack, I fail to see your point about fastly’s laundry?
Unlike cloudflare I dont think fastly is in business for solo devs but for enterprises. And each enterprise has its own requirements.
If fastly prefers to know customer requirements before they start using it or tailor them whats wrong?
Meanwhile cloudflare prevents its OWN EXISTSING customers from benchmarking, which is actually draconian.
Unlike cloudflare I dont think fastly is in business for solo devs but for enterprises. And each enterprise has its own requirements.
If fastly prefers to know customer requirements before they start using it or tailor them whats wrong?
Meanwhile cloudflare prevents its OWN EXISTSING customers from benchmarking, which is actually draconian.
Most of cloudflare features are locked behind sales, powerpoints, and a couple meetings.
Out of curiosity, which features are you referring to? My impression (joined < 1 year ago) is that the vast majority of products are available to free customers (albeit with smaller limits perhaps) while the gap between self-service paid accounts and enterprise is smaller still.
My experience was with the UDP tunneling service and some general CDN stuff. Once you get past the published pricing then it’s sales calls and stuff like that.
Are you talking about Argo Tunnel or something else? Argo Tunnel was made free this year [1]. I don't think you even need a Cloudflare account. Also, I don't believe UDP by itself is supported yet (only QUIC & TCP) according to the github issue [2]. Is it possible there was a miscommunication. Our PMs love to talk to potential customers of products we're building so that we make sure we're building what you want & engaging with you when we have it ready as early beta users.
"General CDN stuff" is a bit too vague so I'm not really sure what you're referring to. I'm also not really sure what "get past the published pricing" means or how that relates to features being "locked" behind sales calls.
I'm sincerely trying to understand your experience and any suggestions you have to understand how I can push internally for us to be better and I appreciate your time in any responses you provide.
[1] https://blog.cloudflare.com/a-free-argo-tunnel-for-your-next... [2] https://github.com/cloudflare/cloudflared/issues/215
"General CDN stuff" is a bit too vague so I'm not really sure what you're referring to. I'm also not really sure what "get past the published pricing" means or how that relates to features being "locked" behind sales calls.
I'm sincerely trying to understand your experience and any suggestions you have to understand how I can push internally for us to be better and I appreciate your time in any responses you provide.
[1] https://blog.cloudflare.com/a-free-argo-tunnel-for-your-next... [2] https://github.com/cloudflare/cloudflared/issues/215
Cloudflare spectrum was what it was, has to dig up the emails. They wanted me to sign a 1 year contract with commit levels which is kinda hard with a growing product I was working on. I wasn’t even able to predict 1 month out let-alone 12.
The question is "who shares more details on pricing", not "who has more unpriced features".
https://www.cloudflare.com/plans/
https://www.cloudflare.com/plans/
The off-the-shelf prices from Cloudflare are also negotiable for larger companies.
It reminds me of being on Slashdot 15 years ago. We've even got one of the engineers posting in the comment thread, starting their own little flamewar. The names of the characters have changed, but there's still a bit of the same old Internet here.
The fact that a lead engineer and CEO took the time to respond here is clarity not a flamewar.
edit maybe I'm wrong,
https://news.ycombinator.com/item?id=29469174
edit maybe I'm wrong,
https://news.ycombinator.com/item?id=29469174
Most insightful comment here IMHO.
Vaguely reminiscent of the Glasgow Ice Cream Wars, I'd rather not buy from anyone involved
Vaguely reminiscent of the Glasgow Ice Cream Wars, I'd rather not buy from anyone involved
[deleted]
Popcorn time ;) Let's wait for @jgrahamc to wake up
Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite.
However TBH benchmarking a distributed system like Cloudflare/Fastly is going to be REALLY HARD. There are 1000s of servers involved and reproducing the results might be impossible.
Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite.
However TBH benchmarking a distributed system like Cloudflare/Fastly is going to be REALLY HARD. There are 1000s of servers involved and reproducing the results might be impossible.
> Simple solution is for Cloudflare to change TOS to allow benchmarking and for 3rd party to publish a reproducible benchmark suite.
Why doesn't Fastly just gasp ask Cloudflare about performing a benchmark? If they say no, publish that.
Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. It's absurd to say that contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison,
> [FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services
>> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;
That is not so different from Fastly's own sign-up process.
[1] https://www.fastly.com/pricing/
[2] https://news.ycombinator.com/item?id=29468455
Why doesn't Fastly just gasp ask Cloudflare about performing a benchmark? If they say no, publish that.
Fastly does not publish their pricing [1]. And, you need to contact a rep to sign up for commercial use [2], which implies everyone gets a different deal. It's absurd to say that contacting Cloudflare to do benchmarking means it's impossible for them to do a comparison,
> [FASTLY BLOG] it's impossible for us to define our own terms for a comparison because Cloudflare’s terms of use prohibit benchmarking of their services
>> [CLOUDFLARE TERMS] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;
That is not so different from Fastly's own sign-up process.
[1] https://www.fastly.com/pricing/
[2] https://news.ycombinator.com/item?id=29468455
Why are u parading all the comment threads slightly supportive of fastly? Unlike Fastly, cloudflare bars its own customers with TOS while fastly tailors their offering to new customers, I just don’t see your point.
It's a decent use case for planetlab.
And reproducible benchmarks with cached, publicly accessible results.
At some point, you either have to make free tiers worse, or deal with thousands of people using them to run the same benchmark over and over.
Cloudflare loves alliances: start a distributed benchmarking alliance! :)
At some point, you either have to make free tiers worse, or deal with thousands of people using them to run the same benchmark over and over.
Cloudflare loves alliances: start a distributed benchmarking alliance! :)
>Popcorn time ;)
Indeed. We already have comments claiming Fastly are dismissive and evasive while others claiming the fault of Cloudflare. The community is divided, which is quite usual on recent HN.
Let there be another holy war of CDN between Cloudflare and Fastly.
Indeed. We already have comments claiming Fastly are dismissive and evasive while others claiming the fault of Cloudflare. The community is divided, which is quite usual on recent HN.
Let there be another holy war of CDN between Cloudflare and Fastly.
> The community is divided
I don't see any division. It's clear to me who has experience to back up their words and who doesn't.
I don't see any division. It's clear to me who has experience to back up their words and who doesn't.
Ok, but let’s ideally not have the comments from that war wash up here.
In the context of "should I use Cloudflare or Fastly as my edge", I lean Cloudflare. That said, I enjoyed listening to the GraphCDN crew choosing Fastly[1] – one crucial feather in the hat being cache invalidation[2] (miss ya, Phil Karlton) – and it sounds like a solid choice.
[1]: https://www.youtube.com/watch?v=lpmpTJc_SP0 [2]: https://graphcdn.io/docs/how-to/purge-the-cache
[1]: https://www.youtube.com/watch?v=lpmpTJc_SP0 [2]: https://graphcdn.io/docs/how-to/purge-the-cache
I've used both Cloudflare and Fastly for many years, and the cache invalidation at Fastly is a way bigger deal than people think. They're not competing for a 5% improvement in performance, they're competing with a totally different set of features which mean you can cache 80% more things, as long as you design for it. (Which I'm sure is why they don't aggressively compete on free plans -- if you try to just throw it in front like Cloudflare you won't see much of a difference from Cloudflare.)
With some careful design to take advantage of it, I've gotten 5-10x faster page loads using Fastly. (The last time I pointed this out, people accused me of being a sockpuppet account. Feel free to Google me!)
With some careful design to take advantage of it, I've gotten 5-10x faster page loads using Fastly. (The last time I pointed this out, people accused me of being a sockpuppet account. Feel free to Google me!)
I can definitely see how new use cases opens up based on a 150ms (global) invalidation. Most of my use cases relies more on local invalidation and I haven't done enough homework about on what latency I really need (read: wrong line of business). That said; in the world of Cloudflare I think durable objects/kv store probably would come to serve as the next architecture for cache itself and lead to reductions in latency. Don't quote me on that though.
Yes. Caching is a different animal when you can count on fast, reliable, targeted invalidation. You can cache basically everything with super long TTLs, and just sniper-shot individual cache objects as soon as the backend updates.
I'm curious, what "totally different set of features" are you referring to?
150ms global cache invalidation, tagged HTTP responses, custom VCL, etc
what is "tagged HTTP responses"? Are you referring to surrogate keys?
[deleted]
Keep in mind that Fastly's C@E might not even be needed, as you get VCL as standard. You don't check pre-made boxes, you write a non-Turing complete but quite powerful language that gets atomically deployed to their whole CDN in a few seconds. I'm a big fan.
I think statistics like these tests are easy to sway in your favor, hence why Fastly is responding. Its not clear that Fastly didn't do the same thing, but a good ole' CDN rivalry does make for a good read. Someone get more mud to sling.
btw. for what it's worth their javascript to wasm is opensource:
- https://github.com/fastly/js-compute-runtime
- https://github.com/tschneidereit/spidermonkey-wasi-embedding
and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help)
its insane what they built, to do it and it will bring the whole wasm community forward if they succeed (and I hope they do)
- https://github.com/fastly/js-compute-runtime
- https://github.com/tschneidereit/spidermonkey-wasi-embedding
and besides that it is slower than nodejs it is still plenty fast (no matter that it is not as fast as they want) btw. it's startup is faster than node. (maybe better pgo might help)
its insane what they built, to do it and it will bring the whole wasm community forward if they succeed (and I hope they do)
Gentle reminder: That's
mostly because they acquired the team behind wasm. Which was already an opensource project at Mozilla.
Post on HN: https://news.ycombinator.com/item?id=24897641
Post on HN: https://news.ycombinator.com/item?id=24897641
It's really the other way around: we joined Fastly because we knew it's a place where we could do this kind of work in the open.
None of the code involved here existed a year ago, and none of it was somehow forced to be open source. (Also, the code in these two repositories is in many ways the most boring part of this solution. See this blog post for an excellent overview of the more interesting parts: https://bytecodealliance.org/articles/making-javascript-run-...)
None of the code involved here existed a year ago, and none of it was somehow forced to be open source. (Also, the code in these two repositories is in many ways the most boring part of this solution. See this blog post for an excellent overview of the more interesting parts: https://bytecodealliance.org/articles/making-javascript-run-...)
The article I linked too ( and the hn comments) in my original post seemed that fastly acquired the wasm team.
Sorry if I was mistaking.
Sorry if I was mistaking.
Oh, no worries at all!
While there was no acquisition involved, a whole group of folks working on WebAssembly at Mozilla (myself included) moved to Fastly last Fall. What I tried to emphasize is that instead of the projects at hand here being open source because they somehow had to be, we all joined Fastly because it's a place where this kind of project can be made open source (and be created in the first place!) :)
While there was no acquisition involved, a whole group of folks working on WebAssembly at Mozilla (myself included) moved to Fastly last Fall. What I tried to emphasize is that instead of the projects at hand here being open source because they somehow had to be, we all joined Fastly because it's a place where this kind of project can be made open source (and be created in the first place!) :)
As a consumer it’s great that both companies are benchmarking. Even if Cloudflare’s was flawed. Maybe a case of, the best way to get the right answer is to post the wrong answer.
We’re at least moving towards objective comparisons, talking about numbers.
Cloudflare prohibiting benchmarking is not good, obviously.
We’re at least moving towards objective comparisons, talking about numbers.
Cloudflare prohibiting benchmarking is not good, obviously.
point #2 is silly. So basically your product is slower but its ok cuz its a beta only... Fine. But then cf did not mislead anyone. Your beta is slow.
Not really. Cloudflare could have used their rust implementation for doing a fair comparison, instead they compared a beta product to a stable product. It's not like CF didn't have a choice to make a fair comparison.
Did CF pretend they were benchmarking something other than the product they said they were benchmarking?
> Today, we’re excited to report that Cloudflare Workers is 196% faster than Fastly’s Compute@Edge based on the time to first byte from the tests we ran on 50 nodes using Catchpoint’s data from across the world.
Kind of? At minimum they made misleading representations. They used abbreviated explanations of how they're better and then expanded on what the test actually consisted of later on.
Kind of? At minimum they made misleading representations. They used abbreviated explanations of how they're better and then expanded on what the test actually consisted of later on.
> At minimum they made misleading representations
Fastly does not have a production-ready JavaScript product. From their blog post,
> Their tests compare JavaScript running on Cloudflare Workers, a mature, generally available product, with JavaScript running on Compute@Edge. Although the Compute@Edge platform is now available for all in production, support for JavaScript on Compute@Edge is a beta product. We clearly identify in our documentation [2] that beta products are not ready for production use. A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle.
Restricting any comparison isn't satisfactory since we care about both the supported languages and performance. Let there be writeups about both and allow users to make up our own minds.
I think the Cloudflare article [2] could be amended to refer to the compared product as "Fastly's JavaScript on Compute@Edge" and all would be fine. Fastly will probably still say they advise against this comparison since it's "beta". Nonetheless, we should all feel free to compare publicly available products and write about them.
[1] https://docs.fastly.com/products/fastly-product-lifecycle#:~....
[2] https://blog.cloudflare.com/network-performance-update-full-...
Fastly does not have a production-ready JavaScript product. From their blog post,
> Their tests compare JavaScript running on Cloudflare Workers, a mature, generally available product, with JavaScript running on Compute@Edge. Although the Compute@Edge platform is now available for all in production, support for JavaScript on Compute@Edge is a beta product. We clearly identify in our documentation [2] that beta products are not ready for production use. A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle.
Restricting any comparison isn't satisfactory since we care about both the supported languages and performance. Let there be writeups about both and allow users to make up our own minds.
I think the Cloudflare article [2] could be amended to refer to the compared product as "Fastly's JavaScript on Compute@Edge" and all would be fine. Fastly will probably still say they advise against this comparison since it's "beta". Nonetheless, we should all feel free to compare publicly available products and write about them.
[1] https://docs.fastly.com/products/fastly-product-lifecycle#:~....
[2] https://blog.cloudflare.com/network-performance-update-full-...
I don't really think it's worth piling on the topic more, but Fastly's post (original topic) expands on why the test is poor in other ways worth reading over again.
Beyond just those points, the whole Cloudflare blog post reads like an attempt at a performance sales kill sheet. (In other words, their sales may point to this as to why folks should choose Cloudflare Workers over Fastly Compute@Edge.)
Beyond just those points, the whole Cloudflare blog post reads like an attempt at a performance sales kill sheet. (In other words, their sales may point to this as to why folks should choose Cloudflare Workers over Fastly Compute@Edge.)
> Fastly's post (original topic) expands on why the test is poor in other ways worth reading over again.
Do you want to highlight any of those? RTT is mentioned in a post from "Cloudflare Research" which was just shared by their head of research [1].
> Beyond just those points, the whole Cloudflare blog post reads like an attempt at a performance sales kill sheet. (In other words, their sales may point to this as to why folks should choose Cloudflare Workers over Fastly Compute@Edge.)
There is nothing wrong with advertising. This comment section is just about keeping them honest.
[1] https://twitter.com/grittygrease/status/1468016152752361472
Do you want to highlight any of those? RTT is mentioned in a post from "Cloudflare Research" which was just shared by their head of research [1].
> Beyond just those points, the whole Cloudflare blog post reads like an attempt at a performance sales kill sheet. (In other words, their sales may point to this as to why folks should choose Cloudflare Workers over Fastly Compute@Edge.)
There is nothing wrong with advertising. This comment section is just about keeping them honest.
[1] https://twitter.com/grittygrease/status/1468016152752361472
how comparing rust to javascript is even remotely fair ?
I think people are misunderstanding what I said. CF could have tested their rust implementation against Fastly's rust implementation.
does the language that you use matter when the real goal is to serve a webpage to the end user in the shortest time possible?
Yes, maintainability, ease of porting code, and ease of hiring knowledgeable workers do matter, and so does performance.
I hope Fastly stops calling things "cold starts" that aren't [1].
And, I think Cloudflare could do a quick find-and-replace on that article [2] to change "Fastly’s Compute@Edge" to "Fastly’s JavaScript on Compute@Edge". A phone call between friendly CEOs probably would have sufficed for that.
We all want to be on the same page when making these comparisons. Some may choose Fastly and write in Rust, some will choose Cloudflare and write in JavaScript.
[1] https://news.ycombinator.com/item?id=29465729#29468068
[2] https://blog.cloudflare.com/network-performance-update-full-...
I hope Fastly stops calling things "cold starts" that aren't [1].
And, I think Cloudflare could do a quick find-and-replace on that article [2] to change "Fastly’s Compute@Edge" to "Fastly’s JavaScript on Compute@Edge". A phone call between friendly CEOs probably would have sufficed for that.
We all want to be on the same page when making these comparisons. Some may choose Fastly and write in Rust, some will choose Cloudflare and write in JavaScript.
[1] https://news.ycombinator.com/item?id=29465729#29468068
[2] https://blog.cloudflare.com/network-performance-update-full-...
Absolutely, runtime overhead plays an outsized role in startup time and latency. garbage collection can also add a ton of latency.
Awesome, now where do I sign for the free fastly account? Oh right, they only do enterprise, so who the hell cares? Cloudflare OTOH caters for the whole spectrum.
I also read that Fastly is slower at specific point of times during a week, slower for free test accounts, slower at executing javascript (hence the switch to rust), and slower in locations carefully selected by CloudFlare.
I'm not saying that the benchmark from CloudFlare is fair, but it's a bit funny that their response contain a "let's switch to rust" or "let's benchmark next to our datacenters".
I'm not saying that the benchmark from CloudFlare is fair, but it's a bit funny that their response contain a "let's switch to rust" or "let's benchmark next to our datacenters".
I mean they switched to Rust because a comparison of two fully developed systems is fairer than a production system and one in testing. Besides, I believe the blog post said that they selected nodes at actual random?
I'm regularly surprised why Cloudflare gets so much love specially here on HN. Cloudflare is slow, it man-in-the-middles websites, it shows a redirect page randomly for no apparent reason on websites that it powers, it does questionable things at the DNS level, it randomly shows captchas for no apparent reason, the list goes on. Almost every website nowadays uses this monstrosity. We're handing over the internet on a silver platter to them. I think there are better alternatives and I'm so glad at least Fastly is debunking their bs experiment.
>no apparent reason
A big reason is making sure the website behind it stays online. Having to deal with weekly/daily DDOS attacks yourself takes a lot of time.
A big reason is making sure the website behind it stays online. Having to deal with weekly/daily DDOS attacks yourself takes a lot of time.
> it man-in-the-middles websites
That's just how reverse proxies work. I'm not saying that this is good, but anyone offering the same service is doing the same thing.
That's just how reverse proxies work. I'm not saying that this is good, but anyone offering the same service is doing the same thing.
This comes across as incredibly defensive in tone from Fastly. They could have just reported the results of their test. But calling their competitors outright liars reflects worse on them than anyone else.
They do have to punch up because they are the little guy in marketing, but they did it in a very sloppy manner with shaky points.
When you're sucker punched with misleading stats, ya kind have to hit back. And didn't sound defensive at all, btw.
Only semi-related, but why does Cloudflare perform so poorly in Oceania? I've noticed less than stellar performance in Oceania with Cloudflare's CDN, so to see similar stats with workers makes it seem it's not a product-specific issue.
I'm based Australia and have used Cloudflare for personal/business; if you aren't on the $200/month plan for your website then you don't get (consistent) access to the Australian-based servers for your visitors (they may be routed to eg Singapore instead during some times of the day) which actually hurts performance if your service is located in (say) AWS/Azure Australia region, at least for the CDN aspect, I assume the same for Workers too.
Same here in New Zealand too. Cloudflare peers only with certain ISPs. The ones they don't peer with sees some traffic (e.g. sites on free plans) routed to offshore Cloudflare POPs around the world instead of local Cloudflare POPs. I understand if the website is on a paid plan they will be more likely to route to a local POP (incurring more costs in doing so).
According to this 2016 post, the problem is Optus and Telstra, and the costs of peering with them (scroll down to "Oceania"):
https://blog.cloudflare.com/bandwidth-costs-around-the-world...
https://blog.cloudflare.com/bandwidth-costs-around-the-world...
I'm in India and if you're on the free CF plan, you often seem to get served from the Singapore POP instead of the Indian ones. I've seen other people on various groups complain about this too.
> _We used a Wasm binary compiled from Rust rather than JavaScript.
> We know support for JavaScript is important to many customers, but we're not yet satisfied with the performance of Compute@Edge packages compiled from JavaScript. That's why it's in beta. When a product is ready for production, we remove the beta designation.
Having said that, I would notice anything over 150ms in my clicks, but wouldn't care whether something took under 100 or under 50 - except that the lower the latency the less the scale needed to serve the same number of active users (it becomes a question of cost rather than response time).
> I can't really square the idea that the 50-150ms time-delays in question comes down to the actual programming language, but it is absolutely believable that a longer test reduces the median latency rather than a high load test for a shorter duration.
It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible:
https://bytecodealliance.org/articles/making-javascript-run-...
The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform.
I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps.
1. https://github.com/bytecodealliance/wasmtime
It seems plausible to me: in Fastly's case, they're using WebAssembly via wasmtime[1] which does support AOT compilation but most JavaScript code is dynamic enough that they still need a runtime JIT engine. I believe the the current approach Fastly is using is to compile Mozilla's SpiderMonkey JIT engine itself into WebAssembly and they've done some really nice work making that load as quickly as possible:
https://bytecodealliance.org/articles/making-javascript-run-...
The catch, of course, is that this still leaves a fair amount work which a JavaScript program has to deal with at runtime compared to a Rust program which the compiler can spend minutes optimizing long before deployment. This is a classic tradeoff for dynamic languages and a lot of people are satisfied with the approach of defaulting to faster developer turnaround and later converting hot spots to something like C or Rust, but I think it's definitely dodgy to use a single example of something you know to be this complex and present the results as generally representative of the entire platform.
I have no knowledge of how Cloudflare ran their tests or reason to suspect malice but not disclosing test methodology and a “no benchmarks” license clause is going to make accusations about the results inevitable. Veterans of the benchmarketing wars have stories about comparisons where someone used a configuration option which happened to disable key performance optimizations for their competitors or used examples which favored their design decisions. Since nobody can read someone's mind to tell their real intent, the best way to avoid acrimony is to have full public disclosure of the benchmarks and their methodology — and to allow other people to conduct benchmarks so they can reproduce results or fill gaps.
1. https://github.com/bytecodealliance/wasmtime
It definitely comes down to the language — JavaScript is an interpreted language, which requires an engine. Rust is statically compiled and runs by itself.
On Cloudflare, JavaScript is a JITted language, not strictly interpreted (it uses V8). Code that's seeing any amount of traffic gets lowered into native code pretty darn quick & V8's profile-guided optimizer is pretty impressive at removing JS overheads.
Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure.
The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute.
Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).
Rust is statically compiled but it runs under WASM. If I'm not mistaken, WASM is also typically interpreted & then JIT'ed to native code (at least it is on V8 IIRC). My guess would be that Fastly lowers it to native code AOT (or at least does a fair bit of optimization) but I'm not 100% sure.
The question is whether Rust WASM will beat CF JS and that part is less clear because the workloads you'd choose for JS vs Rust are so different. Also, these platforms aren't static so it'll be interesting to watch the competition play out as we both push the boundaries of Edge compute.
Disclaimer: I work at Cloudflare on R2. I think both teams do great technical work but I'm more excited about CF. I think our suite of Edge compute products built on top of Workers is stronger (cron triggers, unbound, durable objects & many other interesting things coming).
In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language.
Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).
Startup time might be a factor. But the measurements of the same language running on the same provider's infrastructure go from around 10 to 50ms (on one provider) and 10 to 100ms (on the other). Those huge differences don't look like CPU latencies (the machines aren't 5x or 10x faster in different locations) - they are likely network latencies. So programming language is probably a very minor factor here (at least in the non-beta services, which is what was measured here - maybe it was an issue in the older measurements).
> In general you'd be right, but this is not a benchmark of code running in those languages. There is no loop, no heavy computation - it's just returning the headers. That should take less than 1ms in either language.
They can't have it both ways... Fastly was the one that complained that their JS engine still wasn't up to par (being in beta), so clearly the language of choice (or interpreter/compiler/whatever) DOES matter.
Either the language doesn't matter (in which case, replicate the JS test) or it does, in which case this isn't even a useful comparison.
They can't have it both ways... Fastly was the one that complained that their JS engine still wasn't up to par (being in beta), so clearly the language of choice (or interpreter/compiler/whatever) DOES matter.
Either the language doesn't matter (in which case, replicate the JS test) or it does, in which case this isn't even a useful comparison.
Sorry, I should have been clearer. The new benchmark data does not look like it is influenced by the language in a noticeable way (since we see 5x or 10x differences in the mean even when using the same language, and which provider is faster or slower depends on which location you measure at).
But it's very possible the old data may have been influenced by the language. That's not very interesting, though: as Fastly said, it's just beta performance. And anyhow this is a benchmark with almost no CPU work anyhow.
But it's very possible the old data may have been influenced by the language. That's not very interesting, though: as Fastly said, it's just beta performance. And anyhow this is a benchmark with almost no CPU work anyhow.
I'd also expect unrelated load to be a factor, especially if there's a mechanism which priorities paid customer traffic over the free tier. Given the complexities of loading and running JavaScript, I'd be especially interested in what memory contention looks like — it seems like it could be as simple as Rust needing far less memory than loading all of SpiderMonkey and thus being less likely to hit both low-level CPU cache and high-level runtime cache evictions.
Ah, good point, memory consumption might indeed be a factor here. If that's the case I'd expect higher variance in the measurements perhaps - looking for that in the raw data could be interesting.
[deleted]
[deleted]
The “DeWitt Clause” or sometimes also known as the “Oracle Clause” has been a fixture of ToS for years. The fact that it features in Cloudflare’s docs only serves to remind us that entrenched practices die hard.
https://www.brentozar.com/archive/2018/05/the-dewitt-clause-...
That is pretty scummy that Cloudflare won't allow benchmarks.
"A fairer test on this point would have compared Rust on Compute@Edge with JavaScript on Cloudflare Workers, which are at more comparable stages of the product lifecycle."
Are there any widely agreed upon benchmarks on how emscriptened rust compares to untyped js? It feels like a large portion of the reported Delta boils down to that.
Are there any widely agreed upon benchmarks on how emscriptened rust compares to untyped js? It feels like a large portion of the reported Delta boils down to that.
[deleted]
I am glad I said no to working for both. When I saw both are politically active and showed child like behavior during the interviews, I said no, and decided to dedicate some of my time to dog and cat rescue.
[deleted]
[deleted]
Another appendage measuring contest.
This reads as someone who his losing lots of customers to CF and whines about it.
wgaf
xwdv(2)
Seriously, who cares about 5ms difference? Or even 25ms. I don't. And Asia is just used to longer latencies.
Edge compute became a commodity even before it was born.
The whole point of edge workers is to be able to provide very low latency all around the world... once you have that, it opens up whole new classes of workloads. Yes, 25ms doesn't matter for most current use cases.... but that is partly because the use cases that required lower latency weren't possible before.
I’d love to be able to (affordably) spin up a low-latency Windows desktop for a few hours at a time, that I could use for applications which don’t run well on my bottom-tier M1 Mac
Cloudflare seems to encourage benchmarks of Workers Sites:
https://blog.cloudflare.com/workers-sites/
https://blog.cloudflare.com/workers-sites/
So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible.
But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number?
We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons...
On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.