Azure Functions vs. AWS Lambda – Scaling Face Off(azurefromthetrenches.com)
azurefromthetrenches.com
Azure Functions vs. AWS Lambda – Scaling Face Off
https://www.azurefromthetrenches.com/azure-functions-vs-aws-lambda-scaling-face-off/
61 comments
Wow, that is pretty substantial difference between AWS and Azure. I’m pretty confident Microsoft is hard at work getting these numbers better, but it shows just how lonely AWS is at the top. I’m pretty deep into AWS-land but I’m rooting for the boys & girls in Redmond; more competition == better service.
Concur; despite being so deep in AWS I've drowned here, competition makes all of the offerings in this space better. It'd be nice to have a migration strategy in my back pocket if I needed it; today that's a bit of a pipe dream.
“On Azure I’m using .NET 4.6 on the v1 runtime while on AWS I’m using the same code running under .NET Core 1.0.“
.NET Core has way less baggage than .NET 4.6 [1] so you’d be expecting AWS to come out ahead.
It looks like Azure functions supports .NET Core now [2]. (Edit: just noticed the author is aware of this [3].)
Of course it might also be Blob storage having higher latency than S3.
Having used Azure Functions before I agree with the article’s criticisms about scaling.
[1] https://www.ageofascent.com/2016/02/18/asp-net-core-exeeds-1... Mind this article is talking about comparing ASP.NET with ASP.NET Core, and .NET with .NET Core, not just .NET Core’s performance.
[2] https://github.com/Azure/Azure-Functions/issues/98
[3] https://www.azurefromthetrenches.com/azure-functions-v2-prev...
.NET Core has way less baggage than .NET 4.6 [1] so you’d be expecting AWS to come out ahead.
It looks like Azure functions supports .NET Core now [2]. (Edit: just noticed the author is aware of this [3].)
Of course it might also be Blob storage having higher latency than S3.
Having used Azure Functions before I agree with the article’s criticisms about scaling.
[1] https://www.ageofascent.com/2016/02/18/asp-net-core-exeeds-1... Mind this article is talking about comparing ASP.NET with ASP.NET Core, and .NET with .NET Core, not just .NET Core’s performance.
[2] https://github.com/Azure/Azure-Functions/issues/98
[3] https://www.azurefromthetrenches.com/azure-functions-v2-prev...
He also compares AWS Lambda with Azure blob storage backend and it still lot faster - just 25ms added latency to cross internet from AWS to Azure DC.
[deleted]
It’s also difficult to run true x64 bit workloads on azure functions as it invokes the x86 version of the dotnet executable, regardless if the application is set to x64
My experience so far on Google Cloud has been just as bad. I want a simple app that ingests DMARC reports.
- Google Cloud Functions can't ingest e-mail. Also, JS lockin if you care about which platforms are supported.
- Google App Engine can only ingest e-mail in the Standard environment. OK, fine. But the standard environment regularly takes almost a full minute to load and boot my app, after which the request is killed for taking too long. Additionally, the sample code (!) for opening new Stackdriver contexts breaks with a totally inscrutable error message.
You can say "well, maybe your app sucks", but it's been running on AWS (SES + Lambda) for months and every aspect about it has been painless.
- Google Cloud Functions can't ingest e-mail. Also, JS lockin if you care about which platforms are supported.
- Google App Engine can only ingest e-mail in the Standard environment. OK, fine. But the standard environment regularly takes almost a full minute to load and boot my app, after which the request is killed for taking too long. Additionally, the sample code (!) for opening new Stackdriver contexts breaks with a totally inscrutable error message.
You can say "well, maybe your app sucks", but it's been running on AWS (SES + Lambda) for months and every aspect about it has been painless.
Use Mailgun to trigger a POST to your Cloud Function on receiving an email. Bonus: all the parsing is done for you.
I don’t know how magic is in Cloud Functions, but I personally set up Mailgun postbacks in AWS and it was a huge pain in the ass to configure correctly. I’d love I know if google does something differently that would have been easier.
[deleted]
But the parsing code already works fine, that’s not really the hard bit. Then I get all of the lock-in for questionable benefit and something that Just Works on AWS.
Do you know how Google Cloud Functions compares with AWS and Azure?
GCF still suffers from JS lock-in, as they currently only support Node, thus you can't make a .NET comparison in the first place. That being said, of the various offers I really like the UI and the like.
Disclaimer: I am at Google, but not on GCF.
Disclaimer: I am at Google, but not on GCF.
You technically can hack it to work:
https://medium.com/@salmaan.rashid/running-net-on-google-clo...
Native support for more languages is coming.
https://medium.com/@salmaan.rashid/running-net-on-google-clo...
Native support for more languages is coming.
If you're going to be handling http requests it's almost a given you'll be using AppEngine rather than Cloud Functions. Still "serverless" but more tools and languages to choose from.
AppEngine is more like a PaaS and not like 'serverless'
This. AppEngine assumes an "always on" mentality and is far from "pay for use" unless your "use" is 24/7.
AppEngine standard can actually scale down to 0 instances when not in use to save money. Still very different from a serverless architecture though.
Not the flexible environment iirc. Which is basically the reasonable env to expect, now. Standard env has some aging issues that keep getting worse. most notably jvm 7, if that’s your poison, but just all round crippled env lacking standard libs and with lock in substitutions.
As far as I remember, that is..
As far as I remember, that is..
Java 8 is now supported with the full standard library:
https://cloud.google.com/appengine/docs/standard/java/runtim...
> If you're going to be handling http requests it's almost a given you'll be using AppEngine rather than Cloud Functions
Why? Cloud Functions support HTTP.
I mean, if you can stay within AppEngine’s free tier quotas, there's a clear reason to use it, but otherwise it would seem to be a matter of weighing pros and cons, with HTTP alone not deciding that you should use AE over CF.
Why? Cloud Functions support HTTP.
I mean, if you can stay within AppEngine’s free tier quotas, there's a clear reason to use it, but otherwise it would seem to be a matter of weighing pros and cons, with HTTP alone not deciding that you should use AE over CF.
Are you kidding? You know how much you have to pay for AppEngine's right?!!
Nothing, if it's a small app?
https://cloud.google.com/free/docs/always-free-usage-limits#...
Disclaimer: I work at GCP.
https://cloud.google.com/free/docs/always-free-usage-limits#...
Disclaimer: I work at GCP.
This face-off will be a lot more meaningful in a couple months when we can have both serverless platforms running on .Net Core 2.0. I do look forward to reading the results of that updated face off.
Hi, author here - completely agree.
I did try the preview version of Core 2.0 on Azure Functions but it's absolutely woeful at the moment and the dev team confirmed they've done no performance work at all. As I recall the single request response time was around 15ms faster for Core than for full fat framework but the response to increased load was dreadful - never falling below a several second response time with the 400 concurrent user test and errors creeping in.
I largely set off down this path as even in isolation the Azure numbers were concerning and not at all what I expected from a serverless architecture and so I wanted a reference point.
As soon as I get an update from MS about Core performance work or it leaves preview I'll be rerunning the tests as I'm very aware that doing this across the two runtimes is less than ideal.
However from what I've observed the issues look to be more about the broader architecture used for Azure Functions than the framework runtimes. I'm hoping to learn more from Microsoft.
I did try the preview version of Core 2.0 on Azure Functions but it's absolutely woeful at the moment and the dev team confirmed they've done no performance work at all. As I recall the single request response time was around 15ms faster for Core than for full fat framework but the response to increased load was dreadful - never falling below a several second response time with the 400 concurrent user test and errors creeping in.
I largely set off down this path as even in isolation the Azure numbers were concerning and not at all what I expected from a serverless architecture and so I wanted a reference point.
As soon as I get an update from MS about Core performance work or it leaves preview I'll be rerunning the tests as I'm very aware that doing this across the two runtimes is less than ideal.
However from what I've observed the issues look to be more about the broader architecture used for Azure Functions than the framework runtimes. I'm hoping to learn more from Microsoft.
I'm personally leaning towards using AWS Lambda for .Net because unlike Azure it will also support ASP.NET.
In general, from the perspective of a dev in a MS-shop, AWS gives you a bit more 'responsibility' for infrastructure than Azure, but that level of control lets you get further in practice sooner.
Azure has a lot of great things going for it, but I've had a lot of issues getting newer tech out on it without considerable delay. I hope both Azure and GCE give AWS a run for their money in coming years :)
Azure has a lot of great things going for it, but I've had a lot of issues getting newer tech out on it without considerable delay. I hope both Azure and GCE give AWS a run for their money in coming years :)
The biggest issue with Azure functions is the pricing model where you have to provision and pay for servers to execute your functions... How is that serverless?
You don't have to provision servers with Azure Functions - but you optionally can.
I mean, I guess some consider that you don’t have to maintain the servers and therefore it’s transparent to you and “serverless”.
Does anyone know if any of these services provide the ability to cycle between IPs? One of the use cases for Lambda (and now Azure Functions) is in web scraping where companies get mad if you access them too many times from a single address. Lambda made it clear while they do use a lot of IPs you're not guaranteed to get distribution among them.
Seems like a long shot but I figured this is a good place to bring it up.
Seems like a long shot but I figured this is a good place to bring it up.
I have the same question. Not for scraping but for APIs that are rate-limited by IP address. Like some Google APIs. I certainly wouldn't count on IP distribution from getting around limits for long. Anyway, I'm more interested in understanding how, if at all, we will have to tweak our client code to be well-behaved.
Interesting article, but you don't mention what tier of AppService plan you tested against in Azure (or what memory limit your Lambdas had)? I would also be curious to see the same comparisons, but with Azure Functions in consumption mode which, despite the perceived drawbacks you mention, would really be the closer apples-to-apples test, wouldn't it?
Does AWS have first class fully production Python yet? In Azure a single import (pydocumentdb) took a function up to 20-30s on Python, I guess they're serious about it being experimental still sadly. Rewriting in Node it takes 50ms for the entire run, crazy difference.
I do like the "durable functions" concept in Azure preview. AWS has step functions, but coding in flowcharts isn't scalable.
My money is going to the innovators and first movers, regardless.
That mindset doesn't change these numbers then seeing that AWS released Lambda in 2014...years ahead.
Have to agree with that part, and am in no way taking away from the commendable engineering on the Azure side. But as a developer, it makes more sense for me to invest time and effort getting good at the platform that innovates faster - and the pace at which AWS has been getting out new functionality in the platform has been breathtaking.
Did anyone try Bluemix Functions?
so does this mean that the cold run startup time thing has really been improved enough to build web applications? To me that is the biggest killer in moving forward, not knowing what the end users experience is going to be like because of cold startups that forces a user to wait a long time.
From the article's data, it's really tempting to use AWS Lambda now given this benchmark.
From the article's data, it's really tempting to use AWS Lambda now given this benchmark.
Never ever use averages in your stats. Use medians and percentiles.
Averages are a meaningless stat especially if you benchmark.
Averages are a meaningless stat especially if you benchmark.
I would say leave the average in but also show standard deviation and 95th percentile average. Of course it is still very unlikely that Azure Functions would come out on top in this case.
What leads you to believe performance follows a normal distribution?
For a given especially slow request would you expect a corresponding especially fast request?
For a given especially slow request would you expect a corresponding especially fast request?
Standard deviation applies to all distributions, not just the normal distribution.
You are arguing against yourself here. Having the mean and 95th/99th percentile is going to give you a better picture of the distribution than mean alone. Mean alone actually doesn’t give you ANY idea of the distribution. I don’t know why OP is being downvoted, managing to 95th percentile is very common practice and I also thought was a missing addition to the article.
Never use the best linear unbiased estimator? That's a strong statement.
It’s not the best. There’s a Russian joke that goes “One person is cooling in the morgue, the other is running a high fever. The average temperature in the hospital is 36.6 C though”
Averages are very susceptible to outliers and don’t give a proper insight into your data. What you really want is: how did half of my requests perform, 90% of requests perform, 95% of requests perform, 99% of requests perform
Averages are very susceptible to outliers and don’t give a proper insight into your data. What you really want is: how did half of my requests perform, 90% of requests perform, 95% of requests perform, 99% of requests perform
Median is a type of average. You are confusing "average" with "mean".
To be fair, it's not clear which form of average is used in the article and your assumption that it is the mean is likely correct.
To be fair, it's not clear which form of average is used in the article and your assumption that it is the mean is likely correct.
I have never seen median referred to as a "type of average" in any context.
Is there any foundation to this?
Is there any foundation to this?
Quite literally the first result in Google: http://www.dummies.com/education/math/pre-algebra/the-three-...
Also, from webster's:
"av·er·age\ˈa-v(ə-)rij\ noun
: a number that is calculated by adding quantities together and then dividing the total by the number of quantities
: a level that is typical of a group, class, or series : a middle point between extremes
Full Definition
1 a : a single value (as a mean, mode, or median) that summarizes or represents the general significance of a set of unequal values b : mean 1b 2 a : an estimation of or approximation to an arithmetic mean (...)"
It's common to see arithmetic mean and average to be treated as synonyms - but the mean is really just a type of average.
"av·er·age\ˈa-v(ə-)rij\ noun
: a number that is calculated by adding quantities together and then dividing the total by the number of quantities
: a level that is typical of a group, class, or series : a middle point between extremes
Full Definition
1 a : a single value (as a mean, mode, or median) that summarizes or represents the general significance of a set of unequal values b : mean 1b 2 a : an estimation of or approximation to an arithmetic mean (...)"
It's common to see arithmetic mean and average to be treated as synonyms - but the mean is really just a type of average.
You might like this blog post from John Myles White: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medi...
It compares mode, median, and mean as each being specific implementations of a more general concept of discrepancy.
It compares mode, median, and mean as each being specific implementations of a more general concept of discrepancy.
Colloquially, median and mode are used as average more than you'd expect. For example, we might say "the average joe is right handed", which is most likely the mode.
this is absolutely not a function of Azure vs AWS, the perf diff is completely between .Net Framework (which was used on Azure) and .Net Core (which was used on AWS). This compares Apples and Watermelons.
https://www.ageofascent.com/2016/02/18/asp-net-core-exeeds-1...
http://codinggorilla.domemtech.com/?p=1499
https://www.ageofascent.com/2016/02/18/asp-net-core-exeeds-1...
http://codinggorilla.domemtech.com/?p=1499
He tried Azure functions with C# on .Net Core with even worse results:
https://www.azurefromthetrenches.com/azure-functions-v2-prev...
https://www.azurefromthetrenches.com/serverless-blog-christm...
https://www.azurefromthetrenches.com/azure-functions-v2-prev...
https://www.azurefromthetrenches.com/serverless-blog-christm...
Is .Net Core available on Azure functions?
If not, then the comparison is an accurate snapshot of the current status.
And the performance is dreadful: https://www.azurefromthetrenches.com/azure-functions-v2-prev...