ASP.NET Core – 2300% More Requests Served per Second(ageofascent.com)
ageofascent.com
ASP.NET Core – 2300% More Requests Served per Second
http://www.ageofascent.com/asp-net-core-exeeds-1-15-million-requests-12-6-gbps/
66 comments
I remember reading these results a little while ago and thought the numbers were great until I saw the netty results. There is a lot of work left to be done.
To be fair, almost everything else has a lot of work to do when compared to netty.
Screams micro-benchmark when you can saturate a 10Gb with the server quite readily. If you can't, chances are the problem is somewhere else anyway.
This is the TechEmpower plaintext benchmark (http://frameworkbenchmarks.readthedocs.org/en/latest/Project...). I'd agree that it's kind of a micro-benchmark. There are other TE benchmarks that test more real-world web application scenarios.
My understanding is that the ASP.NET team started at the bottom of the stack and is working their way up. They want to get fundamental HTTP serving basics nailed first. I think that makes sense - they were getting around 170k rps in September, so there were definitely some things to sort out at the basic HTTP serving layer.
My understanding is that the ASP.NET team started at the bottom of the stack and is working their way up. They want to get fundamental HTTP serving basics nailed first. I think that makes sense - they were getting around 170k rps in September, so there were definitely some things to sort out at the basic HTTP serving layer.
http.sys smokes netty in almost every way.
Doesn't looks like it does: https://github.com/aspnet/benchmarks/blob/dev/README.md -- look under experimental baselines
it actually doesn't which is kind of shocking on windows because http.sys is kernel mode
http ASICs?
(Or does that even make sense, given that undiscovered exploits might require a hardware upgrade?)
(Or does that even make sense, given that undiscovered exploits might require a hardware upgrade?)
what were they ?
Netty results are listed on https://github.com/aspnet/benchmarks/ for comparison - latest is 2.8 million requests/s. These are the standard Techempower benchmarks.
It makes me happy to see .net developers get nice things.
It is a great time to be a .Net Developer :)
Unless you want to use F#, then it's an awful time because it still doesn't support .Net Core :(
I was able to get my F# code to compile using .NET Core yesterday! It's difficult to find the correct project.json parameters and you have to build the latest CLI yourself, but it works.
I'm not even concerned about project.json because I don't particularly care for dotnet-cli, so if FSC can target .Net Core that's awesome news :)
I used fsharp-dnx. The problem was it then doesn't support non .NET core targets. So for library development it's a real pain in the arse when you want to build multiple .NET targets (net45,net46,dnx...) with the same project.json.
The whole thing is a total mess at the moment, so I'm personally going to stay away from dnx until MS produce something that's close to their old RTM quality releases. It seems there's even a rival system with the dotnet CLI.
So I disagree strongly with the OP, it's not a great time to be a .NET developer. The improvements to C# and F#, yes, fantastic. But the ecosystem around it is so brittle at the moment; which is something that we never had to deal with in the past, and was one of the big selling points of the MS ecosystem.
The whole thing is a total mess at the moment, so I'm personally going to stay away from dnx until MS produce something that's close to their old RTM quality releases. It seems there's even a rival system with the dotnet CLI.
So I disagree strongly with the OP, it's not a great time to be a .NET developer. The improvements to C# and F#, yes, fantastic. But the ecosystem around it is so brittle at the moment; which is something that we never had to deal with in the past, and was one of the big selling points of the MS ecosystem.
It's still in development. In the past you weren't even able to glance inside the sausage factory. If you want something solid then wait until 1.0 is out, preferably 2.0.
Thanks for the history lesson. I am fully aware of what stage it's at. I am commenting on the solution and what appears to be a meandering target, with huge holes. It's currently at RC but no support for F#, no support for SignalR, VS2015 tooling is comically bad, cripplingly slow build times, and now a totally new toolchain (dotnet-CLI) that doesn't support the same dependency system that dnx supports.
That doesn't sound like a RC to me. I have a general feeling that the RTM standards are slipping massively at MS. Windows 10 was a dog when I tried it - and obviously not finished, VS2015 is very unstable, then the issues above... It feels like stuff is being pushed out the door much earlier than it previously would have been.
I do appreciate the direction they're taking, it just feels rudderless atm.
That doesn't sound like a RC to me. I have a general feeling that the RTM standards are slipping massively at MS. Windows 10 was a dog when I tried it - and obviously not finished, VS2015 is very unstable, then the issues above... It feels like stuff is being pushed out the door much earlier than it previously would have been.
I do appreciate the direction they're taking, it just feels rudderless atm.
They may call it RC, but it's barely beta. Recently they said they'd take their time to get it right - less conference driven development. This is a new start, SignalR will come after RTM, so lots of things won't be there. Think of it as version 1 rather than 5. There's currently no VB support either, so F# may take a bit.
dotnet-cli is supposed to be a replacement for DNX. Personally, I'm trying to push to get a "classic" system-wide profile for .Net Core so I can just use CMake or whatever and not have to deal with the headache of dotnet-cli.
Would you share this process with us somewhere?
CoreCLR has F# in the active ports section, but asp.net core 1.0, I do not see it, other than VB being post RTM work (in q3 2016). But, if you want to do idiomatic F#, suave.io seems to be the consensus. So things to watch for would be suave.io team using .net core 1.0 (once F# is ported to .net core 1.0) . I would not bet my farm on F# - asp.net Core1.0, anytime soon.
If you want to use F# it's been an awesome time since 2009, F# rocks.
I just switched from a 3 years full-time F# position to a full time PHP dev and let me tell you I'm starting to understand how much of a good time I was getting.
I just switched from a 3 years full-time F# position to a full time PHP dev and let me tell you I'm starting to understand how much of a good time I was getting.
How come you didn't just get another f# roll?
Yikes. You must have missed the large sign in that career off ramp 'wrong way, go back!'!
That must have been a weird switch. Can't you use something like Haxe?
It's being worked on:
https://github.com/Microsoft/visualfsharp/wiki/F%23-for-Core...
https://github.com/Microsoft/visualfsharp/wiki/F%23-for-Core...
It is!
Agreed!
Great job Ben, the dotnet core team and the asp.net team!
anyone here could give the ideal dev environment for developping on mac and deploying on linux while programming with asp.net core ?
Easiest way to get set up is to run the installers at http://get.asp.net - there are installers for Windows / OS X / Linux there.
If you'd rather install via command-line, there are walkthroughs for all platforms here: https://docs.asp.net/en/latest/getting-started/index.html
There's an easy tutorial showing how to dev on OS X using Yeoman and VS Code (free) here: https://docs.asp.net/en/latest/tutorials/your-first-mac-aspn...
If you'd rather install via command-line, there are walkthroughs for all platforms here: https://docs.asp.net/en/latest/getting-started/index.html
There's an easy tutorial showing how to dev on OS X using Yeoman and VS Code (free) here: https://docs.asp.net/en/latest/tutorials/your-first-mac-aspn...
Seems like deplying on linux server is still a second class citizen, am i right ? Different, slimmed down framewoek, not so many resources for using clouds other than azure (didn't find any article for deploying on heroku using official tools for example).
That's not really surprising, because ms still got to make some money somehow, but that means i'll have to wait a bit longer to get anything descent on the server side..
That's not really surprising, because ms still got to make some money somehow, but that means i'll have to wait a bit longer to get anything descent on the server side..
Part of the point of the new asp.net core is that it's slimmed down everywhere, not just linux (you're not getting jipped, they're just cutting off weight you don't need).
And to be fair, other cloud providers are responsible for implementing their own C# IaaS stack. To use heroku as an example, they implement their own stack and add languages that they support at their own pace (I think they only allowed JS/Python/PHP/Ruby the last time I used them).
And to be fair, other cloud providers are responsible for implementing their own C# IaaS stack. To use heroku as an example, they implement their own stack and add languages that they support at their own pace (I think they only allowed JS/Python/PHP/Ruby the last time I used them).
This article is about that "different slimmed down framework"; which is 24x faster than the previous full framework.
Easiest way to ease yourself into ASP.NET programming on a Mac is probably to use Xamarin Studio:
http://lastexitcode.com/blog/2016/01/01/DNXSupportInXamarinS...
It's free too (use the download link at the end of the page):
https://xamarin.com/studio
http://lastexitcode.com/blog/2016/01/01/DNXSupportInXamarinS...
It's free too (use the download link at the end of the page):
https://xamarin.com/studio
I don't know about ASP.NET Core, but I couldn't get .NET Core to work on my Mac. I installed the "official PKG." I then ran `dotnet restore` for a project I was trying to run (https://github.com/atemerev/skynet). It downloaded a couple hundred packages, then failed with an error like, "there is no run-time assembly compatible with osx.10.9-x64." I guess it requires a newer version of OS X, but neglects to check at install time.
I continue to use Mono and Xamarin Studio, although I'm stuck on an old version of Xamarin for now because I'm using Mavericks. At some point, I'd like to try OmniSharp with Emacs.
I continue to use Mono and Xamarin Studio, although I'm stuck on an old version of Xamarin for now because I'm using Mavericks. At some point, I'd like to try OmniSharp with Emacs.
It's on! Waiting patiently for the Techempower entry...
It's unfortunate that they don't have the numbers for Linux and instead used the Windows operating system. I'd like to see the Linux benchmark be on par with Windows (or ideally better) but I'm not sure if there is an interest from Microsoft to meet or exceed performance of Windows with a Linux based host.
Update: Checked with the team, latest Linux benchmark for plaintext with pipelining from January was 928,023 rps, which is a lot closer to the Windows numbers (78%). Updated here: https://github.com/aspnet/benchmarks/#plain-text-with-http-p...
Linux benchmarks are included here: https://github.com/aspnet/benchmarks/
The team definitely wants to make ASP.NET Core run fast on Linux, too. The ASP.NET team perf rig runs both Ubuntu 14.04 LTS and Windows Server 2012 R2. A lot of the perf enhancements are due to community pull requests, mostly from Ben (author of parent post). The perf work's been discussed a lot on the weekly community standup calls (http://live.asp.net), here's one going over the rig and perf in general: https://www.youtube.com/watch?v=7O81CAjmOXk&list=PL0M0zPgJ3H...
There's a lot going on right now - both ASP.NET Core and .NET Core are new to Linux, the teams are working on getting 1.0 releases out the door, etc. I expect perf on both Windows and Linux to continue to improve, both before and after 1.0 release. This week on the community standup, Damian said that perf is one area that's lower risk as RTM approaches, and that they're hoping for more improvement there.
And this is very much an open source project, as you can see from the merged pull requests in general (https://github.com/aspnet/KestrelHttpServer/pulls?q=is%3Apr+...) and the fact that Ben (non m-dollar) is battling with a Microsoft dev to be the top contributor. It'd be great to get some more devs with Linux HTTP serving background contributing.
I think I remember that they only posted benchmark updates when there was a significant change. Checking with the team now.
Source: long time ASP.NET groupie, Microsoft employee (until they find out I'm goofing off on HN instead of replying to e-mail).
Linux benchmarks are included here: https://github.com/aspnet/benchmarks/
The team definitely wants to make ASP.NET Core run fast on Linux, too. The ASP.NET team perf rig runs both Ubuntu 14.04 LTS and Windows Server 2012 R2. A lot of the perf enhancements are due to community pull requests, mostly from Ben (author of parent post). The perf work's been discussed a lot on the weekly community standup calls (http://live.asp.net), here's one going over the rig and perf in general: https://www.youtube.com/watch?v=7O81CAjmOXk&list=PL0M0zPgJ3H...
There's a lot going on right now - both ASP.NET Core and .NET Core are new to Linux, the teams are working on getting 1.0 releases out the door, etc. I expect perf on both Windows and Linux to continue to improve, both before and after 1.0 release. This week on the community standup, Damian said that perf is one area that's lower risk as RTM approaches, and that they're hoping for more improvement there.
And this is very much an open source project, as you can see from the merged pull requests in general (https://github.com/aspnet/KestrelHttpServer/pulls?q=is%3Apr+...) and the fact that Ben (non m-dollar) is battling with a Microsoft dev to be the top contributor. It'd be great to get some more devs with Linux HTTP serving background contributing.
I think I remember that they only posted benchmark updates when there was a significant change. Checking with the team now.
Source: long time ASP.NET groupie, Microsoft employee (until they find out I'm goofing off on HN instead of replying to e-mail).
Who in your opinion is the most responsible for the remarkably positive change in certain parts of Microsoft? (ASP.Net and Azure would be two examples of the new and improved Microsoft, Office would be an example of the old Microsoft mentality.)
Much of the openness seems to coincide with when Satya Nadella became CEO.
Do you think he has any interest in enhancing the programmability of Excel, for the first time in over a decade? (Serious question)
Desktop, no. Hosted version, maybe.
Linux was represented on this graph from the article:
https://d3renderer.azurewebsites.net/plaintext/v7?src=1
https://d3renderer.azurewebsites.net/plaintext/v7?src=1
It stops between Nov and Dec. Looks like they haven't tested those improvements on Linux or they're Windows specific.
The performance improvements that Ben Adams and other ASP.NET Core contributors have made haven't been Windows specific for the most part. Personally, I do most of my perf testing on Windows, but that doesn't mean Linux is completely left out.
Last month, I did some Linux profiling in our perf lab. As part of that, I ran the exact same benchmark shown in that graph on the exact same equipment and got over double the RPS numbers we got back in November much like we did on Windows.
I just updated the https://github.com/aspnet/benchmarks/ repo with the linux numbers I got last month since Ben's blog post has brought some increased attention to it. We plan to do a more complete benchmark run soon so we can update more numbers.
Last month, I did some Linux profiling in our perf lab. As part of that, I ran the exact same benchmark shown in that graph on the exact same equipment and got over double the RPS numbers we got back in November much like we did on Windows.
I just updated the https://github.com/aspnet/benchmarks/ repo with the linux numbers I got last month since Ben's blog post has brought some increased attention to it. We plan to do a more complete benchmark run soon so we can update more numbers.
Updated graph auto updated by merging this PR https://github.com/benaadams/d3renderer/pull/2
[deleted]
what's special in ASP.NET core that allows then to achieve this?
As I understand it, because it comes with less bloat than its predecessors (i.e. more focused library). From the blog post itself:
"ASP.NET Core and .NET Core come with the great advantage of only including the libraries and functions you explicitly want to use in your application rather than bringing in the entire framework. So you only “pay”, programmatically speaking, for what you use."Really, that's it? I thought it'd involve something about concurrency, networking or some IOCP magic going on.
The fastest code is the one that doesn't run. Joking aside, there have been numerous low level improvements with Ben making big contributions - thanks Ben!
Was it this bad before?
I still don't get why anyone would brag about how poorly they optimized something the first time around.
ASP.NET wasn't necessarily slow previously - for example, StackOverflow is running on it, and they're getting some pretty good perf (albeit with pretty beefy machines): http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...
Most sites, even relatively high traffic sites, don't serve 50k requests per second.
However, it wasn't granular or modular. For a lot of sites, it was a good balance of features to performance, but in a lot of cases you paid a performance penalty for features you weren't using. That includes the memory footprint penalty - each ASP.NET site loaded up a chunky system.web DLL with the kitchen sink. Back in the day that was fine, but as there are a lot of front-end heavy sites now that use servers mostly as API endpoints, and package managers are pretty standard, devs like to only pay the performance cost (including memory footprint) for the features, middleware, etc., that they're using. That was a big design consideration for ASP.NET Core, and you can really see the results in these specific benchmarks (Techempower plaintext).
Most sites, even relatively high traffic sites, don't serve 50k requests per second.
However, it wasn't granular or modular. For a lot of sites, it was a good balance of features to performance, but in a lot of cases you paid a performance penalty for features you weren't using. That includes the memory footprint penalty - each ASP.NET site loaded up a chunky system.web DLL with the kitchen sink. Back in the day that was fine, but as there are a lot of front-end heavy sites now that use servers mostly as API endpoints, and package managers are pretty standard, devs like to only pay the performance cost (including memory footprint) for the features, middleware, etc., that they're using. That was a big design consideration for ASP.NET Core, and you can really see the results in these specific benchmarks (Techempower plaintext).
Speaking of performance. I read that blog entry, and saw that they use their own DB library called Dapper:
https://github.com/StackExchange/dapper-dot-net
I was suprised to see that it's almost 4x faster than linq2sql and more than 10x faster than entity framework. (according to the table "Performance of SELECT mapping over 500 iterations - POCO serialization"). There must be a lot of slow sites out there running on entity.
https://github.com/StackExchange/dapper-dot-net
I was suprised to see that it's almost 4x faster than linq2sql and more than 10x faster than entity framework. (according to the table "Performance of SELECT mapping over 500 iterations - POCO serialization"). There must be a lot of slow sites out there running on entity.
They're different kinds of tools - maybe like comparing an RV to a motorcycle, or a guided missile to a sniper rifle. Or a...
Well, anyhow, while at a high level they're both tools that map between database commands and statically typed objects, the approach is very different.
Entity Framework is a full object relational mapper - it's generally the active record pattern, and it supplies things like migrations and entity tracking. The design philosophy is to abstract the database mechanics as much as possible, so you just work with collections of objects (e.g. DbSet<Person>) - add new Person instances to the set, set properties on them, etc. EF converts your intent into database commands (SQL queries in many cases, although EF7 supports some non-relational databases and in-memory storage, too). EF is tracking the state of all your objects, so it knows whether they need to be updated in the data store. That might be fine for a lot of cases - if you're managing a small list of products or customers, working on an intranet app, etc., you're not going to see a difference. I've seen (and fixed) a lot of horrible SQL queries that intranet devs built by concatenating strings; EF is at least going to usually give you decent, secure SQL. (If you're gnashing your teeth right now because EF crushed your dreams in the past, they've done some decent work on SQL generation lately, including some good stuff in EF7). It works pretty well on a lot of apps in lots of dev shops, and can be tuned to work well on big apps if you know what you're doing.
Dapper, and other "micro-ORMs" want to work closer to the metal. Dapper's tuned to taking the results of a database command (SQL query) and populate static objects. There's no entity tracking, state management, etc. Take results of a database query, stuff it into a bunch of object properties, walk away. Because of that, benchmarks showing the performance of select mapping over 500 iterations - POCO serialization are going to run a lot faster, just as (sorry, can't help myself) a motorcycle's going to beat an RV on a race down a tiny dirt trail.
So there may be some slow sites that are running on EF that would be faster on Dapper, but not without evaluating how they're interacting with the database, how they're doing database updates, etc.
Also, to be fair, you can disable a lot of features in EF (or other more fully featured ORMs) and get perf numbers pretty close to micro-ORM's like Dapper. See this comparison: https://github.com/StackExchange/dapper-dot-net/issues/246#i...
However, if you're turning off all the features and hand crafting SQL statements and working with untracked entities, you probably want a micro-ORM, anyways.
Well, anyhow, while at a high level they're both tools that map between database commands and statically typed objects, the approach is very different.
Entity Framework is a full object relational mapper - it's generally the active record pattern, and it supplies things like migrations and entity tracking. The design philosophy is to abstract the database mechanics as much as possible, so you just work with collections of objects (e.g. DbSet<Person>) - add new Person instances to the set, set properties on them, etc. EF converts your intent into database commands (SQL queries in many cases, although EF7 supports some non-relational databases and in-memory storage, too). EF is tracking the state of all your objects, so it knows whether they need to be updated in the data store. That might be fine for a lot of cases - if you're managing a small list of products or customers, working on an intranet app, etc., you're not going to see a difference. I've seen (and fixed) a lot of horrible SQL queries that intranet devs built by concatenating strings; EF is at least going to usually give you decent, secure SQL. (If you're gnashing your teeth right now because EF crushed your dreams in the past, they've done some decent work on SQL generation lately, including some good stuff in EF7). It works pretty well on a lot of apps in lots of dev shops, and can be tuned to work well on big apps if you know what you're doing.
Dapper, and other "micro-ORMs" want to work closer to the metal. Dapper's tuned to taking the results of a database command (SQL query) and populate static objects. There's no entity tracking, state management, etc. Take results of a database query, stuff it into a bunch of object properties, walk away. Because of that, benchmarks showing the performance of select mapping over 500 iterations - POCO serialization are going to run a lot faster, just as (sorry, can't help myself) a motorcycle's going to beat an RV on a race down a tiny dirt trail.
So there may be some slow sites that are running on EF that would be faster on Dapper, but not without evaluating how they're interacting with the database, how they're doing database updates, etc.
Also, to be fair, you can disable a lot of features in EF (or other more fully featured ORMs) and get perf numbers pretty close to micro-ORM's like Dapper. See this comparison: https://github.com/StackExchange/dapper-dot-net/issues/246#i...
However, if you're turning off all the features and hand crafting SQL statements and working with untracked entities, you probably want a micro-ORM, anyways.
Great explanation. If you want to get even closer to the metal there is Simple.Data
http://simplefx.org/simpledata/docs/
where you write queries like
var albums = Database.Open().Albums.FindAll(db.Albums.GenreId == 1);
//or FindAllByGenreId(1)
foreach (var album in albums)
{
Console.WriteLine(album.Title);
}
Because 1M req/s web severs are just flying about? I'm totally out of date on such thing but last I read we had Yaws built off Erlang and nginx at around 80k?
There must really be something I didn't understand here.
There must really be something I didn't understand here.
Some people deeply commit to micro-benchmarks. And sure, netty wins the micro-benchmark. But Java's concurrency model is harder to get right AND fast, so maybe it's a wash for real programs?
You can just be assured that ASP.NET is not going to suplex you with poor performance.
You can just be assured that ASP.NET is not going to suplex you with poor performance.
You have to remember though that part of this is making the core features of a new build of a runtime with new features get massive improvements to performance.
It's exciting for ASP.NET, but part of this is also to say, "The .NET Core platform is not a joke. In fact it's a good deploy target."
It's exciting for ASP.NET, but part of this is also to say, "The .NET Core platform is not a joke. In fact it's a good deploy target."
Not going to lie, i've had my pain. Dependencies are still hell, getting stripe to work meant taking an open source library, and modifying it... and there's still some stuff I want to use, but haven't gotten working yet.
That said the platform keeps getting better, and I love what i'm seeing. I feel like it combines a lot of what I loved, with a lot of what I learned to love in Ruby.
Plus Microsoft is offering so many sweet deals with Azure, its hard to pass it up.