NUMA Siloing in the FreeBSD Network Stack [pdf](people.freebsd.org)
people.freebsd.org
NUMA Siloing in the FreeBSD Network Stack [pdf]
https://people.freebsd.org/~gallatin/talks/euro2019.pdf
136 comments
> ... just shares potential competitive advantages like this?
Is it really a competitive advantage though? I would think that their brand and content library are the main ones. A CDN server doesn't really count IMHO: other streaming services can create their own appliances, though I'm guessing they'll use Linux due to its ubiquity.
Though as a long-time FreeBSD fan and part-time user, I'm happy for the patches. I run a large Isilon OneFS cluster, and so hopefully these changes will help in NFS performance. :)
There is another aspect to this contribution: the FreeBSD kernel code is ever-changing, so unless Netflix wants to carry forward this patch internally into the future, it is prudent to get it committed upstream so it can be cared for by the community at-large.
Is it really a competitive advantage though? I would think that their brand and content library are the main ones. A CDN server doesn't really count IMHO: other streaming services can create their own appliances, though I'm guessing they'll use Linux due to its ubiquity.
Though as a long-time FreeBSD fan and part-time user, I'm happy for the patches. I run a large Isilon OneFS cluster, and so hopefully these changes will help in NFS performance. :)
There is another aspect to this contribution: the FreeBSD kernel code is ever-changing, so unless Netflix wants to carry forward this patch internally into the future, it is prudent to get it committed upstream so it can be cared for by the community at-large.
> Is it really a competitive advantage though?
It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here).
Reducing the number of nodes helps with management of the network as well.
It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here).
Reducing the number of nodes helps with management of the network as well.
It's funny to think that pushing 196gbps is possibly just as little as ~8000 simultaneous streams.
Also a shame that it is TLS'd. The content already has its own DRM/encryption.
Also a shame that it is TLS'd. The content already has its own DRM/encryption.
TLS plays a different role. It makes sure you're receiving the Netflix's DRMed and encrypted content, and not some MITMed content that also happens to be correctly DRMed and encrypted, but contains something else.
Easy enough to make the content authenticate itself, instead of repeating cryptographic work for each viewer.
E.g. every 64k block contains a SHA256 hash of the next 64k block. When a user seeks, you provide the hash of the first block they'll be receiving over a trusted channel. (It already can exist at start of stream RSA signed or whatever).
But that's probably a pain when you have so many different devices to support.
(It also makes it harder to figure out what content people are watching... Though I think they use variable bitrate encoding which will fall to traffic analysis).
E.g. every 64k block contains a SHA256 hash of the next 64k block. When a user seeks, you provide the hash of the first block they'll be receiving over a trusted channel. (It already can exist at start of stream RSA signed or whatever).
But that's probably a pain when you have so many different devices to support.
(It also makes it harder to figure out what content people are watching... Though I think they use variable bitrate encoding which will fall to traffic analysis).
This exists in a more advanced form already
https://wicg.github.io/webpackage/draft-yasskin-http-origin-...
https://wicg.github.io/webpackage/draft-yasskin-http-origin-...
No... it's not the same as signed HTTP exchanges, which you'd generally use inside TLS, and involves signing all of the data in an exchange. It wouldn't work very well for streaming, either.
The important feature of the scheme that I mentioned is that there's no per-client encryption needed, except maybe a small operation on each client seek in the stream. In general, you just serve the file, and the file contents are self-authenticating against tampering.
You might use signed HTTP exchanges to push the first hash in the chain on a seek, I guess. Or just run it over TLS. Or just have a 64k block pre-signed at the beginning stream with the hashes of 2k chosen seek points that the client can store.
The important feature of the scheme that I mentioned is that there's no per-client encryption needed, except maybe a small operation on each client seek in the stream. In general, you just serve the file, and the file contents are self-authenticating against tampering.
You might use signed HTTP exchanges to push the first hash in the chain on a seek, I guess. Or just run it over TLS. Or just have a 64k block pre-signed at the beginning stream with the hashes of 2k chosen seek points that the client can store.
TLS keeps your viewing history private from third parties (e.g. your ISP).
What’s the problem with TLS? The performance overhead of it is minimal these days.
It depends on the application; but for bulk transit of already encrypted (presumably by DRM) data, it is a significant overhead for little value.
You can go back and look at Netflix presentations from when they switched to TLS, and they were no longer able to saturate their network cards (and here, they're at 50% on network still)
You can go back and look at Netflix presentations from when they switched to TLS, and they were no longer able to saturate their network cards (and here, they're at 50% on network still)
Yes, but the parent was talking sharing the competitive advantage of the patch.
But (a) what other streaming services even have CDN appliances, and (b) of the ones that do, which are using FreeBSD as a base?
If none of Netflix's competitors will use this patch, then it makes no material difference whether they share it or not.
But (a) what other streaming services even have CDN appliances, and (b) of the ones that do, which are using FreeBSD as a base?
If none of Netflix's competitors will use this patch, then it makes no material difference whether they share it or not.
A) at least FB and Google, probably Amazon too, plus you know, lots of CDNs have CDN appliances.
B) as far as I'm aware, everyone else is running Linux, except maybe one of the CDNs, but I don't know if that CDN is actually running appliances.
However, everyone and their uncle are launching a streaming service in the next 18 months, some of those might get enough traffic to justify appliances, and they can take Netflix's work here to get a competitive box off the bat.
Also, at any point, one of the current big guys can say "hey, netflix gets 200gbps from one box, why do we have 4? Why don't we try FreeBSD on some of these"
CDN appliances are pretty isolated, so it doesn't matter as much if most of their proprietary stack isn't ported. Chances are, they've got someone on staff with FreeBSD experience, and FreeBSD experience ages better, because there's less churn.
B) as far as I'm aware, everyone else is running Linux, except maybe one of the CDNs, but I don't know if that CDN is actually running appliances.
However, everyone and their uncle are launching a streaming service in the next 18 months, some of those might get enough traffic to justify appliances, and they can take Netflix's work here to get a competitive box off the bat.
Also, at any point, one of the current big guys can say "hey, netflix gets 200gbps from one box, why do we have 4? Why don't we try FreeBSD on some of these"
CDN appliances are pretty isolated, so it doesn't matter as much if most of their proprietary stack isn't ported. Chances are, they've got someone on staff with FreeBSD experience, and FreeBSD experience ages better, because there's less churn.
I would say that has less to do with Netflix and more the recent generation of Intel/EPYC server computing. These numbers seem pretty standard for this generation of Skylake/Cascade Lake and one NIC per NUMA also feels pretty standard to me. The patches also seem to be the usual problem when you are starting to care about NUMA, where there are a couple places where it just wasn't added but you need it.
Agreed.
I believe technology as a whole is light-years ahead of where it would be if companies treated all of their findings and system designs as closely-guarded secrets. I am relatively early in my career and I get to open-source a project that I have been working on for the first time and it feels great.
I believe technology as a whole is light-years ahead of where it would be if companies treated all of their findings and system designs as closely-guarded secrets. I am relatively early in my career and I get to open-source a project that I have been working on for the first time and it feels great.
Just imagine how far ahead the other industries would be if we handled patent and trademark law like we do in the software industry.
I am actually not too sure whether it matters. The amount of people that have the skills to implement such a solution is not that high. There are are only so many people who can do high performance kernel, encryption and networking work.
They might give the advantages away if they release everything, so that others can simply deploy it or offer it as a hosted solution. But the pure documentation what they did are more like advertisements for interesting engineer work @ Netflix.
They might give the advantages away if they release everything, so that others can simply deploy it or offer it as a hosted solution. But the pure documentation what they did are more like advertisements for interesting engineer work @ Netflix.
I like it, I think its spreading, and I think its a good thing for the world.
"Top U.S. CEOs say companies should put social responsibility above profit" https://www.reuters.com/article/us-jp-morgan-business-roundt...
"Top U.S. CEOs say companies should put social responsibility above profit" https://www.reuters.com/article/us-jp-morgan-business-roundt...
The part that worries me is that companies are increasingly seeing it as their social responsibility to gang up and destroy "bad" people like Stallman.
Stallman has always been a neckbeard and either the industry grows up and shows some responsibility (which means kicking out or massively restrain the neckbeards) or it's going to be held liable itself for their actions.
Social responsibility over profit is not legal in the USA. Fiduciary duty, EBay v Newmark
> I get that, in part, this is a way to get people excited to work for Netflix
It's even more amazing to me that this is a successful tactic in light of the fact that there's no shortage of people willing to work for ad-tech firms. If working for a company with good values was such a draw, you'd think it would give them more of a competitive advantage. Maybe the real reason is that Netflix's "good values" are genuine, not just a draw for employees. I'm really not sure.
It's even more amazing to me that this is a successful tactic in light of the fact that there's no shortage of people willing to work for ad-tech firms. If working for a company with good values was such a draw, you'd think it would give them more of a competitive advantage. Maybe the real reason is that Netflix's "good values" are genuine, not just a draw for employees. I'm really not sure.
The whole ad tech space as a data nerd is super enticing. I could work on some really cool data warehouse / database issues but at the end of the day I’d be helping an ad tech company...
Worse yet is I’ve met some really cool people in the space but it is still ad tech
Worse yet is I’ve met some really cool people in the space but it is still ad tech
I don't see how netflix' "good values" are so much better than adtech. They're not exactly saving the planet, feeding the starving or growing artificial organs. Adtech is about getting eyeballs and converting that into money. Netflix is about getting eyeballs for hours and people paying for that privilege. Contributing to OSS projects they use is sort of the "not eating babies" baseline, even oracle and google do it!
Netflix doesn't involve stalking people online and then selling the data you derive to the highest bidder.
They also have their own CDN, to reduce load on ISPs.
https://media.netflix.com/en/company-blog/how-netflix-works-...
https://media.netflix.com/en/company-blog/how-netflix-works-...
Specifically, the CDN part is where Netflix uses FreeBSD. Their control plane runs on Linux.
you make it sound altruistic. it’s to avoid paying ISPs
It's to avoid paying last mile ISPs who have turned their monopolies into a mafia-like shakedown scheme.
Those last mile ISPs have to install one of these boxes to see a benefit.
Netflix still has to work things out and get the last mile ISPs to play ball ($$$).
But this potentially saves Netflix a whole ton on transit.
Netflix still has to work things out and get the last mile ISPs to play ball ($$$).
But this potentially saves Netflix a whole ton on transit.
>just shares potential competitive advantages like this?
It is abit like Facebook Sharing their work on Servers Hardware and Database improvements. Ultimately they dont think any of those technology are their completive advantage , their core business is Social ( and Ad Tracking, but that is another topic ).
Similar to Netflix, and would be the same as Disney. Their core business is not in the delivery, but the content.
Having said that it is still very nice of them to share their work. FreeBSD, Ruby Rails, Chaos Engineering, there are lots of things I love about their Tech Stacks.
It is abit like Facebook Sharing their work on Servers Hardware and Database improvements. Ultimately they dont think any of those technology are their completive advantage , their core business is Social ( and Ad Tracking, but that is another topic ).
Similar to Netflix, and would be the same as Disney. Their core business is not in the delivery, but the content.
Having said that it is still very nice of them to share their work. FreeBSD, Ruby Rails, Chaos Engineering, there are lots of things I love about their Tech Stacks.
Perhaps it's just because researchers need to publish in order to, you know, build their careers.
Realistically, Disney, or whomever competition, is not going to have enough technical expertise to deploy FreeBSD.
The BSD logo was created by one of Pixar's founders[1], and Disney now owns Pixar. Disney also has money. I'm sure deploying FreeBSD is well within their reach.
1) https://www.freebsd.org/copyright/daemon.html
1) https://www.freebsd.org/copyright/daemon.html
... You don't think Disney, a company with a revenue of roughly 60 billion dollars, can hire people with enough technical expertise to deploy FreeBSD?
Disney’s animation studio is based on Linux infrastructure: https://www.velvetjobs.com/job-posting/sr-systems-engineer-l...
I trained the entire Disney offshore replacement team. They understand FreeBSD very well.
Out of curiosity, does this mean that they’re using (or have used) FreeBSD in their infrastructure or for some other purpose, or just a high level of Unix competence in that team? Any interesting details that you’re able to share?
Author here: The talk will be on Youtube eventually, and a lot of points are explained in more detail in the actual talk. I was just going to bed in advance of traveling back to the states tomorrow, but I'll try to answer any question in the morning.
Do you think these kinds of results can be achieved through improved schedulers and allocators?
Why is the hw.model on EPYC redacted, when the chips are listed in the slides? Secret engineering sample? :)
He said in the talk that yes, the chips used in these measurements were not publicly released. IIRC he said that they were slightly lower clock speed than the publicly available chips.
[deleted]
Nice to see AMD replacing Intel, they've gone with EPYC 7551 & 7502P, from 2x Intel “Skylake” / “Cascade Lake” Xeon
edit: it looks like they hit 200GB/s with both Intel and AMD
edit: it looks like they hit 200GB/s with both Intel and AMD
Yes, we haven't gone with anything. These are proof of concept prototype systems. They serve real customer traffic in my testing, but are unicorns at this point.
It's disappointing to hear that AMD has not provided adequate performance monitoring tools though. I would have thought by the second generation of Zen that this would be an area that AMD had given attention to.
AMD's engineering team is much smaller, and they've always been behind on performance tools. HPC people have been complaining ever since Opteron. It's probably not going to change.
I wonder how this would compare with Receive Side Scaling (RSS), which you can use to pin NIC queues to cpus, and then pin the rest of the handling to the same cpu, avoiding a significant amount of interprocess communication. NUMA concerns may be more important than CPU pining within a domain though.
RSS is more about spreading a workload over multiple cores to not overwhelm any single core, particularly in high PPS situations.
This is about PCI-E devices being hooked up to a NUMA node to avoid saturating the link between nodes. There's a limited amount of bandwidth, and crossing nodes saturates this and increases latency, both of which will have limiting effects on your total possible throughput.
RSS configuration doesn't need you to set up your hardware in any specific way, with this you need to ensure that the set of disks and nic are hooked up to coupled domains - e.g. if you place the two NICs on the same NUMA node, then no software configuration is going to fix that, and you'd have to go and physically rearrange things to fix it.
You might still use RSS to distributed the workload across multiple cores within that NUMA domain when using this setup.
This is about PCI-E devices being hooked up to a NUMA node to avoid saturating the link between nodes. There's a limited amount of bandwidth, and crossing nodes saturates this and increases latency, both of which will have limiting effects on your total possible throughput.
RSS configuration doesn't need you to set up your hardware in any specific way, with this you need to ensure that the set of disks and nic are hooked up to coupled domains - e.g. if you place the two NICs on the same NUMA node, then no software configuration is going to fix that, and you'd have to go and physically rearrange things to fix it.
You might still use RSS to distributed the workload across multiple cores within that NUMA domain when using this setup.
If they are just serving/streaming static data from a single server, wouldn't an entirely different architecture make more sense? For example, why even use a CPU? They have the financial resources to build their own hardware, e.g. encryption ASICs, and they can do interesting things like bundle the pipeline for multiple viewers watching the same movie at the same timepoint.
Commodity server gear is really cheap, and is more or less "fast enough". The wins from serving say, >1Tbps from a 1U box with an ASIC or FPGA is negated by the downsides of having such a large failure domain, and the cost of development for what would be a pretty low volume part.
We talk about this stuff every once in a while, but it doesn't really make sense to do right now. (Disclaimer: I work at Netflix)
We talk about this stuff every once in a while, but it doesn't really make sense to do right now. (Disclaimer: I work at Netflix)
Is it still accurate that for network bound servers that FreeBSD still outperforms Linux?
Author here.. It depends heavily on the workload. We have man years of optimizations for our CDN workload that we have been upstreaming to FreeBSD. See, for example, the other talk I co-presented regarding Kernel TLS on FreeBSD.
https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf
https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf
What made Netflix pick FreeBSD over Linux in the first place?
It was mentioned in the QnA section of a talk posted somewhere in this comment section (probably https://www.youtube.com/watch?v=vcyQBup-Gto).
At least originally, it was because of the license.
At least originally, it was because of the license.
BSDs unwavering commitment to reliability. A Linux kernel crashes. A lot. FreeBSD ran cdrom.com for 365+ days without rebooting. Just saying
What's "a lot" here, and are you sure that's still true? Because my suspicion is that it might have been true before FAANG sunk billons of R&D into Linux, but I would be shocked if Linux kernels cash at any rare worth noticing today. Although it's splitting hairs, really; when is the last time you saw any major kernel crash?
Every year I still see a Linux system or two live-lock. When I was still doing Solaris and FreeBSD admining that never happened.
However most systems aren't pushed to extremes, so these corner cases aren't seen very often, regardless of OS.
However most systems aren't pushed to extremes, so these corner cases aren't seen very often, regardless of OS.
Reminder: Facebook tried to run the WhatsApp stack on Linux but they kept having kernel panics
I've got Linux servers that have gone that long without rebooting (and some that are much longer!), but I generally view it as a bad thing and a failing on my part, and would feel the exact same way if I was still running illumos and freebsd servers. Software updates happen for a reason, even on BSD, and servers running for that long mean that they aren't getting those updates.
Stock, probably not. FreeBSD is a nicer platform to modify for your use case though.
I think it's safe to say any kind of sweeping generalization like this is unlikely to be broadly true (in either direction). Both work quite well.
I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.
Author here.. We're doing ~200Gb/s through the kernel.
That's why I carefully inserted my weasel words of "I think" ;)
Thank you for your contributions. Great information.
Thank you for your contributions. Great information.
Thanks! I often do the same :)
Netflix is not bypassing the FreeBSD kernel. They bypass userland for bulk data though --- with sendfile, and kTLS, their webserver (nginx) has to negotiate TLS and parse requests and open files, but then it's pretty simple.
If it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data.
Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection setup, DMA orchestration and occasional TLS renegotiation have to be handled.
Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection setup, DMA orchestration and occasional TLS renegotiation have to be handled.
I know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)?
AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it would be funny to use it to mine bitcoin, since it supports SHA256. AMD — Advanced Mining Devices!)
AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it would be funny to use it to mine bitcoin, since it supports SHA256. AMD — Advanced Mining Devices!)
Intel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant.
The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM, etc, and also used internally by the CPU to generate RDRAND/RDSEED data. It was probably never intended to be used by OS drivers and certainly not intended to be any kind of accelerator.
The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM, etc, and also used internally by the CPU to generate RDRAND/RDSEED data. It was probably never intended to be used by OS drivers and certainly not intended to be any kind of accelerator.
Supposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.
By chipset, you mean northbridge? Or the CPU?
The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...):
https://www.intel.com/content/www/us/en/wireless-network/acc...
The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...):
https://www.intel.com/content/www/us/en/wireless-network/acc...
The northbridge. my understanding is that they no longer sell the discrete cards to perform these tasks, and instead offload it to chips that come on the boards.
[deleted]
That might be true, but you can still buy the cards from 3rd party resellers, e.g., https://www.newegg.com/p/2AS-006R-00046 .
Yeah, sorry, I misremembered. The middle device of the chain in the motivating example given for P2P-DMA was for compression, not encryption.
https://www.snia.org/sites/default/files/SDC/2018/presentati...
https://www.snia.org/sites/default/files/SDC/2018/presentati...
From what I recall, the chelsio cards only support a mode of encryption suitable for storage devices, and it's not something you'd use for streaming media.
No, not true. The Chelsio card support GCM and CBC crypto in lookaside (like QAT) using the ccr(4) OCF driver, inline ("NIC TLS") with out - of - tree patches, and TLS offload in TOE mode.
Now that ktls is upstream, we are looking at using the ccr crypto acceleration. We've already tested them in inline mode. TOE is not an option for us, since we do innovation in the TCP stack.
Now that ktls is upstream, we are looking at using the ccr crypto acceleration. We've already tested them in inline mode. TOE is not an option for us, since we do innovation in the TCP stack.
Thanks for clearing that up.
In the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.
I think he's talking about the general case (and since XDP is in the discussion, probably the general case for Linux) and not trying to speak specifically to what Netflix is doing for their CDN.
It depends on how exactly each is tuned. Cache and buffer sizes, drivers, TCP congestion control algorithms, and more can affect this hugely, and are different. The defaults on linux (and probably on freebsd) also change reasonably often, so you can expect linux 4.x to have singificant differences from 5.x.
As I understood Linux made a lot of progress in the last couple of years with say zero-copy patches, etc. I thought they reached some kind of parity.
AFAIK Linux is actually more performant these days, network-wise.
The Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future.
On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.
On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.
Why not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.
We have!
https://docs.keydb.dev/blog/2019/06/17/blog-post/
Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.
https://docs.keydb.dev/blog/2019/06/17/blog-post/
Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.
Have you tried reaching out to the linux netdev mailing list about this? They may at least be able to explain their design decision.
Can you move all softirqs into different cores? That's usually one of the dpdk tuning steps.
My reading of the source code is there is no way to prevent this. The spinlock is guarding the check which determines if there is work to do.
Its been a few months and I had intended to go back and try using a try_lock(). But I’m not normally a kernel dev.
Its been a few months and I had intended to go back and try using a try_lock(). But I’m not normally a kernel dev.
> I really hope it gets this kind of love in the future.
Have you considered using FreeBSD? ;)
Have you considered using FreeBSD? ;)
Unfortunately the vast majority of KeyDB users run Linux.
Have you tried optimising and benchmarking on a 2.6 kernel? I found the more recent kennels in rhel7 that the amount of kernel work being done very hard to control. The first core on any numa node is especially problematic and induce horrific jitter
This may be a dumb question, but why would streaming video need to be encrypted? Is this just part of the "encrypt everything" best practice these days? Is there metadata accompanying the video data that shouldn't be unencrypted? Is it just so it's not possible to eavesdrop on the fact that I'm watching "Marvelous Mrs. Maisel"?
The main reason is probably DRM.
Well, no. You talking about HDCP. In this case it's just TLS.
There is a lot of reasons why everything should be behind TLS.
There is a lot of reasons why everything should be behind TLS.
[deleted]
It’s for privacy. That’s all you need to know.
Also to prevent the video stream from being modified / replaced in transit.
I think Netflix views its user-consumption data as a competitive advantage, and preventing 3rd parties from finding out what people watch, when, etc, helps protect that advantage.
So why is it important to have a multisocket NUMA machine? Why not just save yourself a lot of hassle by having one socket? I know that the previous generation AMD machine had unavoidable NUMA but the new one doesn't.
I've run this particular AMD system in 3 ways:
- Non-NUMA
- 2 nodes per socket
- 4 nodes per socket
The 4NPS gives the best performance, followed by 2NPS, followed by non-NUMA. This surprised me as well.
The 4NPS gives the best performance, followed by 2NPS, followed by non-NUMA. This surprised me as well.
Communication over DDR4 is way faster than communication over PCIe Ethernet. Even 40Gbit is slow compared to RAM.
Maybe but this post is about making the two sides of the computer NOT communicate.
For the typical case, yeah, you don't want communication.
But if CPU#1 wants to access a file that is on CPU#2 NVMe nodes, NUMA allows you to share those files across memory (and its a "local" file according to the OS), instead of over NFS or SMB.
--------
And yes, as much as we like to pretend that there's no communication and everything scales horizontally... in practice... people like sharing files between systems. NUMA allows for these files (and other resources: such as PCIe network cards or GPUs) to be shared between systems at the speed of DDR4 memory.
But if CPU#1 wants to access a file that is on CPU#2 NVMe nodes, NUMA allows you to share those files across memory (and its a "local" file according to the OS), instead of over NFS or SMB.
--------
And yes, as much as we like to pretend that there's no communication and everything scales horizontally... in practice... people like sharing files between systems. NUMA allows for these files (and other resources: such as PCIe network cards or GPUs) to be shared between systems at the speed of DDR4 memory.
Something something performant concurrency
It's money. Having a two-socket machine instead 2 one-socket machines, even when performance is only 80%, is saving both OPex and CAPex.
One bigger host meams less management overhead. (see other comments about being u willing to run multiple ips and what not)
It may be a bit early to have a presentation from the latest Epyc processors. Most of the work was likely done with previous processors, but their slides said their AMD boxes are single socket.
It may be a bit early to have a presentation from the latest Epyc processors. Most of the work was likely done with previous processors, but their slides said their AMD boxes are single socket.
Density/power/cooling are real constraints when you aren't 100% bought-in to the cloud.
This talk is about Zen+ Epyc, not Zen2 (which is where the non-cache memory gets uniform). I don't know if they have release quality Epyc 7003 (Zen2) samples available yet, and if they do, NFLX probably isn't allowed to publish benchmarks about them. There's almost certainly still some value in their existing NUMA work even on Zen2, as things like L1/L2/L3 cache have locality even if memory and PCIe does not.
Pretty sure Intel single socket of this generation is totally non-viable for this workload due to lack of PCIe lanes. Maybe viable when Intel gets gen4 PCIe.
Pretty sure Intel single socket of this generation is totally non-viable for this workload due to lack of PCIe lanes. Maybe viable when Intel gets gen4 PCIe.
Skylake-X has 44 PCI 3.0 lanes, that's 352GT/s or about 345gbps application bandwidth. It's certainly more than enough to push 100gbps from disk to net. These guys are pushing 200gbps, but they're doing it with two CPUs, two sets of NVMe devices, and two NICs, and a bunch of hacks to make the operating system pretend all this stuff is not in the same box. It seems way more straight-forward to me if they had made it all be actually NOT in the same box!
> Skylake-X has 44 PCI 3.0 lanes, that's 352GT/s or about 345gbps application bandwidth. It's certainly more than enough to push 100gbps from disk to net. These guys are pushing 200gbps
We're in total agreement :-). Their dataflow model requires something like 2x that in PCIe bandwidth and 4x in memory in the optimal case, as covered in the slides. 2x200 gbps = 400 gbps, which is a bit more than 345 gbps.
Maybe they could push 345/2 = 172 Gbps out of a single Skylake-X, best case. For some workloads, that might be the right local optima! They must have decided that the marginal cost of a 2P system was worth the extra ~25 Gbps to saturate the 200 Gbps pipe fully.
> they're doing it with two CPUs, two sets of NVMe devices, and two NICs, and a bunch of hacks to make the operating system pretend all this stuff is not in the same box. It seems way more straight-forward to me if they had made it all be actually NOT in the same box!
I've spoken with NFLX engineers in the past and my recollection is that in many installations, NFLX only get to install one box. (Or something like that. Might just be a cost thing.) So they need to make that one box fast.
I guess the other factor is the IP management overhead discussed in the slides. Two boxes necessitates the costly 2nd IP, as far as I know. It's hard to imagine the cost of an IP address dominating the marginal cost of a 2P socket system and 2nd Xeon, but I guess AWS is friggin expensive.
We're in total agreement :-). Their dataflow model requires something like 2x that in PCIe bandwidth and 4x in memory in the optimal case, as covered in the slides. 2x200 gbps = 400 gbps, which is a bit more than 345 gbps.
Maybe they could push 345/2 = 172 Gbps out of a single Skylake-X, best case. For some workloads, that might be the right local optima! They must have decided that the marginal cost of a 2P system was worth the extra ~25 Gbps to saturate the 200 Gbps pipe fully.
> they're doing it with two CPUs, two sets of NVMe devices, and two NICs, and a bunch of hacks to make the operating system pretend all this stuff is not in the same box. It seems way more straight-forward to me if they had made it all be actually NOT in the same box!
I've spoken with NFLX engineers in the past and my recollection is that in many installations, NFLX only get to install one box. (Or something like that. Might just be a cost thing.) So they need to make that one box fast.
I guess the other factor is the IP management overhead discussed in the slides. Two boxes necessitates the costly 2nd IP, as far as I know. It's hard to imagine the cost of an IP address dominating the marginal cost of a 2P socket system and 2nd Xeon, but I guess AWS is friggin expensive.
I assumed these were cache servers that went into ISPs, does anyone know what they mean by "Increases AWS cloud management overhead" on a couple of the slides?
Guessing that they have managed services in aws for maintaining all the nodes and where they are. If each box has 2x the ips, they would need more mgmt resources to keep track of what is on that box, if it's online, stats, etc.
What a progress! In 2015 they were serving only ~9gb/sec https per server: https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf
Are those changes in 12.1? If not, what would be a time frame?
Not all of them are landed in CURRENT yet, much less stable/12. They'll be in 13.0; I can't speak to any future stable 12.2. On the other hand, CURRENT is pretty solid. A lot of folks, Netflix included, just run CURRENT.
Wait, they run CURRENT in production? Is that... safe?
There's at least one video online [1] that talks about Netflix's process for internal FreeBSD releases. They do 5 weeks of development and then 5 weeks of testing.
[1] https://www.youtube.com/watch?v=vcyQBup-Gto (about the 12 minute mark)
[1] https://www.youtube.com/watch?v=vcyQBup-Gto (about the 12 minute mark)
In this particular (Netflix) case, what's the worst that could happen? One of their OCAs panics and falls over... and all customer traffic gets shifted to another OCA in the same or another location?
Sure, they might run into some bug in -CURRENT, but what's the chance of a bunch of their OCAs all hitting it at the same time?
Sure, they might run into some bug in -CURRENT, but what's the chance of a bunch of their OCAs all hitting it at the same time?
I've been running CURRENT in production myself for about 2 years now. No problems so far.
If you have kernel developers on your staff: yes
Yes.
It’s my understanding that many organizations prefer *BSDs over Linux not because BSD is more performant ... but because you’re way more likely to get your organizations patch accepted into the upstream than Linux.
I doubt that is the case. The FreeBSD guys can be just as hardass about submissions as the Linux folks. However, there are considerations:
1) GPL vs BSD licenses. Companies like BSD licenses much more than GPL licenses. GPL adherents can whine all they want, but this is simply true.
2) BSD has a long history of having very good networking stacks--albeit on specific hardware. Linux supported everything initially--including really cheap crap--and consequently its networking stack was a lot more ad hoc. FreeBSD chose specific hardware for stability--but then supported that much more completely.
3) FreeBSD has a long history of being the servers in Internet infrastructure. There are specific architectural choices in the kernel because of this. There is probably still some inertia, too, in that the kind of old guard people who REALLY grok networking are still more comfortable on FreeBSD machines.
Consequently, it is hardly surprising that an advanced networking development would take place on FreeBSD.
1) GPL vs BSD licenses. Companies like BSD licenses much more than GPL licenses. GPL adherents can whine all they want, but this is simply true.
2) BSD has a long history of having very good networking stacks--albeit on specific hardware. Linux supported everything initially--including really cheap crap--and consequently its networking stack was a lot more ad hoc. FreeBSD chose specific hardware for stability--but then supported that much more completely.
3) FreeBSD has a long history of being the servers in Internet infrastructure. There are specific architectural choices in the kernel because of this. There is probably still some inertia, too, in that the kind of old guard people who REALLY grok networking are still more comfortable on FreeBSD machines.
Consequently, it is hardly surprising that an advanced networking development would take place on FreeBSD.
Re: 1 and 2.
Given that there are vendors that use FreeBSD for their appliances, they really don't want to have to send out techs to customers sites to fix things. So when the appliance makers choose hardware, they talk to component vendors about quality.
It's no surprise that you see commits from Intel and Chelsio employees in the FreeBSD logs: companies like Netflix, Isilon, NetApp, and Juniper partner with them to make sure things aren't buggy.
These collaborations lead to point 3.
Given that there are vendors that use FreeBSD for their appliances, they really don't want to have to send out techs to customers sites to fix things. So when the appliance makers choose hardware, they talk to component vendors about quality.
It's no surprise that you see commits from Intel and Chelsio employees in the FreeBSD logs: companies like Netflix, Isilon, NetApp, and Juniper partner with them to make sure things aren't buggy.
These collaborations lead to point 3.
I get that, in part, this is a way to get people excited to work for Netflix, however, the people working on this are, probably, pretty proud of what they achieved and thus like to share it with us, their colleagues at other places of work and they have the freedom to do so because of our industry's great track record of knowledge sharing.