Researchers Map Locations of Servers in Netflix’s Content Delivery Network(spectrum.ieee.org)
spectrum.ieee.org
Researchers Map Locations of Servers in Netflix’s Content Delivery Network
http://spectrum.ieee.org/tech-talk/telecom/internet/researchers-map-locations-of-4669-servers-in-netflixs-content-delivery-network
36 comments
In a similar vein there's a project at the moment to map the world's internet infrastructure called The New Cloud Atlas (http://newcloudatlas.org) which describes itself thus:
> The New Cloud Atlas is a global effort to map each data place that makes up the cloud in an open and accountable way. We have set out to find and map each warehouse data centre, each internet exchange, each connecting cable and switch. Anything of any physical significance in the operation of the cloud should be observed is some way, and recorded for everyone to see and use.
> The New Cloud Atlas is a global effort to map each data place that makes up the cloud in an open and accountable way. We have set out to find and map each warehouse data centre, each internet exchange, each connecting cable and switch. Anything of any physical significance in the operation of the cloud should be observed is some way, and recorded for everyone to see and use.
I have a friend that runs IT for a large college campus in the Midwest. They not only have a Netflix cache, but an Apple one as well.
The Netflix one runs at a few gigabit/sec at all times. The Apple one keeps the WAN from going down on update day.
The Netflix one runs at a few gigabit/sec at all times. The Apple one keeps the WAN from going down on update day.
What cache server do they use?
Netflix provides their own - https://openconnect.netflix.com/en/
I would imagine Apple provides something similar.
I would imagine Apple provides something similar.
From what I've been told, it's a rack-mounted unit that's sealed. That's about it. The Apple unit contains iTunes content as well as iOS/OSX system updates.
Honestly, why that's research worthy? That's not a new knowledge, that's information is somewhat known to all employees, partners, various NMS vendors, ISPs. 'Nice' graph and latex formatted article doesn't make that scientific research. I guess there is to much useless "since" in style of 'I run ping/traceroute from the couple of locations and here is my scientific research'.
It's not about "ping and traceroute"; it's about organizing and mapping the data (both logically and geographically) that come out tools like ping and traceroute.
They aren't saying they've demonstrated quantum teleportation, or anything. But for the subject matter at hand (and given that Netflix runs one of the largest CDNs in the world), it's quite impressive.
They aren't saying they've demonstrated quantum teleportation, or anything. But for the subject matter at hand (and given that Netflix runs one of the largest CDNs in the world), it's quite impressive.
I thought that the "Netflix CDN" was AWS.
Hasn't been for a while, the egress would be insanely expensive.
They use devices they call "Openconnect" for peering in ISPs https://openconnect.netflix.com/en/
They use devices they call "Openconnect" for peering in ISPs https://openconnect.netflix.com/en/
Thanks for clarifying. Very informative.
It sounds like Netflix's host naming scheme is what made it possible for the researchers to do this. I assume that means using regions and incrementing counters in the hostnames.
Who wants to bet that in the next year we'll see their hostnames change to be random-looking instead?
Who wants to bet that in the next year we'll see their hostnames change to be random-looking instead?
This won't happen. It would cause havoc when attempting to work with peering partners and transit providers when troubleshooting.
Can anyone provide more information on the server infrastructure than this article:
http://www.techhive.com/article/2158040/how-netflix-streams-...
It claims each server serves up to 10,000 streams simultaneously. So operational peak capacity is at ~50M simultaneous streams.
Also, 10k streams at 1080p would be ~70Gbps per machine at full HD, or ~20Gbps per machine at SD. With some blend of course being the norm.
There is of course a big difference between being capable of and actual and FUD.
I'd really love to know what the internals of one of their streaming servers might look like.
Anyone here who has architected a box with that capacity?
http://www.techhive.com/article/2158040/how-netflix-streams-...
It claims each server serves up to 10,000 streams simultaneously. So operational peak capacity is at ~50M simultaneous streams.
Also, 10k streams at 1080p would be ~70Gbps per machine at full HD, or ~20Gbps per machine at SD. With some blend of course being the norm.
There is of course a big difference between being capable of and actual and FUD.
I'd really love to know what the internals of one of their streaming servers might look like.
Anyone here who has architected a box with that capacity?
I recall seeing presentations at various FreeBSD events from Netflix folks talking about them, so a search for something like that might find what you're looking for.
If memory serves... SuperMicro motherboards, multiple Mellanox/Chelsio 10 Gbps NICs (probably upgraded to 40 Gbps by now, or getting close to that point), a bunch (up to 28, perhaps?) of hard drives (with no/little redundancy for thecontent), and a ton of RAM (for ZFS ARC).
They run on (heavily tuned) FreeBSD operating system.
If memory serves... SuperMicro motherboards, multiple Mellanox/Chelsio 10 Gbps NICs (probably upgraded to 40 Gbps by now, or getting close to that point), a bunch (up to 28, perhaps?) of hard drives (with no/little redundancy for thecontent), and a ton of RAM (for ZFS ARC).
They run on (heavily tuned) FreeBSD operating system.
Someone pointed out the original paper, which has better estimates than my back of the napkin calculation for traffic throughput. Good stuff.
Still would love to know more about the internals of their servers and support infrastructure.
Still would love to know more about the internals of their servers and support infrastructure.
https://openconnect.netflix.com/en/hardware/
I believe there are some slight differences between what Netflix uses and the OpenConnect machines placed inside ISP networks, but not by very much.
I believe there are some slight differences between what Netflix uses and the OpenConnect machines placed inside ISP networks, but not by very much.
Awesome thanks!
Zero severs in Russia, China, Turkey, India..? Over 2 bn. people left out of the market? I know not all have network access, but still. Not Russia or India at least?
https://torrentfreak.com/vpn-provider-pia-exits-russia-serve...
Russia is not a friendly country to do business in
China is closed off behind the Great Firewall and probably already has a local netflix clone
India, I do not know, could be cost issues?
Russia is not a friendly country to do business in
China is closed off behind the Great Firewall and probably already has a local netflix clone
India, I do not know, could be cost issues?
India could be a tax issue.
As a tech sector APAC manager I have been instructed more than once not to visit. Not for any internal conference, and definitely not for any customer contact, and to take verifiable steps to avoid making "management decisions" if visiting for a holiday. Typical quote, "You do not want to be the guy that triggered tax obligations for us in India".
IANAL but I believe it's due to the broad framing of India's "place of effective management" rules.
This sort of thing ends up being mostly to Singapore's benefit, transnationals won't place their APAC region office in India as a result.
As a tech sector APAC manager I have been instructed more than once not to visit. Not for any internal conference, and definitely not for any customer contact, and to take verifiable steps to avoid making "management decisions" if visiting for a holiday. Typical quote, "You do not want to be the guy that triggered tax obligations for us in India".
IANAL but I believe it's due to the broad framing of India's "place of effective management" rules.
This sort of thing ends up being mostly to Singapore's benefit, transnationals won't place their APAC region office in India as a result.
As long as your main revenue base is not India, and you're not visiting to make board decisions visiting India, on the business of a subsidiary, should not be an issue.
You're looking at taxation for resident income, and source income. Sure, a local entity should pay tax at source, in your case India, but unless you're sitting in India and taking global decisions or decisions creating significant global impact, no need to pay on resident income.
IANAL but this is my experience interacting mainly with large corporate. You might want to ask legal in your company or have an international independent auditor advise; don't want the wool may well being pulled over your eyes. And that would be bad, as it would be a dereliction of trust. And I've seen that a lot when, non India specific, but offshored locations try to build ill-founded silos.
You're looking at taxation for resident income, and source income. Sure, a local entity should pay tax at source, in your case India, but unless you're sitting in India and taking global decisions or decisions creating significant global impact, no need to pay on resident income.
IANAL but this is my experience interacting mainly with large corporate. You might want to ask legal in your company or have an international independent auditor advise; don't want the wool may well being pulled over your eyes. And that would be bad, as it would be a dereliction of trust. And I've seen that a lot when, non India specific, but offshored locations try to build ill-founded silos.
talk to the licensing lawyers and copyright holders. it has nothing to do with infrastructure and everything to do with licensing content.
Outside of the US, content on Netflix varies from "bad" to "meh".
To be fair, so do most competitors.
In Germany Netflix is "meh" but the only half-decent alternative is Amazon Prime, which isn't better at all.
In Germany Netflix is "meh" but the only half-decent alternative is Amazon Prime, which isn't better at all.
Are there English subtitles in Amazon Prime in Germany?
Subtitles are a deal-breaker for me, so I've been sourcing my movies from a local brick-and-stone DVD rental after Netflix clamped down on the VPN and DNS workarounds. Surprisingly the series offering in Netflix is quite OK in this regard though.
Subtitles are a deal-breaker for me, so I've been sourcing my movies from a local brick-and-stone DVD rental after Netflix clamped down on the VPN and DNS workarounds. Surprisingly the series offering in Netflix is quite OK in this regard though.
Not always. Also, there seems to be only one audio track per product, so e.g. "Mr Robot" and "Mr Robot [OV]" are two different products (you can't just switch half-way through an episode like you can with a DVD or Netflix). This is possibly infuriating if you actually rent/buy a TV show and find out you either can't keep up with the original voices or the dubbed audio is unbearable.
>To be fair, so do most competitors.
Not these "pirates" guys. They seem to have all of the good stuff and some more, so they outcompete Netflix outside the US quite well.
Not these "pirates" guys. They seem to have all of the good stuff and some more, so they outcompete Netflix outside the US quite well.
Why not use VPN services like F-Secure + Netflix to access movies? Is Netflix blocking them? I imagine they should not be, because of the recent ISP-throttling scandals.
There are several levels of answers.
Is Netflix blocking VPNs? Yes. They didn't use to, but they started doing it around a month or so ago. Some VPNs could route around it, some couldn't.
Alternative answer: using a VPN is against Netflix' TOS, so it's not really different from regular piracy. You could argue that you pay for convenience, but you could also argue that if you are going to open yourself to a lawsuit, you might not want to do it with your personal credit card.
Is Netflix blocking VPNs? Yes. They didn't use to, but they started doing it around a month or so ago. Some VPNs could route around it, some couldn't.
Alternative answer: using a VPN is against Netflix' TOS, so it's not really different from regular piracy. You could argue that you pay for convenience, but you could also argue that if you are going to open yourself to a lawsuit, you might not want to do it with your personal credit card.
Allegedly, they have gone on some banhammer-binges recently for VPN users, as well as blocking many forms of VPN. Even spinning up a VPS in a datacenter in your own state and using a VPN may not work for you. I assume copyright holders have prompted this regulation, because I doubt Netflix cares too much what you watch.
Not sure about F-Secure specifically but Netflix has recently been blocking DNS and VPN region-switching.
India has about 50-60 ms RTT to SG so it isn't that much of an issue.
How many did they find in Kansas?
I'm also curious if the public disclosure of this info means the researchers didn't bother to do the same reverse engineering for say YouTube?
Disclosure: I work at Google on Cloud, but don't work on edge caching.