There’s something rotten in the state of online video streaming(gigaom.com)
gigaom.com
There’s something rotten in the state of online video streaming
http://gigaom.com/2014/02/06/theres-something-rotten-in-the-state-of-online-video-streaming-and-the-data-is-starting-to-emerge/?post_id=744271220_10152208272006221#_=_
61 comments
I can't find it now, but I have read an article about a guy who wrote a script to ping speedtest once/minute. It made a clear, measurable difference in his download rate from other sites as long as he kept the script running.
Yep! Another commenter mentioned writing a script to do this. I would very much appreciate it if you could try to find the search terms to locate that article and post the article as a reply. I would appreciate it very much!
Try some alternative performance-testing sites, and see if there's a difference with those; if there might be different QoS rules in play for specific web sites.
The speedof.me site is one I've occasionally used, and that also avoids the need for Adobe Flash.
The speedof.me site is one I've occasionally used, and that also avoids the need for Adobe Flash.
I would love to see some further investigation into this. My initial thought is that if it's shown likely, someone needs to make a little utility that just loads speedtest.net every minute or so in the background.
In my area, Comcast runs its own Speedtest.net server to hide the fact that the closest peering point that leads back to the same state is a few thousand miles away and is insanely oversubscribed.
If Speedtest had more integrity, ISP-owned servers would be deprioritized.
If Speedtest had more integrity, ISP-owned servers would be deprioritized.
Not just that, Ookla /actively partners/ with the ISPs. Comcast's site is actually at speedtest.comcast.net.
There's a financial dis-incentive for them to do so.
There's a financial dis-incentive for them to do so.
Of course this is not all that dishonest. It is just a matter of what you want to measure. For last-mile speed measurement (which is what I personally use speedtest.net most for) using ISP provided local servers makes a lot of sense because it also removes most external factors from the measurement.
And really there is no "central point of Internet" which to measure speed to/from.
And really there is no "central point of Internet" which to measure speed to/from.
When the last mile was usually the limiting factor, last mile speed was the most important thing to measure. Now that major ISPs like Comcast are either throttling streaming video or routing streaming video through highly suboptimal connections, we need more speed tests that indicate real-world performance.
That said, I think my mention of integrity in my previous comment was excessive.
That said, I think my mention of integrity in my previous comment was excessive.
How many servers on the Internet sit on the same switch on the same data center from your ISP's cable modem head end, though?
Netflix is unwatchable during prime time where I live.
However, I have also noted that on two separate occasions when I began examining the Frontier FIOS site, I received a personal knock on my door from my local Comcast representative. Wondering if there was a connection to my search for another provider and the person at my door, I asked her if she was talking to everyone in the neighborhood or just me. It was just me--they sent a person out to personally ask me about my Comcast service.
All they have to do is fix the evening streaming.
However, I have also noted that on two separate occasions when I began examining the Frontier FIOS site, I received a personal knock on my door from my local Comcast representative. Wondering if there was a connection to my search for another provider and the person at my door, I asked her if she was talking to everyone in the neighborhood or just me. It was just me--they sent a person out to personally ask me about my Comcast service.
All they have to do is fix the evening streaming.
No one has pointed out that this is bad for the providers throttling traffic.
If Time Warner, Comcast, and AT&T want their names to be associated with budget brands then its ok. Customers accept poor quality when the price is really low and there is no lock in (pre-paid mobile phone service.) Good luck getting those people to purchase premium services from you years in to the future.
On Hacker News we complain a lot about things most people don't care about like privacy and security. Most big companies can disregard those issues with minimal visible impact to their business long term. Video streaming quality is different because its what basically all of your customers are doing. Anyone with an IQ over 70 knows something is wrong, and you can't excuse it away.
Its possible that the peak bandwidth doesn't exist. In that case these ISPs are overselling their "inventory" much like airlines oversell seats. It could be early signs of infrastructure issues to investors.
If Time Warner, Comcast, and AT&T want their names to be associated with budget brands then its ok. Customers accept poor quality when the price is really low and there is no lock in (pre-paid mobile phone service.) Good luck getting those people to purchase premium services from you years in to the future.
On Hacker News we complain a lot about things most people don't care about like privacy and security. Most big companies can disregard those issues with minimal visible impact to their business long term. Video streaming quality is different because its what basically all of your customers are doing. Anyone with an IQ over 70 knows something is wrong, and you can't excuse it away.
Its possible that the peak bandwidth doesn't exist. In that case these ISPs are overselling their "inventory" much like airlines oversell seats. It could be early signs of infrastructure issues to investors.
> If Time Warner, Comcast, and AT&T want their names to be associated with budget brands then its ok.
It is definitely not OK. Time Warner is absolute garbage and has been for a very long time. Cablevision has an excellent network.
I know this very well.
Unfortunately, some towns contracted with Time Warner and some with Cablevision and you can't pick who you want as your provider ... unless you decide on where you live based on the available ISP.
As much as I love a good connection, I will not make my decision on where to live based on the available internet provider. Reputation is worthless unless people have a choice.
It is definitely not OK. Time Warner is absolute garbage and has been for a very long time. Cablevision has an excellent network.
I know this very well.
Unfortunately, some towns contracted with Time Warner and some with Cablevision and you can't pick who you want as your provider ... unless you decide on where you live based on the available ISP.
As much as I love a good connection, I will not make my decision on where to live based on the available internet provider. Reputation is worthless unless people have a choice.
> unless you decide on where you live based on the available ISP.
Interestingly, if I ever move out of my current neighborhood, this (along with apartment price) will dictate where I look. I'm pretty happy with Cablevision for now, though.
Interestingly, if I ever move out of my current neighborhood, this (along with apartment price) will dictate where I look. I'm pretty happy with Cablevision for now, though.
What happens when you pick a town with a good ISP, which then gets bought by GiantEvilCo a month after you move in?
I don't think voting with your feet and mortgages is a solution. Perhaps forgoing entirely? When Charter dicked me around in 08, I went back to dialup for over a year before DSL became available at my house. Screw 'em. I can still read HN and text-based websites at 56k.
Which isn't a whole lot slower than AT&T is giving me now. But at least they don't treat me like a subhuman ATM like Charter did.
I don't think voting with your feet and mortgages is a solution. Perhaps forgoing entirely? When Charter dicked me around in 08, I went back to dialup for over a year before DSL became available at my house. Screw 'em. I can still read HN and text-based websites at 56k.
Which isn't a whole lot slower than AT&T is giving me now. But at least they don't treat me like a subhuman ATM like Charter did.
They're not overselling their inventory, their purposefully throttling data from competitors to either a) extort more money from them or b) promote their own competing offerings.
Most in the US have no other options for high speed Internet access other than their local cable monopoly which is about as an extreme "lock in" as you can get.
Sure they're pissing people off but there is no alternative unless the FCC steps in and forces them into an "open carrier" model or at least some model other than what we have now which clearly isn't working.
Most in the US have no other options for high speed Internet access other than their local cable monopoly which is about as an extreme "lock in" as you can get.
Sure they're pissing people off but there is no alternative unless the FCC steps in and forces them into an "open carrier" model or at least some model other than what we have now which clearly isn't working.
It really is moral hazard that the same entity providing you with streaming video via cable or similar can also throttle your access with impunity to competitors streaming video via web.
Nobody dares to speak to how bad this is for the traffic. The "infrastructure issues" were introduced a long time ago when it was decided that all data needs to go through egress points, and conform to arbitrary standards for metadata. Japan doesn't have this problem because their internet actually is decentralized, and their culture isn't at threat from users streaming their own video to as many people as they wish (and the network supports it).
It's very common for video sites to distribute video via p2p like bittorrent so that the hosting site load is much much less.
Make the cable companies common carriers!
https://petitions.whitehouse.gov/petition/restore-net-neutra...
https://petitions.whitehouse.gov/petition/restore-net-neutra...
I see that petition has already meet the required signature count. I haven't followed the state of petitions.whitehouse.gov; do we expect them to actually comment on this petition?
I am not sure. There seems to be evidence that they do not respond to a few petitions, even when they meet the 100K signatures (at least immediately)[1]. Since [1]'s release, it appears that they have responded to a few of the petitions mentioned, and if you look at the site itself[2] (assuming "popular" is actually sorting by the number of signatures) there are only 15 unanswered petitions that meet the 100K requirement (though they changed the limit a few times and some may have met the original requirement). I suspect that they only respond to petitions that are in the mutual interest of the public and the politicians themselves but I have no evidence to support it.
[1] http://www.nextgov.com/emerging-tech/2014/01/white-house-owe... [2] https://petitions.whitehouse.gov/petitions/popular/0/2/0
[1] http://www.nextgov.com/emerging-tech/2014/01/white-house-owe... [2] https://petitions.whitehouse.gov/petitions/popular/0/2/0
This actually isn't a net neutrality issue because back in 2009 when the FCC was writing the open internet order it decided not to mess with peering and interconnection disputes. So designating ISPs as common carriers could be a first step here, but it wouldn't be the only step.
Could you explain in a little more detail? I was under the impression that the FCC would fix prices for common carriers, and therefore they would have to at least charge everyone in the same manner.
I wonder if publishing these metrics would go a long way to solving the problem. I know I'd must prefer to sign up with an ISP that doesn't throttle traffic I care about. I guess the big problem with that idea is that in many areas of North America there is very little competition in the ISP marketplace.
If the government was going to do something, I'd prefer it concentrate on increasing competition. Ideally, every home should have at least two or three options for high speed network access.
If the government was going to do something, I'd prefer it concentrate on increasing competition. Ideally, every home should have at least two or three options for high speed network access.
> If the government was going to do something, I'd prefer it concentrate on increasing competition.
Indeed, competition would help tremendously. Mandated wholesale access to the last mile delivery would be a great start (and something that used to be mandated for DSL from ILECs). I believe AT&T, Verizon, Comcast, etc can do a good job of connecting my house to their network, and deliver my data to their office; from there, I'd prefer to have someone else take responsibility.
Bonuses if new regulation in the same spirit covered all mediums (or at least more of them), and addressed unfair pricing practices by AT&T from the last time (direct to consumer offerings were priced significantly below the wholesale pricing, which was anti-competitive). Some parameters for dealing with congestion in the shared medium between the end user and the central office(s) might be appropriate too.
Indeed, competition would help tremendously. Mandated wholesale access to the last mile delivery would be a great start (and something that used to be mandated for DSL from ILECs). I believe AT&T, Verizon, Comcast, etc can do a good job of connecting my house to their network, and deliver my data to their office; from there, I'd prefer to have someone else take responsibility.
Bonuses if new regulation in the same spirit covered all mediums (or at least more of them), and addressed unfair pricing practices by AT&T from the last time (direct to consumer offerings were priced significantly below the wholesale pricing, which was anti-competitive). Some parameters for dealing with congestion in the shared medium between the end user and the central office(s) might be appropriate too.
Google is making performance metrics for YouTube public to help people when choosing an ISP (in places where there is a choice): http://www.ntca.org/new-edge/video/google-to-rate-isps-based...
Direct link to the Google Reports site:
http://www.google.com/get/videoqualityreport/#how_video_gets...
"But they built the last mile so they should be able to do whatever they want with it..." /sarcasm
You know that comment is coming to this discussion at some point.
Wish there was a good answer to this. Making internet connectivity a 'utility' has its downsides and upsides. It really is becoming a pressing issue though (and not just because of netflix being slowed).
You know that comment is coming to this discussion at some point.
Wish there was a good answer to this. Making internet connectivity a 'utility' has its downsides and upsides. It really is becoming a pressing issue though (and not just because of netflix being slowed).
weren't they payed with tax money to build the infrastructure?
Varies by municipality, but pretty much everywhere the "last mile" is a local monopoly, granted by the town, county, or city with the rights to do so. And so long as that's the case there's no real competition and thus no way for Internet users to route around these abusive practices of their ISPs.
What's funny is that now the incumbents have the monopoly (paid for using taxpayer money) they're pushing to prevent new competitors like Google fiber from working with the municipalities at all: http://consumerist.com/2014/01/30/kansas-legislature-wants-t...
Sort of, my understanding is that they mostly received tax breaks with possibly some subsidies.
Either way, they received an incentive to do so.
Either way, they received an incentive to do so.
Verizon got something like $10billion in cash to build FiOS out in New York...and never did
I've heard this before. How can they "get away" not building it, legally speaking? Makes no sense to me in this context
Reminds me of the Comcast situation with Bitorrent in 2007 [0].
I have little doubt that ISPs will continue to escalate this game of service degradation and denial without strict regulation.
0: http://consumerist.com/2007/10/27/damning-proof-comcast-cont...
I have little doubt that ISPs will continue to escalate this game of service degradation and denial without strict regulation.
0: http://consumerist.com/2007/10/27/damning-proof-comcast-cont...
Many people don't realize much of the bit torrent debate ended because bit torrent switched from TCP to µTP. The problem was bit torrent clients caused massive congestion by making hundreds of TCP connections. µTP is extremely congestion/network friendly. It backs off based on latency, so it doesn't fill every buffer on the network.
The BitTorrent discussion didn't end because of uTP. It ended because ISPs got bigger fish to fry. Fish with actual money, like Google/Youtube and Netflix.
Yup. In 2007, bittorrent was 25% of internet traffic. With streaming becoming the standard way to consume since, bittorrent has become a drop in the bucket.
At least torrenters only download content once...
At least torrenters only download content once...
This is fine let them keep screwing around with Netflix, Amazon, and Google(Youtube) it won't be long before they get pissed and build their own internet backbone/service providers(already happening).
I can't wait to get Amazon High Speed Internet.
So please, let them keep acting jerks.
I can't wait to get Amazon High Speed Internet.
So please, let them keep acting jerks.
I wish ISPs business where completely out of the content business.
Slowing competing content could be justified as a "fiduciary duty", yet it's much simpler than that: Gatekeepers and highwaymen, extracting private taxes on content they did not create.
Slowing competing content could be justified as a "fiduciary duty", yet it's much simpler than that: Gatekeepers and highwaymen, extracting private taxes on content they did not create.
Is there any interest in a Chrome plugin that would passively report anonymized download speed for things like YouTube or Amazon? I can imagine that kind of info would be useful.
YouTube used to offer a view of your download speed, compared to other customers of your ISP, other ISPs, and other countries, but it appears to be closed down, sadly.
* youtube.com/my_speed
* youtube.com/my_speed
I think if you could do some detection of what connectivity is being used, and other sorts of relevant connection data, it would be a valuable motivator to see, "We've detected you are 30% slower than other customers with x and y differences in service"
This is super hand wavy, but I think with proper data it could be a useful tool.
This is super hand wavy, but I think with proper data it could be a useful tool.
That's what many of the tests at M-Labs are trying to do (I encourage people to take a look http://www.measurementlab.net/) but packets don't traverse networks in a straightforward path. I think focusing on that data and trying to figure out ways to prove traffic manipulation and pinpoint consistent places/times/applications where it slows would be valuable. It's also worth understanding how the FCC currently measures speed and treats speed tests. It's not at all straightforward http://www.fcc.gov/measuring-broadband-america/2013/develope...
We should have a technically-dominated discussion about tests and the right way to look for problems, that we aren't having today because it's so wonky. We totally need data, but we also need to figure out what data will be most useful to prove bad behavior. For example, if my speed drops to 1 Mbps for 30 minutes during prime time but my speeds the rest of the day are 55 Mbps then that blip is heavily discounted, but shouldn't be given that I'm probably trying to stream video.
We should have a technically-dominated discussion about tests and the right way to look for problems, that we aren't having today because it's so wonky. We totally need data, but we also need to figure out what data will be most useful to prove bad behavior. For example, if my speed drops to 1 Mbps for 30 minutes during prime time but my speeds the rest of the day are 55 Mbps then that blip is heavily discounted, but shouldn't be given that I'm probably trying to stream video.
Indeed; ISP-level stats aren't fine-grained enough. We need route- and hop-level stats to find the real bottlenecks.
We've all known this was coming for years, actually, it is amazing it has taken this long.
If AT&T, Comcast, Verizon and other ISPs are intentionally limiting bandwidth to certain content providers, as the measurements seem to indicate, they may be actually doing us a favor. By doing this, they are causing people to discuss the issue and eventually lead to market changing trends or new regulations and potential legal hazards for ISPs that do this in the future.
If anything, Amazon is a smart company, there might be an opportunity here for last-mile internet delivery...Amazon Fiber Prime anyone?
If anything, Amazon is a smart company, there might be an opportunity here for last-mile internet delivery...Amazon Fiber Prime anyone?
I wish ISPs could stay out of the content business since their content curation and presentation are usually subpar at best.
I wish we could amend the constitution that they are not allowed to do anything at all but sell bits to consumers. No selling of content, access to consumers, devices or ice cream or whatever other scheme they come up with. Their sole interest has to be selling more bits! If they even as much as think about doing anything else there need to bd draconian penalties. I would even argue that they are endangering infrastructure and charge them with terrorism.
The most interesting thing is the astroturffing comments form unnamed Verizon employees in the first 10 comments
So now not only's the catalog is meh, the quality/stream too.
Why are they pushing people so much to piracy?
And don't get me started about the situation once you're outside the US.
Why are they pushing people so much to piracy?
And don't get me started about the situation once you're outside the US.
>While I was seeing my episode of The Good Wife falter at what appeared to be 1.9 Mbps, I was able to measure connection speeds of 28 Mbps to my house using a Speedtest.net test from Ookla. This is exactly the dichotomy that the M-Lab data is showing, and my example is not an isolated one; Comcast users have been complaining for months.
How do we know that it is not just the provider being sapped for bandwidth? I notice that speed tests I run in the mid-day and evening come back pretty much the same, but Netflix takes noticeably longer to buffer in the evening. I always assumed this was because their servers were much busier then.
How do we know that it is not just the provider being sapped for bandwidth? I notice that speed tests I run in the mid-day and evening come back pretty much the same, but Netflix takes noticeably longer to buffer in the evening. I always assumed this was because their servers were much busier then.
I'm guessing it's that link to Netflix from your provider that is busier at that time. I use a local provider for internet service, that despite offering crappy customer service, has given me consistent speeds, and no buffering issues with Netflix or Youtube at all.
The first rotten thing there is DRM. There are no DRM-free video services where you can buy anything (Headweb doesn't count because it's regionally restricted). So there simply is nothing to chose from. Failure at step 0. It's not just rotten - it's completely decomposed.
It is interesting that in the article about Verizon's shaping of traffic to AWS that the tech support had the author go to speedtest.net and test his traffic in order to tell him "see no problems!". I've had similar responses from AT&T when I was a DSL subscriber.
I understand that traffic shaping should only affect particular sites and that,in theory,there would be no need for a temporary disabling of traffic shaping for a speedtest. My theory is that due to the high volume of packets needed in order to have a good result on speedtest.net, the analyses of packets at the router for traffic shaping is enough of a slowdown to affect the results.
Anyone else ever notice this? Any alternative explanations as to why going to speedtest.net would suddenly make paused streaming video run again? I notice this mainly when watching live streaming youtube events or sometimes when I am watching netflix during peak hours.
I would love to hear alternative hypothesis as to what is going so that they could be tested!