The Mystery of the Slow Downloads(panic.com)
panic.com
The Mystery of the Slow Downloads
https://panic.com/blog/mystery-of-the-slow-downloads/
103 comments
The service you get from a business tier is nothing like the service you get from residential. In Australia I managed a thousand or so ADSL connections with TID [1] and the support was excellent. Telstra is much maligned in Australia, but it's not like they can't provide good service.
[1] https://www.telstra.com.au/business-enterprise/solutions/net...
[1] https://www.telstra.com.au/business-enterprise/solutions/net...
[deleted]
Comcast Business Tier support had been decent. However, now they threaten a $99 service call charge if they can't replicate the issue or it goes away by the time the tech shows up. Response time is usually within a day but for something up the network a bit they'll just throw blame at the customer and bill them.
Otherwise, I will say that lately their commitment to keeping equipment up on the east coast has been admirable. My house has been without power since Friday but they threw generators on all the fiber nodes' power supplies to keep things up. pic: https://pbs.twimg.com/media/DXjUdljXUAElsRX.jpg
Otherwise, I will say that lately their commitment to keeping equipment up on the east coast has been admirable. My house has been without power since Friday but they threw generators on all the fiber nodes' power supplies to keep things up. pic: https://pbs.twimg.com/media/DXjUdljXUAElsRX.jpg
Interesting. Is a power supply required for cable based internet? If not, then they certainly would need to do that in order to maintain a level of feature parity.
It would be pretty annoying to have a generator running at home, but no internet (although it certainly still happens with downed wires). Or worse, you have power, but the fiber box doesn't.
It would be pretty annoying to have a generator running at home, but no internet (although it certainly still happens with downed wires). Or worse, you have power, but the fiber box doesn't.
So that magical box on the pole has batteries inside. Depending on how they have it set up, you get around 24-48 hours of battery power before the fiber node converting optical to coaxial goes dead. They often have a LED on the outside showing red or green depending on status of the node.
A mini generator solves the issue of powering the node if your power utility is worthless and can't restore things after a few days.
A mini generator solves the issue of powering the node if your power utility is worthless and can't restore things after a few days.
That is incredible!
Were you able to make use of it despite your house not having power?
Yes, the cable wires are below power lines and even when they get ripped down from the poles they usually keep working.
EDIT: I am guessing Comcast learned after Sandy took out power for a month. We had power for a few days until the batteries died out on the nodes when Sandy hit... even with all the lines ripped down everywhere.
EDIT: I am guessing Comcast learned after Sandy took out power for a month. We had power for a few days until the batteries died out on the nodes when Sandy hit... even with all the lines ripped down everywhere.
My current place of employment has Comcast. We have a major outage every couple months during which all work ceases. I rely on an Internet connection for everything I do professionally so I stay away from cable because stability > speed.
Funny story: my employer switched to Comcast business because our old DSL line went down about once a month. We've had one outage in a year, but everyone got hit by that.
(this outage: https://www.wired.com/story/how-a-tiny-error-shut-off-the-in... )
(this outage: https://www.wired.com/story/how-a-tiny-error-shut-off-the-in... )
Meanwhile I've had Comcast residentually for about a year now without any noticed outages. At work we've got a local ISP (who we actually do design work for!) and our internet goes down once or twice a week at least.
There's really no point in saying a nationwide company is reliable or not because it's incredibly region-specific. At home I've never had connection issues, I generally get the speeds I pay for, and I've never had any billing issues. But you ask someone who lives 100 miles in any direction with Comcast and they'll have a completely different story.
There's really no point in saying a nationwide company is reliable or not because it's incredibly region-specific. At home I've never had connection issues, I generally get the speeds I pay for, and I've never had any billing issues. But you ask someone who lives 100 miles in any direction with Comcast and they'll have a completely different story.
We have a few offices in the USA and the offices are connected to Comcast. I was really surprised how good they were after reading all the blog posts about how bad they were. In both locations they offered us the best service (Sip trunking and internet) at the best price. Not only that, they also had to shell out for a major build to one of our facilities without us flipping any part of the bill. When we've had problems, required changes, they were really fast and competent. I can't speak for their consumer division but I would recommend them based on my experiences.
I'd be very interested in reading about what you're doing with your home network, and how much you're paying for their services.
When I last looked, Comcast charged around $200/mo for 250Mbps down / 20Mbps up, unlimited bandwidth, and a static IP. At the time it seemed too much, but now I pay around $150 for for 130Mbps down / 13Mbps up, unlimited bandwidth, and a dynamic IP. So I might have to give it a bit more serious consideration.
When I last looked, Comcast charged around $200/mo for 250Mbps down / 20Mbps up, unlimited bandwidth, and a static IP. At the time it seemed too much, but now I pay around $150 for for 130Mbps down / 13Mbps up, unlimited bandwidth, and a dynamic IP. So I might have to give it a bit more serious consideration.
Where do you live with those completely insane prices? Spectrum(TimeWarner) in New York is $60 for 100/10. Which usually pegs at 120/12 any time of the day I use it.
Public-facing email/web/etc server (that replaced my colo box), everything else is behind a NAT "router" that also uses one of the static IPs. I do some minor web hosting for friends, etc.
I started out with their 27/7 plan, then for years had 50 down/10 up, but I recently talked with their retention department and managed to upgrade to 150/20 with my same five statics, for $140-150ish/month. No bandwidth caps (it's not even metered) or anything.
I started out with their 27/7 plan, then for years had 50 down/10 up, but I recently talked with their retention department and managed to upgrade to 150/20 with my same five statics, for $140-150ish/month. No bandwidth caps (it's not even metered) or anything.
I also have Comcast Business, but I'd give them a 40/100. (But most of that score is because the service usually works.)
> I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
I've had two major outages:
* One outage where IPv6 connectivity was completely lost, and IPv4 connectivity was completely unaffected. I noticed this immediately, because I don't use Comcast's DNS servers; instead I use Google's, over IPv6 exclusively, so it was immediately apparent.
* One outage where IPv4 SYN packets would go unanswered, however, once answered, the connection was quite stable. IPv6 was completely unaffected.
In both instances, the tech on the phone insisted that "my signal strength" was "very weak", and that they'd send a tech out to fix that right away. The representatives both time were unable to comprehend — let alone answer — my question of "how would signal strength only affect one of IPv4/v6? Do you think the copper can somehow discriminate which protocol is in use?" In the end, the representative adjusted by strength until it was supposedly awesome, and in neither case did the problem get resolved at that point, of course.
It took much insisting to get them to look at their own network in the first case, and to replace my modem in the latter case. (I have no idea what went wrong in the latter case, but my guess is "something with NAT" and/or the IPv4 hardware. Regardless, replacing the modem did fix the issue.)
I've also tried — and have yet to convince them of this: every single one of their black Comcast-branded modems will forget the 2 GHz WiFi WPA-PSK key on any sort of power cycle or reset, and revert back to the default, but only for the key. (All other settings, such as port forwardings or the SSID for the AP remain intact.) This, I believe, only happens if you use an actual key; if you use a password from which a key is derived, you won't be affected.
They've also tried to upsell me during outages … which, really? Why do companies do that?
I did at least finally get a representative that was human enough to laugh when I told them that the modem was doing the "blinken light dance". (The LEDs on the front panel go through a sequence as the device boots and acquires a connection.)
> When I called to get PTR records added for my five static IPs,
I would love to have this … I just didn't think I could request such a thing without the hell of trying to explain what a PTR record is.
> I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
I've had two major outages:
* One outage where IPv6 connectivity was completely lost, and IPv4 connectivity was completely unaffected. I noticed this immediately, because I don't use Comcast's DNS servers; instead I use Google's, over IPv6 exclusively, so it was immediately apparent.
* One outage where IPv4 SYN packets would go unanswered, however, once answered, the connection was quite stable. IPv6 was completely unaffected.
In both instances, the tech on the phone insisted that "my signal strength" was "very weak", and that they'd send a tech out to fix that right away. The representatives both time were unable to comprehend — let alone answer — my question of "how would signal strength only affect one of IPv4/v6? Do you think the copper can somehow discriminate which protocol is in use?" In the end, the representative adjusted by strength until it was supposedly awesome, and in neither case did the problem get resolved at that point, of course.
It took much insisting to get them to look at their own network in the first case, and to replace my modem in the latter case. (I have no idea what went wrong in the latter case, but my guess is "something with NAT" and/or the IPv4 hardware. Regardless, replacing the modem did fix the issue.)
I've also tried — and have yet to convince them of this: every single one of their black Comcast-branded modems will forget the 2 GHz WiFi WPA-PSK key on any sort of power cycle or reset, and revert back to the default, but only for the key. (All other settings, such as port forwardings or the SSID for the AP remain intact.) This, I believe, only happens if you use an actual key; if you use a password from which a key is derived, you won't be affected.
They've also tried to upsell me during outages … which, really? Why do companies do that?
I did at least finally get a representative that was human enough to laugh when I told them that the modem was doing the "blinken light dance". (The LEDs on the front panel go through a sequence as the device boots and acquires a connection.)
> When I called to get PTR records added for my five static IPs,
I would love to have this … I just didn't think I could request such a thing without the hell of trying to explain what a PTR record is.
Ex-Comcast employee here. Just to try and help crystallize some things
>replacing the modem did fix the issue.
Comcast's IPv6 deployment has been historically...interesting.
A lot of DOCSIS 2.0 devices still exist on Comcast's network. And most of them (claim) they support IPv6. Suffice it to say they cause a metric ton of problems in of themselves
It's entirely possible that your signal strength may have been "too weak" when reading it in Comcast's "Einstein" troubleshooting tool. That wouldn't manifest itself in the way you describe, but the phone operator may be assuming the technician will call in to the NOC (or "XOC", the "Xfinity Operations Center" in Comcast terminology) to correct the problem you were calling about. The one they didn't understand
To add to this, Comcast equipment is awful. Truly awful.
All OEM's (D-Link, Linksys, Netgear etc) do similar to this. But Comcast seems to outsource development for its equipment to the lowest possible bidders. When I left, Comcast had just started deploying the "XB3" generation of modems
The first one they released. The Cisco DPC3939 would hang on World of Warcraft's patches. The only thing I could deduce as to why was it spawned a few hundred connections (WoW uses a Bitorrent style downloader)
The DPC3939 and DP3941 Comcast modems would typically take 5-10 minutes to reboot. Unfortunately tamper seals prevented me from wiring the thing up to see WHY it would take that god damn long
>They've also tried to upsell me during outages … which, really? Why do companies do that?
Comcast sees technical support as a cost center. They outright tell all agents that they ARE a cost on their first day. I snapped this lovely picture early on in my time at Comcast, which details part of the "S4 Call Flow" in use at Comcast https://i.imgur.com/GXm6Bx0.jpg (dated 2014)
>replacing the modem did fix the issue.
Comcast's IPv6 deployment has been historically...interesting.
A lot of DOCSIS 2.0 devices still exist on Comcast's network. And most of them (claim) they support IPv6. Suffice it to say they cause a metric ton of problems in of themselves
It's entirely possible that your signal strength may have been "too weak" when reading it in Comcast's "Einstein" troubleshooting tool. That wouldn't manifest itself in the way you describe, but the phone operator may be assuming the technician will call in to the NOC (or "XOC", the "Xfinity Operations Center" in Comcast terminology) to correct the problem you were calling about. The one they didn't understand
To add to this, Comcast equipment is awful. Truly awful.
All OEM's (D-Link, Linksys, Netgear etc) do similar to this. But Comcast seems to outsource development for its equipment to the lowest possible bidders. When I left, Comcast had just started deploying the "XB3" generation of modems
The first one they released. The Cisco DPC3939 would hang on World of Warcraft's patches. The only thing I could deduce as to why was it spawned a few hundred connections (WoW uses a Bitorrent style downloader)
The DPC3939 and DP3941 Comcast modems would typically take 5-10 minutes to reboot. Unfortunately tamper seals prevented me from wiring the thing up to see WHY it would take that god damn long
>They've also tried to upsell me during outages … which, really? Why do companies do that?
Comcast sees technical support as a cost center. They outright tell all agents that they ARE a cost on their first day. I snapped this lovely picture early on in my time at Comcast, which details part of the "S4 Call Flow" in use at Comcast https://i.imgur.com/GXm6Bx0.jpg (dated 2014)
I've got their "Business Wireless Gateway", a Cisco bit that really shouldn't have the Cisco name on it.. I turned off its built-in WiFi and use a Ubiquiti UAP-AC-PRO. The BWG just functions as a modem and somewhere to plug the WAN interface of my USG "router" into.
What is the price difference (in percent) between a typical residential service and a typical business service?
50-100% more, depending on speed and features.
However, Business Class typically has no metering/caps, better support, and a faster SLA for getting things fixed (depending on your area and the type of outage).
I have a similar issue - very slow download - with my ADSL access provided by SFR in France.
I did similar tests, by downloading the same file from servers hosted at different providers, and the bandwidth was always very slow, except when I changed the HTTP port from 80 (the standard) to an arbitrary chosen number! This change alone made me able to use the full bandwidth provided by my physical link! It was the proof that my ISP was running some kind of traffic shaping, which is a shame.
More information if you read French: https://twitter.com/ngrilly/status/756453318113783809?s=19
I did similar tests, by downloading the same file from servers hosted at different providers, and the bandwidth was always very slow, except when I changed the HTTP port from 80 (the standard) to an arbitrary chosen number! This change alone made me able to use the full bandwidth provided by my physical link! It was the proof that my ISP was running some kind of traffic shaping, which is a shame.
More information if you read French: https://twitter.com/ngrilly/status/756453318113783809?s=19
No such issue over here (SFR FTTH), bandwidth is extremely stable and through the roof.
Free SAS is absolutely terrible in that regard. I ran a few experiments (at home - when I was still a customer - and at friends') over the years similar to that blog post, and starting 17:00 things get throttled down to death due to clogged pipes. It's almost on the clock. The behaviour is immediately obvious to anyone having some background in network engineering, and support is tone deaf to the issue.
Free SAS is absolutely terrible in that regard. I ran a few experiments (at home - when I was still a customer - and at friends') over the years similar to that blog post, and starting 17:00 things get throttled down to death due to clogged pipes. It's almost on the clock. The behaviour is immediately obvious to anyone having some background in network engineering, and support is tone deaf to the issue.
[deleted]
I'm not surprised it works flawlessly with SFR FTTH. The traffic shaping I noticed only happens in the areas where SFR outsources the ADSL termination to Axione, which is the case where I live (near La Rochelle).
Same conclusion from me.
Free will throttle everything from 17:00 to 00:00 and during the week ends these last few weeks.
On the other hand I don't know any other provider who just gives a fixed IP.
Free will throttle everything from 17:00 to 00:00 and during the week ends these last few weeks.
On the other hand I don't know any other provider who just gives a fixed IP.
> On the other hand I don't know any other provider who just gives a fixed IP.
SFR does (at least on FTTH) and you also get native IPv6 (not 6rd) and (IIRC) a /48. That is, unless you're subscribed via RED (which I currently am) in which case the IPv6 feature is disabled on the remote end (you can enable it on the NB6 router but it just sits there waiting for the prefix to be announced). On cable the router is different but they're on DOCSIS 3.0 mostly by now and I hear at least some people do get IPv6.
Orange still leases dynamic IPv4, dunno if they enable IPv6 by default now or if it's getting tiered like SFR.
SFR does (at least on FTTH) and you also get native IPv6 (not 6rd) and (IIRC) a /48. That is, unless you're subscribed via RED (which I currently am) in which case the IPv6 feature is disabled on the remote end (you can enable it on the NB6 router but it just sits there waiting for the prefix to be announced). On cable the router is different but they're on DOCSIS 3.0 mostly by now and I hear at least some people do get IPv6.
Orange still leases dynamic IPv4, dunno if they enable IPv6 by default now or if it's getting tiered like SFR.
>Orange still leases dynamic IPv4, dunno if they enable IPv6 by default now or if it's getting tiered like SFR.
AFAIK it is now enabled by default -- but maybe only on new boxes.
But they leases dynamic ipv6.
Exactly like static v4 if you want a static v6, you have to pay for it.
AFAIK it is now enabled by default -- but maybe only on new boxes.
But they leases dynamic ipv6.
Exactly like static v4 if you want a static v6, you have to pay for it.
Same story with Free ADSL: I used to get terrible speeds from anything on AWS (including our own services), around 100 KB/s, until very recently when suddenly I was able to saturate my ADSL line from them. Now I don't dread having to download GB-sized log files.
It didn't really make sense to me why AWS was a target of their bandwidth extortion tactics until I realized Netflix uses AWS.. So I guess thank you Netflix for paying the ransom?
It didn't really make sense to me why AWS was a target of their bandwidth extortion tactics until I realized Netflix uses AWS.. So I guess thank you Netflix for paying the ransom?
> sends the user to run the same script on the control server, which we chose to host with Linode
Just went through a 3 month long issue with Linode, slow download speeds at peak hours on Cox cable. They said to call Cox, Cox escalated me to a guy who did not understand mtr tool. On Linode's end, they would not acknowledge any issue & burned up many hours of time trying to convince me there was no issue.
Ended up switching to Digital Ocean since Linode refused to call Cox on my behalf after I failed to get through. We also had the slowness issue with Linode with users in Dallas, where our servers were located. I suspect this will only get worse as net neutrality is repealed. I'm not saying these ISPs are slowing things on purpose, but if they do so accidentally, then imagine if they have green light to do so on purpose.
Linode also uses Cogent, FYI.
Just went through a 3 month long issue with Linode, slow download speeds at peak hours on Cox cable. They said to call Cox, Cox escalated me to a guy who did not understand mtr tool. On Linode's end, they would not acknowledge any issue & burned up many hours of time trying to convince me there was no issue.
Ended up switching to Digital Ocean since Linode refused to call Cox on my behalf after I failed to get through. We also had the slowness issue with Linode with users in Dallas, where our servers were located. I suspect this will only get worse as net neutrality is repealed. I'm not saying these ISPs are slowing things on purpose, but if they do so accidentally, then imagine if they have green light to do so on purpose.
Linode also uses Cogent, FYI.
To any of my fellow network engineers:
Although you very well might be, do not be tempted to stop reading after you get to the sentence, "The Panic web servers have a single connection to the internet via Cogent."
Believe it or not, it's not (just) Cogent this time. It's Comcast, too. Or, more specifically, saturated peering links between them.
To the others: this will likely not come as a surprise to any neteng who has had to deal with either of them.
Although you very well might be, do not be tempted to stop reading after you get to the sentence, "The Panic web servers have a single connection to the internet via Cogent."
Believe it or not, it's not (just) Cogent this time. It's Comcast, too. Or, more specifically, saturated peering links between them.
To the others: this will likely not come as a surprise to any neteng who has had to deal with either of them.
Deutsche Telekom is actually doing this on purpose here in Germany in order to extract money from service providers. Hetzner has documented this here: https://wiki.hetzner.de/index.php/Double_Paid_Traffic/en
That was interesting. I do believe that's the first time I've heard a positive story about Comcast. It's amazing how much cooperation has to happen to make the Internet work.
Is this really a positive story about Comcast? I suppose you can say all's well that ends well, but that they were collateral damage in Comcast's attempts to make Netflix double pay for traffic in the first place more than cancels that out, in my opinion.
I believe there are lots of engineers there that want to do the right thing. It's all the layers of executives that mess it up.
Years ago when Comcast first put in service to our neighborhood the link was plagued by packet loss, to the extent that the connection was highly unreliable. I tried getting help through customer support and got nowhere for two weeks. Eventually I found a number for their local office, called, and authoritatively requested to be put through to "network engineering." Surprisingly that worked, and I ended up on the phone with a really nice guy who took me seriously from the moment I explained what I did for a living. He asked me to email him my tracerts and ping tests, which I did. A day or so later there were three trucks in the court and after that the service was flawless. I think most engineers, at heart, are like that guy.
Yeah the thing is that good engineers are expensive and they’re not going to be answering customer service calls for the most part.
The trick is getting escalated to someone who knows what they’re doing.
The trick is getting escalated to someone who knows what they’re doing.
I know it's completely anecdotal, but I've been with Comcast for two years now and I've always had a good customer service experience with them. That said, I'm in an area with a fair amount of ISP competition, so it's possible they're more careful here.
Entirely possible it's just a roll of the dice. My friend and I both had Comcast at our places only .5 miles from one another and the experiences were polar opposite.
She never got the information for logging into the website to pay and missed bills (I was at the set up so it wasn't that she just forgot, neither of us knew then about it), at my setup the tech made sure I could login before they left.
When she had a bum modem Comcast sent several folk out to investigate by turning the modem off and on; when I had an issue, I spent 30 minutes on chat and they sent a new one next day delivery.
When she returned both our modems for us when we left the area, they lost hers and tried to charge her saying she never turned it in, never an issue with mine.
Even leaving Comcast was a world of difference as she was sent to customer retention despite leaving the state to one without Comcast, but I was just allowed to cancel peacefully. (granted I told them I was moving abroad...)
So more anecdata, but ymmv and will always vary imo with Comcast. I would not call their support good. Just never too bad.
(The local isp, however, had fantastic service, both network and human)
Edit: fixed Android's "awesome" autocorrect which changed were to we're
She never got the information for logging into the website to pay and missed bills (I was at the set up so it wasn't that she just forgot, neither of us knew then about it), at my setup the tech made sure I could login before they left.
When she had a bum modem Comcast sent several folk out to investigate by turning the modem off and on; when I had an issue, I spent 30 minutes on chat and they sent a new one next day delivery.
When she returned both our modems for us when we left the area, they lost hers and tried to charge her saying she never turned it in, never an issue with mine.
Even leaving Comcast was a world of difference as she was sent to customer retention despite leaving the state to one without Comcast, but I was just allowed to cancel peacefully. (granted I told them I was moving abroad...)
So more anecdata, but ymmv and will always vary imo with Comcast. I would not call their support good. Just never too bad.
(The local isp, however, had fantastic service, both network and human)
Edit: fixed Android's "awesome" autocorrect which changed were to we're
Same here... All my friends complain about Comcast all the time and I’m thinking there must be something wrong with me cause all my interactions with them have been positive, and my service and price are decent.
(I promise I’m not a paid comcast shill...)
(I promise I’m not a paid comcast shill...)
My experience with Comcast has been decidedly split. I know a few people who work on the engineering side at Comcast and they—to an individual—have been great. Comcast also has sponsored engineering meet ups in Philadelphia. The three or four people who have come out to work on issues with my service have likewise been great.
At the same time, every one of my tens of interactions with customer service at Comcast have been a disaster. Unfortunately, the customer service experience can be the one that overshadows the rest, and I'm sure there are plenty of customers who don't have the positive in-person experiences to balance out the poor customer service experiences. Tough to keep it all in perspective. Also important to note that I haven't been using Comcast since early 2017, so things could have changed in the interim.
At the same time, every one of my tens of interactions with customer service at Comcast have been a disaster. Unfortunately, the customer service experience can be the one that overshadows the rest, and I'm sure there are plenty of customers who don't have the positive in-person experiences to balance out the poor customer service experiences. Tough to keep it all in perspective. Also important to note that I haven't been using Comcast since early 2017, so things could have changed in the interim.
Since Comcast offshored its support call centers, every single contact I've had with anybody there (including "supervisors") has been excruciatingly, um, unhelpful.
That’s been my experience with Charter. Near perfect service from them yet I see other friends in my area who complain about them constantly.
One somehow managed to turn a free upgrade from 60gb to 100gb internet service into a complaint because he had to call to ask...
One somehow managed to turn a free upgrade from 60gb to 100gb internet service into a complaint because he had to call to ask...
I envy your luck. I've had non-problematic experiences, but also several negative ones, including one that I (hopefully) resolved after many months.
Most of their support staff are friendly, but rarely helpful. One or two have been actively horrible, but that's not the norm.
To date the big events are:
1. I started having some lag spikes, and after the normal troubleshooting, they scheduled a tech to visit as the issue was probably my cable modem. Want to guess what the tech DIDN'T HAVE WITH HIM? (And not a "I swear i loaded one I'm so sorry" but just a "yeah, I can't even say if it's your modem since I have nothing to test with. Do you have another we can use?"
2. The problem was the modem, so I bought a replacement. After a while a modem rental fee appeared on my bill. I called, they apologized and removed the fee. It silently reappeared the next month. I called again, and they said the serial number was in their inventory as theirs, so reality didn't matter. After a couple of more tries (never assume anything the person says is accurate. Even "there are no notes about that on your account") I was fed up enough that when fios became available I jumped on it. While cancelling the service the guy was apologetic about the modem mix up, removed the last months fee and I wasn't required to send them my modem, even though it was clear I wasn't going to change my mind. headdesk
3. Now on a new coast, I go to cancel my rental service when I buy a house. Friendly rep said it was taken care of. 2 months later they are calling me about a bill because the service was never cancelled. So I pay, but I'm very thorough in making sure they say it's done. Another month, another bill, and the account still isn't cancelled. This guy was an ass, and it was a waste of precious time repeating myself to him, so I asked what it would cost to be completely rid of them. I paid that money, confirmed very explicitly that there was no other reason to interact with them ever again, and verified online a few times that my account was indeed cancelled. That was the end of October. Last week I get a call from a bill collection agency, Comcast says I owe MORE money.
First call is a waste of time - no sign on my account that I had previously called, says that I owe for the month OF my cancellation, refuses to transfer me to anyone and offered me the phone number FOR VERIZON!.
Second call was much the same, but I demanded they make a note that I said I had previously cancelled. But the reason I "owed" the money was suddenly different.
Third chat was online. Absolutely no difference other than not having to deal with their infuriating voice system. (Pet peeve: systems that ask for account numbers, but don't give them to the support person.)
Fourth call I just explained repeatedly that I would have to be transferred to someone because I wouldn't even pay to get rid of them, since that doesn't work. I also mentioned that I had the complaint forms for the FCC and BBB along with Twitter all open in my browser, ready for use. No idea if it was the threat or just my persistence, but she had her supervisor "pull my account" (which at least one prior call had said they had done) - she saw my list of angry calls, but as she tried to walk me through why I owed money, she noticed a request for cancellation on Sep 1. Service was actually Nov 5. Suddenly I don't owe anything.
Tomorrow I check to see if the bill collector has been called off, then to see if anything has made it to my credit report either way.
Whole thing consumed several hours of my life, but it's not worth my time or the blood pressure to pursue further correction.
I vowed that if monopoly ever required I work with Comcast again, I would record every piece of data - dates, times, confirmation numbers, and I'd also callback immediately to confirm the notes on my account. Basically don't just assume stupidity, assume malice. Assume that anything they said was a lie until you prove otherwise, because that is the level of prep needed, regardless of whether it is malice or incompetence. With luck I will never have to.
Comcast heard Time Warner had displaced them as having the worst customer service as was all "hold my beer". Ugh.
Most of their support staff are friendly, but rarely helpful. One or two have been actively horrible, but that's not the norm.
To date the big events are:
1. I started having some lag spikes, and after the normal troubleshooting, they scheduled a tech to visit as the issue was probably my cable modem. Want to guess what the tech DIDN'T HAVE WITH HIM? (And not a "I swear i loaded one I'm so sorry" but just a "yeah, I can't even say if it's your modem since I have nothing to test with. Do you have another we can use?"
2. The problem was the modem, so I bought a replacement. After a while a modem rental fee appeared on my bill. I called, they apologized and removed the fee. It silently reappeared the next month. I called again, and they said the serial number was in their inventory as theirs, so reality didn't matter. After a couple of more tries (never assume anything the person says is accurate. Even "there are no notes about that on your account") I was fed up enough that when fios became available I jumped on it. While cancelling the service the guy was apologetic about the modem mix up, removed the last months fee and I wasn't required to send them my modem, even though it was clear I wasn't going to change my mind. headdesk
3. Now on a new coast, I go to cancel my rental service when I buy a house. Friendly rep said it was taken care of. 2 months later they are calling me about a bill because the service was never cancelled. So I pay, but I'm very thorough in making sure they say it's done. Another month, another bill, and the account still isn't cancelled. This guy was an ass, and it was a waste of precious time repeating myself to him, so I asked what it would cost to be completely rid of them. I paid that money, confirmed very explicitly that there was no other reason to interact with them ever again, and verified online a few times that my account was indeed cancelled. That was the end of October. Last week I get a call from a bill collection agency, Comcast says I owe MORE money.
First call is a waste of time - no sign on my account that I had previously called, says that I owe for the month OF my cancellation, refuses to transfer me to anyone and offered me the phone number FOR VERIZON!.
Second call was much the same, but I demanded they make a note that I said I had previously cancelled. But the reason I "owed" the money was suddenly different.
Third chat was online. Absolutely no difference other than not having to deal with their infuriating voice system. (Pet peeve: systems that ask for account numbers, but don't give them to the support person.)
Fourth call I just explained repeatedly that I would have to be transferred to someone because I wouldn't even pay to get rid of them, since that doesn't work. I also mentioned that I had the complaint forms for the FCC and BBB along with Twitter all open in my browser, ready for use. No idea if it was the threat or just my persistence, but she had her supervisor "pull my account" (which at least one prior call had said they had done) - she saw my list of angry calls, but as she tried to walk me through why I owed money, she noticed a request for cancellation on Sep 1. Service was actually Nov 5. Suddenly I don't owe anything.
Tomorrow I check to see if the bill collector has been called off, then to see if anything has made it to my credit report either way.
Whole thing consumed several hours of my life, but it's not worth my time or the blood pressure to pursue further correction.
I vowed that if monopoly ever required I work with Comcast again, I would record every piece of data - dates, times, confirmation numbers, and I'd also callback immediately to confirm the notes on my account. Basically don't just assume stupidity, assume malice. Assume that anything they said was a lie until you prove otherwise, because that is the level of prep needed, regardless of whether it is malice or incompetence. With luck I will never have to.
Comcast heard Time Warner had displaced them as having the worst customer service as was all "hold my beer". Ugh.
I have found the phrase "I will be contacting my state attorney general office, consumer fraud division" to be effective in resolving issues like that. I've had to follow up and actually do that once, they were quite helpful and the issue went away pretty quickly
You can glean a lot of upset-customer anecdata just by hanging around your local Comcast office/storefront and listening in.
> The Panic web servers have a single connection to the internet via Cogent.
Say no more. Hope this company gives their NetOps department the requested budget next time. Would have been extra credit if they started monitoring their download servers for TCP retransmits and grouping it by netblock to look for trends (or just move it to the cloud where it's someone else's problem)
Say no more. Hope this company gives their NetOps department the requested budget next time. Would have been extra credit if they started monitoring their download servers for TCP retransmits and grouping it by netblock to look for trends (or just move it to the cloud where it's someone else's problem)
Panic has something like 20 employees; I would be mildly surprised if there’s anyone dedicated to this area full-time.
That's a shame. They should have picked up someone for this around employee number 8-10.
Panic's business is making apps that run on your local machine. Most of their functionality isn't cloud-based. There's no way a full-time server admin should be one of the first ten employees; it's not even obvious that they have enough work to keep a sysadmin busy full-time doing stuff that isn't contributing directly to the development of their apps. When you only have ten employees, you don't pay one of them to spend all day watching log files.
Which is why distribution using the App store is so great. You don't have to worry about distribution, payment... Too bad not all Apps will work in the restrictions Apple created.
Well, they tried to do their own thing outside of the app store. You'll need someone working on reachability, deployments, performance, security, monitoring, etc.
Or just use the Appstore.
Or just use the Appstore.
After learning what they do I am even surprised the author did extra mile to fix the issue this way. I would probably just go with some CDN.
See lots of posts in here about Comcast customer service issues. Here's a link to the executive escalation form:
http://customer.xfinity.com/help-and-support/vp-contact-form
In my experience, the people responding to those tickets get shit done.
http://customer.xfinity.com/help-and-support/vp-contact-form
In my experience, the people responding to those tickets get shit done.
Thanks!!
I've been fighting dreadful peering between CenturyLink and a variety of endpoints. I can saturate my gigabit connection on speedtest.net with a directly-peered test host, but YouTube videos at peak times of day buffer at 720p, and downloading from any USA OpenBSD mirror (of which there are only a few, and none in Seattle) nets me about 4mbps, compared to 450mbps or so I can get on a Comcast connection.
What's funny is that using a VPN or tunnel to which CL has a good connection resolves the issue, because the VPN/tunnel host has better peering than CL does.
The difference between Panic's anecdote and mine, is that CenturyLink support won't even acknowledge the issue exists.
What's funny is that using a VPN or tunnel to which CL has a good connection resolves the issue, because the VPN/tunnel host has better peering than CL does.
The difference between Panic's anecdote and mine, is that CenturyLink support won't even acknowledge the issue exists.
Many US ISPs have inadequate peering with YouTube and Netflix. I had the same issue with TWC and I've seen reports of it with other providers.
One thing to try is using different DNS servers. That may give you a different server with less traffic. If that doesn't work a VPN, as you mentioned, will.
One thing to try is using different DNS servers. That may give you a different server with less traffic. If that doesn't work a VPN, as you mentioned, will.
I live in Seattle and use Centurylink and experience none of this. My children watch youtube all evening with no degradation of quality, even while my wife and I watch netflix. I have never experienced slowness with centurylink -- I wonder if there is some bad hop that the vpn is avoiding?
I notice some videos play fine at 1080p60, while others auto-throttle down to 360p. I wonder if this indicates that YouTube videos can be stored on a variety of datacenters (with a variety of paths between them and CenturyLink). Google's video quality report also rates CenturyLink in Seattle as "standard definition" [1]. I don't have Netflix, but I just scored 910 Mbps on fast.com (Netflix's speed test), so I think their connection to Netflix must be good.
A quick Google search reveals a thread of similar stories: https://www.reddit.com/r/centurylink/comments/5y1jzy/does_ce...
[1] https://www.google.com/get/videoqualityreport/
A quick Google search reveals a thread of similar stories: https://www.reddit.com/r/centurylink/comments/5y1jzy/does_ce...
[1] https://www.google.com/get/videoqualityreport/
I don't have an in to Google, but a bit of fairly simple logic would suggest that while YouTube presumably puts the short tail of popular videos on all nodes, the long tail will be much more distributed and less replicated; the costs inevitably force them that way. I can't prove it but as a long-tail consumer myself I believe I've witnessed some videos getting pulled in from some slow long-term storage before, things like conference videos from five years ago in the low hundreds of views.
Try restructuring your body as a corporation.
The real takeaway is that Cogent over-promises and under-delivers. If you want reliable transit, switch providers.
I'm not sure that's something that you can take away from the article, Cogent was performing well on other ISPs, Comcast was not providing enough bandwidth for Cogent due to a disput external to Panic.
While we can't know I'd be willing to be the "unspecified" traffic re-engineering was to prioritize Cogent's Panic traffic specifically. Comcast is still the bad guy.
While we can't know I'd be willing to be the "unspecified" traffic re-engineering was to prioritize Cogent's Panic traffic specifically. Comcast is still the bad guy.
Working in the field of networking I can say it's highly unlikely that Cogent is performing traffic shaping specific to Panic traffic. That's a ton of changes to a ton of hardware with a risk for screwing things up that just isn't worth it. Cogent isn't going to go to those lengths for a client that has a single connection and hosts a small / moderate site. Beyond that: the connection that Panic has to Cogent is most likely through the data center they're colocating with, not a direct connection to Panic equipment. That's not for certain, it's entirely possible that Panic paid for a drop from Cogent; however I'm guessing that's not the case.
What I find odd is that they would colocate their hardware in a DC with only a single provider. The company I work for has four transit providers so as to ensure uptime / reliability (don't get me started on how often we experience unplanned maintenances from our upstream providers). Seems like Panic may want to consider a different host for their colocation or examine (provided they're dealing with Cogent directly) a secondary ISP.
What I find odd is that they would colocate their hardware in a DC with only a single provider. The company I work for has four transit providers so as to ensure uptime / reliability (don't get me started on how often we experience unplanned maintenances from our upstream providers). Seems like Panic may want to consider a different host for their colocation or examine (provided they're dealing with Cogent directly) a secondary ISP.
>What I find odd is that they would colocate their hardware in a DC with only a single provider.
I think it's extremely unlikely the provider only has cogent. More likely is the fact that the link between the provider and comcast preferred cogent - which panic would have no way to change.
I think it's extremely unlikely the provider only has cogent. More likely is the fact that the link between the provider and comcast preferred cogent - which panic would have no way to change.
That's not true. panic.com has an A record (and www is a CNAME to @) that points to an IP address that's not PI and advertised only as part of 38.0.0.0/8, obviously, only from Cogent.
It is. In my experience, Cogent has more peering problems than other transit providers do (e.g. L3). The submitted article is
a clear example of this. I don't find these peering problems surprising, given Cogent is much cheaper vs the competition.
>prioritize Cogent's Panic traffic specifically. Comcast is still the bad guy.
No, from the article it really sounds like Comcast added more capacity to their Cogent-Comcast peering links, at their expense. Comcast was not obligated to do this for Panic in any way. How does that make Comcast 'the bad guy'? Cogent should have worked with Comcast to fix this issue in the first place. Panic is paying Cogent for transit, Cogent should be the one getting Comcast to increase their peering bandwidth. If Panic has to go around the company they're are paying to get their problem fixed, that's terrible service.
>prioritize Cogent's Panic traffic specifically. Comcast is still the bad guy.
No, from the article it really sounds like Comcast added more capacity to their Cogent-Comcast peering links, at their expense. Comcast was not obligated to do this for Panic in any way. How does that make Comcast 'the bad guy'? Cogent should have worked with Comcast to fix this issue in the first place. Panic is paying Cogent for transit, Cogent should be the one getting Comcast to increase their peering bandwidth. If Panic has to go around the company they're are paying to get their problem fixed, that's terrible service.
The people downvoting you don't have experience with transit or Cogent - they are literally the worst peer. Their company motto is to undercut everyone else on price and oversaturate their peers as much as possible. They are a horrible provider. They have been de-peered by large providers like Layer3 many times due to flagrant abuse.
That said the price is great and I had a large presence at Cogent's Herndon location for the better part of a decade. Terrible service but price was so great, couldn't resist.
That said the price is great and I had a large presence at Cogent's Herndon location for the better part of a decade. Terrible service but price was so great, couldn't resist.
I think you mean level3,, not layer3. And where did that get them?
http://news.level3.com/2017-11-01-CenturyLink-completes-acqu...
Ahh, right. After being literally the best backbone provider in the world, they were destroyed by last-mile ISPs in the US who choked them out of the game. Speaking of a shining example of why we need stronger anti-monopoly laws in the US...
http://news.level3.com/2017-11-01-CenturyLink-completes-acqu...
Ahh, right. After being literally the best backbone provider in the world, they were destroyed by last-mile ISPs in the US who choked them out of the game. Speaking of a shining example of why we need stronger anti-monopoly laws in the US...
Eh, I was never impressed with Level3. Their network was huge, and they generally had zero clue what was happening on it. Big networks aren't always better. Level3 got to be huge by running up massive amounts of debt and pricing services uncompetitively. People buying Level3 transit were those in the "you never get fired for buying IBM" mindset.
FTA:
> We colocate our own servers, rather than using AWS or any other PaaS, and we also don’t currently use a CDN or any other cloud distribution platform.
I wonder if Panic would never have encountered this slow download problem if they did use AWS or another PaaS.
> We colocate our own servers, rather than using AWS or any other PaaS, and we also don’t currently use a CDN or any other cloud distribution platform.
I wonder if Panic would never have encountered this slow download problem if they did use AWS or another PaaS.
I have had random issues with Cloudfront returning error pages saying it is busy. AWS is not perfect either.
Their bandwidth costs would skyrocket. AWS and other cloud providers overcharge considerably on outbound transfer.
>Their bandwidth costs would skyrocket.
I don't think they would. Or at least, the costs would still be tiny.
Like Panic, my company mostly sells desktop apps. We're smaller than Panic, but we still have a considerable quantity of downloads each day. We use AWS S3 + AWS CloudFront to host our app downloads, and the cost is so small as to be irrelevant to my business.
I don't think they would. Or at least, the costs would still be tiny.
Like Panic, my company mostly sells desktop apps. We're smaller than Panic, but we still have a considerable quantity of downloads each day. We use AWS S3 + AWS CloudFront to host our app downloads, and the cost is so small as to be irrelevant to my business.
It’s possible I’m over exaggerating. Some of the AWS service price gouging grinds my gears.
S3 and Fastly (or some other very inexpensive CDN) would be fine.
S3 and Fastly (or some other very inexpensive CDN) would be fine.
You're not really.
My costs: $799 dedicated on 1gbps connection (overkill)
AWS Costs: $6000+
AWS is simply impossible for me.
My costs: $799 dedicated on 1gbps connection (overkill)
AWS Costs: $6000+
AWS is simply impossible for me.
The TLDR as video (have not watched it; at work) is a great idea :)
I read 'Cogent' and knew it was going to be a peering issue. They are 'special'.
You should really use a cdn, your users abroad will have extremely poor speeds connecting to cogent overseas from their residential connections.
It's almost mandatory unless you have 3-4 POPs strategically placed geographically.
It's almost mandatory unless you have 3-4 POPs strategically placed geographically.
> And then, the craziest thing happened… They wrote back quickly.
Now they are responsive, because of the whole Net Neutrality repeal backlash. They are scared to provide factual evidence of foul play - it will help the courts to blast FCC for their decision.
But if they win in courts, Comcast and other crooks will definitely double down on starving peering, as expected to extort money. And obviously, they won't ever write back on this topic or will write that there is nothing wrong on their side.
The only way to prevent that peering extortion by Comcast and the like, is to have explicit rules that forbid it.
Now they are responsive, because of the whole Net Neutrality repeal backlash. They are scared to provide factual evidence of foul play - it will help the courts to blast FCC for their decision.
But if they win in courts, Comcast and other crooks will definitely double down on starving peering, as expected to extort money. And obviously, they won't ever write back on this topic or will write that there is nothing wrong on their side.
The only way to prevent that peering extortion by Comcast and the like, is to have explicit rules that forbid it.
Isn't Cogent a "low tier" bandwidth provider?
They are, but even the larger players get into peering disputes with eyeball networks. Level3 and Comcast had a huge peering spat a few years ago (via Netflix). Everyone's ports run hot with China Telecom and China Unicom.
It's almost always political (revenue) problems and rarely technical.
That said, I wouldn't pick up Cogent with their negative attitudes towards IPv6.
It's almost always political (revenue) problems and rarely technical.
That said, I wouldn't pick up Cogent with their negative attitudes towards IPv6.
>I wouldn't pick up Cogent with their negative attitudes towards IPv6.
Ignoring and not adopting IPv6 is bad enough, but being negative about it? What did they say?
Sometimes I think we’re never going to fully adopt IPv6. It’ll just be layers of CGNAT.
Ignoring and not adopting IPv6 is bad enough, but being negative about it? What did they say?
Sometimes I think we’re never going to fully adopt IPv6. It’ll just be layers of CGNAT.
It’s my understanding that doing NAT at the scale becomes very expensive due to state tracking and syncing between gateways to the point that ipv6 is getting to be the cheaper option.
There might yet be hope.
There might yet be hope.
Google offered all free IPv6 peering to all major transit carriers. Cogent turned them down and said, no, you need to pay Cogent for this. They were the only carrier to do this. So Cogent uses other transit carriers to reach Google. If you're on Cogent, you don't have a direct peering connection to Google. They probably did this to balance out their ratios with other carriers, but it's bad from a customer perspective.
You're better off using another tier 1 carrier than Cogent if you use a fair amount of Google.
You're better off using another tier 1 carrier than Cogent if you use a fair amount of Google.
I'm not seeing any connectivity between Cogent and Google.
Try Cogent's Looking Glass[1] and test connectivity to www.google.com, for example. ("Destination unreachable: No route")
[1] http://www.cogentco.com/en/network/looking-glass
[1] http://www.cogentco.com/en/network/looking-glass
> why me? Why was I able to get this corrected with an e-mail when Cogent couldn’t?
Because Panic doesn't compete with Xfinity.
Because Panic doesn't compete with Xfinity.
Speaking of slow downloads, when I save a webpage in Chromium, it seems as if it downloads the page byte by byte, reporting progress after every byte. And another quirky thing is that if I close the tab during a download, then the download is cancelled. Did anyone else notice this?
For some reason the byte counter represents number of saved files when you do a "Save complete web page" in Chromium. It still displays like "4/53 B, 5 secs left". Weirdly sloppy.
It's common. Is it because of the protocols the website/connection is using? If it is like stream, the total size won't be known until the end.
Usually a missing Content-Length in the response header.
Same here, I think it is due to ublock origin or some other extension.
I'd love a service that gave me the right contact for the right problem in all the businesses possible. Finding the right person while googling seems like a matter of luck.
Surprising to see an ISP listening to its clients for once.
I've worked with Comcast for quite a few years though Contingent Technology Solutions, and I have noticed a change in their quality lately. It's almost like they care a little bit more. Don't quote me on that, though.
It is still a mystery after watching that.
I'd be surprised if it was an issue related to real capacity as opposed to traffic shaping. This reads like the exact kind of thing that net neutrality is meant to prevent.
There's no reason to use traffic shaping to get the before situation, when you can simply not upgrade peering connections.
The real question is why Comcast decided to upgrade the connections? Not upgrading Cogent peering is a pretty defensible position [1], although maybe they didn't want to look that much worse than their competition.
[1] See the previous results of Cogent peering disputes, which is about half of all the highly publicized peering disputes.
The real question is why Comcast decided to upgrade the connections? Not upgrading Cogent peering is a pretty defensible position [1], although maybe they didn't want to look that much worse than their competition.
[1] See the previous results of Cogent peering disputes, which is about half of all the highly publicized peering disputes.
Anyway, years later - I'd give Comcast Biz a score of 95/100. When I called to get PTR records added for my five static IPs, I ended up on the phone with the guy who was typing them in. For the very few times I've had to call them (less than once a year), I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
Had upstream issues once (even to their own speedtest server, so it wasn't going outside of the Comcast network), and after a couple of calls I woke up one Saturday morning to two techs and a manager on my front porch. "We brought an unlocked modem and everything we might need, not leaving until this is fixed." They were at my place for about an hour, then went down the road to fix something on a junction box, and I had full speeds back.
Couldn't get them to leave that uncapped modem, though...