Massive Dyn DNS outage
301 comments
All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down.
Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?
If ONLY they had 10X developers ...
EDIT: joking aside, the issue with multiple DNS providers is primarily (in my experience at the company I'm at having investigated this in the past) intelligent DNS entries. Example, 'return these A records, in this order, based on the number of requests, roughly balanced'.
There's no universal standard, just common aspects. DNS Provider A has one set of features, names for returning A records in some way, based on some weighted averaging, provider B has a different mechanism. So as an infrastructure person you have to:
1. Investigate provider A and B features for intelligent DNS (it's not even universally called intelligent DNS!), and mentally parse the commonalities and differences
2. Create a custom mechanism to keep them in sync internally. So you hope that A and B have an API for maintaining the records.
3. Ensure that when someone in your org wants to make an update, A and B update at the same time in the same way
4. APIs don't change.
EDIT: joking aside, the issue with multiple DNS providers is primarily (in my experience at the company I'm at having investigated this in the past) intelligent DNS entries. Example, 'return these A records, in this order, based on the number of requests, roughly balanced'.
There's no universal standard, just common aspects. DNS Provider A has one set of features, names for returning A records in some way, based on some weighted averaging, provider B has a different mechanism. So as an infrastructure person you have to:
1. Investigate provider A and B features for intelligent DNS (it's not even universally called intelligent DNS!), and mentally parse the commonalities and differences
2. Create a custom mechanism to keep them in sync internally. So you hope that A and B have an API for maintaining the records.
3. Ensure that when someone in your org wants to make an update, A and B update at the same time in the same way
4. APIs don't change.
In my experience Dyn pushes this functionality very hard during customer on boarding, effectively as a lock in for their platform.
Well I really think those features are useful, though. They let you get some powerful de-facto load balancing and HA and other complex 'features' without any new hardware or much complexity. They give you some easy creative freedom in designing a redundant infrastructure, give you some geo-aware DNS the likes of which I remember paying Akamai a lot of money for years go, etc. But the lock-in problem is absolutely right.
I'm also familiar with the aggressiveness of that sales team. I prefer another provider and they were trying to solicit our business by specifically calling out our provider as amenable to a DDOS attack which had occurred.
Someone with a simpler setup with standard DNS features will find it much easier to use multiple providers, of course.
I'm also familiar with the aggressiveness of that sales team. I prefer another provider and they were trying to solicit our business by specifically calling out our provider as amenable to a DDOS attack which had occurred.
Someone with a simpler setup with standard DNS features will find it much easier to use multiple providers, of course.
From where I'm looking at the internet (central Europe), I don't notice anything.
Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)
Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn
Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)
Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn
> Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)
Works on my continent.
Works on my continent.
Not really, I can't access our production servers which are in US east. Can't access Intercom with which we provide customer support. Our clients are mailing us that payment provider doesn't work either. So we're losing money while being in central EU.
I'm in Europe now and worked fine something like 30 min ago.
Now facing similar issues reported in original post.
Now facing similar issues reported in original post.
I'm in Las Vegas and I haven't had access to several sites all day. I don't think this is limited to eastern U.S.
Same Im in Illinios and I can't play destiny which is made by bungie which is down for some fucking reason
Yep same issue in Denver
Things are down here on the west coast too...
The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn.
We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.
We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.
That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.
How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.
Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed.
Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.
Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.
Why not? That's the whole idea behind having more than one NS server isn't it?
Exactly: there's nothing wrong with only using one provider if you're not willing to pay for two services but if you can't afford downtime you really need active diversity all the way down.
Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.
Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.
N.b., unless things have changed, I don't think Dyn (for one) allows secondary DNS. Maybe that has changed. {?}
agreed 100%. This kind of thing shouldn't happen and this widespread. It's like no one is preparing for worse case scenarios until AFTER something tragic happens.
On the West Coast and I just lost twitter/soundcloud/github - 9:40 am PST
On the West Coast and I just lost twitter/soundcloud/github - 9:40 am PST
loss of twitter, soundcloud, and github is what passes for tragic these days?
As a programmer, the loss of GitHub is a pretty big deal in terms of work productivity, and as a maintainer of production systems that rely on assets from GitHub, it's a pretty big availability issue.
I mean, i get it, its a big deal to us. But tragic? Syria is tragic. Github is just above annoying.
Let's say that it's tragic for the people whose job it is to maintain those services.
Try PayPal. That's gonna be a big issue for everyone who uses their payment processing.
Yeah, as if a "spare DNS" has anything to do with solving the problem. You could get into your own infrastructure, but your customers are still screwed.
..and they thought war happens only with guns and tanks. don't the know, digital is the new platform to kill each other?
Maybe it is time to widely adopt namecoin ?
Yes!! For sure, after such an attack it will force the internet into rethinking their DNS strategies.
Why commit to having a single point of weakness?
For those interested: https://namecoin.org/
For those interested: https://namecoin.org/
they need a round robin dns that is geographically dispersed
Important: PagerDuty.com seems affected by this outage. So keep a real good eye on your graphs today -- you might not receive the alert.
pagerduty has switched to AWS + 4 different root TLDS:
$ host -t NS pagerduty.com
pagerduty.com name server ns-219.awsdns-27.com.
pagerduty.com name server ns-739.awsdns-28.net.
pagerduty.com name server ns-1198.awsdns-21.org.
pagerduty.com name server ns-1569.awsdns-04.co.uk.
I think I'm satisfied.
$ host -t NS pagerduty.com
pagerduty.com name server ns-219.awsdns-27.com.
pagerduty.com name server ns-739.awsdns-28.net.
pagerduty.com name server ns-1198.awsdns-21.org.
pagerduty.com name server ns-1569.awsdns-04.co.uk.
I think I'm satisfied.
I'm not trying to single out PagerDuty in being vulnerable. They run a particularly crucial service, and I'm sure they are doing everything they can to get out from under this.
That said, I still can't load www.pagerduty.com in a browser right now. :/
That said, I still can't load www.pagerduty.com in a browser right now. :/
I just realised how online adult entertainment has the most redundancy of any Internet service category, bar none.
You’re right :)
$ dig @8.8.8.8 ns +short pornhub.com
ns2.p44.dynect.net.
ns3.p44.dynect.net.
ns4.p44.dynect.net.
sdns3.ultradns.net.
sdns3.ultradns.com.
sdns3.ultradns.org.
sdns3.ultradns.biz.
ns1.p44.dynect.net.
Amazon does too:
C:\>nslookup
> set type=ns
> amazon.com
Non-authoritative answer:
amazon.com nameserver = ns3.p31.dynect.net
amazon.com nameserver = pdns1.ultradns.net
amazon.com nameserver = ns4.p31.dynect.net
amazon.com nameserver = pdns6.ultradns.co.uk
amazon.com nameserver = ns1.p31.dynect.net
amazon.com nameserver = ns2.p31.dynect.netI'm sure this is to circumvent blacklisting rather than for this scenario.
I can’t think of any type of blacklisting this would prevent. Care you elaborate?
I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.
"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.
[x-posted on https://news.ycombinator.com/item?id=12759697 as well]
"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.
[x-posted on https://news.ycombinator.com/item?id=12759697 as well]
how did you post on twitter which is down too?
Cached DNS response.
Twitter isn't down, just DNS resolution of twitter.com
Anyone with an app affected by this because you use Heroku's SSL endpoint can fix this by switching to SNI. Unfortunately Heroku's API is also affected. There is a workaround... just edit your hosts file to point to the proper ip addresses temporarily.
add to /etc/hosts
23.21.149.112 api.heroku.com 107.21.99.123 ssl-doctor.heroku.com
Then use Heroku CLI to switch your app to the new SNI endpoint
heroku certs:add --type sni server.crt server.key -a YOUR_APP
That will spit out a new host name that you can point your DNS to and still be online.
add to /etc/hosts
23.21.149.112 api.heroku.com 107.21.99.123 ssl-doctor.heroku.com
Then use Heroku CLI to switch your app to the new SNI endpoint
heroku certs:add --type sni server.crt server.key -a YOUR_APP
That will spit out a new host name that you can point your DNS to and still be online.
Thanks a million, from me and the people who invested in my startup.
edit: apparently SNI isn't supported on legacy dynos.
edit: apparently SNI isn't supported on legacy dynos.
GitHub employee here. We're monitoring an incident with our upstream DNS provider:
https://twitter.com/githubstatus/status/789433336083001344
https://twitter.com/githubstatus/status/789433336083001344
Hahahahaha you do realize Twitter is one of the affected sites, right?
Exactly, I see tweet in Google SERPS for GitHub , I try to access and twitter too is down like Github.
not very useful to post a status update on a site that is experiencing the same issues with the same DNS provider.
maybe post a github gist? oh wait...
maybe post a github gist? oh wait...
> our upstream DNS provider
Maybe you should have more than one. Then I could actually carry out my work...
Maybe you should have more than one. Then I could actually carry out my work...
Can you CC on HN, twitter doesn't work for people as well. HN seems to be fine.
> We're monitoring an incident with our upstream DNS provider.
[pic](https://mikevanrossum.nl/stuff/gh.png)
[pic](https://mikevanrossum.nl/stuff/gh.png)
This link seems to work for me to reach the github status page (requires https certificate override, of course):
https://107.22.212.99/
(Alias for https://status.github.com, via https://octostatus-9676240.us-east-1.elb.amazonaws.com )
https://107.22.212.99/
(Alias for https://status.github.com, via https://octostatus-9676240.us-east-1.elb.amazonaws.com )
Can we add a line to our /etc/hosts file with a hard-coded DNS entry to fix it in the meantime? What's the IP address?
For me (EU) github.com resolves to 192.30.253.112.
Added these to my hosts file. Working fine as a temporary workaround.
199.16.156.70 twitter.com
104.244.43.231 abs.twimg.com
104.244.43.231 pbs.twimg.com
192.30.253.113 github.com
151.101.24.133 assets-cdn.github.com
199.16.156.70 twitter.com
104.244.43.231 abs.twimg.com
104.244.43.231 pbs.twimg.com
192.30.253.113 github.com
151.101.24.133 assets-cdn.github.com
I looked up an ip for twitter (via a google search) and still timeout on pings.
[edit: twitter and github are both accessible again.]
[edit: twitter and github are both accessible again.]
[deleted]
sadly Twitter is also down...
github homepage is unicorn now
DYN is experiencing a DDoS
https://www.dynstatus.com/incidents/nlr4yrr162t8
https://www.dynstatus.com/incidents/nlr4yrr162t8
Yep. Seems to be the root of the problem.
Issue came from dynect.net:
$ dig +trace github.com
; <<>> DiG 9.8.3-P1 <<>> +trace github.com
[...]
;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 154 ms
github.com. 172800 IN NS ns1.p16.dynect.net.
github.com. 172800 IN NS ns3.p16.dynect.net.
github.com. 172800 IN NS ns2.p16.dynect.net.
github.com. 172800 IN NS ns4.p16.dynect.net.
;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 54 ms
;; connection timed out; no servers could be reachedIf you need access to GitHub, add these to your hosts file:
192.30.253.113 github.com
151.101.4.133 assets-cdn.github.com
192.30.253.113 github.com
151.101.4.133 assets-cdn.github.com
As of 2016-10-21T09:20:19Z, it's doing better:
--------------------------------------------------------------------
<Commodore Pet 16 Unix>:$ dig +trace github.com
; <<>> DiG 9.8.3-P1 <<>> +trace github.com ;; global options: +cmd . 17430 IN NS i.root-servers.net. . 17430 IN NS a.root-servers.net. . 17430 IN NS b.root-servers.net. . 17430 IN NS l.root-servers.net. . 17430 IN NS d.root-servers.net. . 17430 IN NS g.root-servers.net. . 17430 IN NS h.root-servers.net. . 17430 IN NS c.root-servers.net. . 17430 IN NS e.root-servers.net. . 17430 IN NS j.root-servers.net. . 17430 IN NS k.root-servers.net. . 17430 IN NS m.root-servers.net. . 17430 IN NS f.root-servers.net. ;; Received 228 bytes from 8.8.8.8#53(8.8.8.8) in 382 ms
com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. ;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 363 ms
github.com. 172800 IN NS ns1.p16.dynect.net. github.com. 172800 IN NS ns3.p16.dynect.net. github.com. 172800 IN NS ns2.p16.dynect.net. github.com. 172800 IN NS ns4.p16.dynect.net. ;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 134 ms
github.com. 300 IN A 192.30.253.113 github.com. 86400 IN NS ns4.p16.dynect.net. github.com. 86400 IN NS ns3.p16.dynect.net. github.com. 86400 IN NS ns1.p16.dynect.net. github.com. 86400 IN NS ns2.p16.dynect.net. ;; Received 130 bytes from 2001:500:90:1::16#53(2001:500:90:1::16) in 25 ms
I ended up sending all my developers a patch to their hosts file... Hopefully they'll roll it back shortly.
--Donald
; <<>> DiG 9.8.3-P1 <<>> +trace github.com ;; global options: +cmd . 17430 IN NS i.root-servers.net. . 17430 IN NS a.root-servers.net. . 17430 IN NS b.root-servers.net. . 17430 IN NS l.root-servers.net. . 17430 IN NS d.root-servers.net. . 17430 IN NS g.root-servers.net. . 17430 IN NS h.root-servers.net. . 17430 IN NS c.root-servers.net. . 17430 IN NS e.root-servers.net. . 17430 IN NS j.root-servers.net. . 17430 IN NS k.root-servers.net. . 17430 IN NS m.root-servers.net. . 17430 IN NS f.root-servers.net. ;; Received 228 bytes from 8.8.8.8#53(8.8.8.8) in 382 ms
com. 172800 IN NS m.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. ;; Received 488 bytes from 192.228.79.201#53(192.228.79.201) in 363 ms
github.com. 172800 IN NS ns1.p16.dynect.net. github.com. 172800 IN NS ns3.p16.dynect.net. github.com. 172800 IN NS ns2.p16.dynect.net. github.com. 172800 IN NS ns4.p16.dynect.net. ;; Received 178 bytes from 192.42.93.30#53(192.42.93.30) in 134 ms
github.com. 300 IN A 192.30.253.113 github.com. 86400 IN NS ns4.p16.dynect.net. github.com. 86400 IN NS ns3.p16.dynect.net. github.com. 86400 IN NS ns1.p16.dynect.net. github.com. 86400 IN NS ns2.p16.dynect.net. ;; Received 130 bytes from 2001:500:90:1::16#53(2001:500:90:1::16) in 25 ms
I ended up sending all my developers a patch to their hosts file... Hopefully they'll roll it back shortly.
--Donald
They're investigating now: https://www.dynstatus.com/incidents/nlr4yrr162t8
I thought DNS (particularly public) was basically immune to DDoS?
If one DNS server is down, use the cached result or another server.
DNS is some of the most distributable, cachable data I can imagine.
If one DNS server is down, use the cached result or another server.
DNS is some of the most distributable, cachable data I can imagine.
The problem here lies with the authoritative nameservers. You have to retrieve the record from somewhere initially. The trouble here is that since these companies are using Dyn and only Dyn to be their Nameservers. When a dns request is made for github.com they're attempting to contact ns3.p20.dynect.net or another nameserver provided by Dyn. So yes caching will work for a little while, but only for the length of the TTL which your client is designed to respect. Once that TTL expires, your client and upstream DNS provider will attempt to contact that nameserver for a fresh record. Since you can't contact the nameservers, the website is effectively offline for all name resolution. Now if you know the record you can always forge the record locally in /etc/hosts or in a local dns resolver which allows for overrides. Hope this helps.
It was true that one DNS nameserver record == one physical server even at large providers. But we're beyond that now. Each nameserver at a physical location can be a cluster of hosts. Beyond that, with the use of anycast that single nameserver record may map to different clusters positioned around the world. This is how the root servers work and why they are more difficult to attack.
Of course small DNS providers will find it hard to run a system this way, but the larger providers follow the same architecture - anycast and multiple servers at each location. Google and OpenDNS for a start use this pattern - the famous 8.8.8.8 and 8.8.4.4 are in fact multiple server clusters all around the world.
Of course small DNS providers will find it hard to run a system this way, but the larger providers follow the same architecture - anycast and multiple servers at each location. Google and OpenDNS for a start use this pattern - the famous 8.8.8.8 and 8.8.4.4 are in fact multiple server clusters all around the world.
Sure - I assumed that's what the large organizations were doing but for the sake of explanation to those who didn't understand why the problem existed in the first place I didn't want to complicate my answer with something even more complex :) Appreciate your help doing so
Seriously, please someone explain this. I thought DNS info is propagated among DNS servers, and it is cached on all of them... How can an attack to a single DNS company cause this outage??
Explained above for ya
True, but DNS records have TTLs that expire, which forces you to reach back out to the authoritative server for the zone. A very decent number of records will have a TTL of 1hr, which means that a total DNS server outage lasting longer than an hour = 100% failure rate. You could always increase the TTL to a day (or whatever other arbitrary value), but that means that you'll be waiting upwards of a day for clients to see any updates to your DNS records. It's a tradeoff between different types of resiliency.
This is a simple case of a DNS provider evidently not having their crap together when it comes to a DDoS of any decent size. There are plenty of other DNS providers out there who are working fine currently, this DDoS is limited to just the one. It's all of that company's clients that are impacted.
I'm sure that DynETC will post something afterwards about how this was the largest DDoS they've ever encountered by many orders of magnitude and that there was no way for anyone to ever be prepared to take so much traffic... but at this point I think they're inept.
edit: just came back up. The TTLs on a few common records explain why this was so obviously a problem...
- www.reddit.com: 300 seconds (pointer to a fastly.net address, which is hosted via dynect)
- prod.reddit.map.fastlylb.net: 30 seconds
- api.twitter.com: 300 seconds
- herokussl.com: 3600 seconds
So, an outage of five minutes is enough to take out access to all of reddit and twitter. An outage of an hour is enough to take out heroku. Of course, these are best case scenarios: in reality, 50% of your users would lose access to reddit after 15 seconds (the fastly record), and so on.
This is a simple case of a DNS provider evidently not having their crap together when it comes to a DDoS of any decent size. There are plenty of other DNS providers out there who are working fine currently, this DDoS is limited to just the one. It's all of that company's clients that are impacted.
I'm sure that DynETC will post something afterwards about how this was the largest DDoS they've ever encountered by many orders of magnitude and that there was no way for anyone to ever be prepared to take so much traffic... but at this point I think they're inept.
edit: just came back up. The TTLs on a few common records explain why this was so obviously a problem...
- www.reddit.com: 300 seconds (pointer to a fastly.net address, which is hosted via dynect)
- prod.reddit.map.fastlylb.net: 30 seconds
- api.twitter.com: 300 seconds
- herokussl.com: 3600 seconds
So, an outage of five minutes is enough to take out access to all of reddit and twitter. An outage of an hour is enough to take out heroku. Of course, these are best case scenarios: in reality, 50% of your users would lose access to reddit after 15 seconds (the fastly record), and so on.
> DNS is some of the most distributable, cachable data I can imagine.
Indeed. How can it be so broken? And, it seem that it would make sense to use the last known value if upstream is unavailable.
Indeed. How can it be so broken? And, it seem that it would make sense to use the last known value if upstream is unavailable.
That would be a horrible idea from a security standpoint. Although not easy by any means for an ordinary attacker, at least one exploit scenario is obvious.
This is going to be a fun day. This little DNS outage is likely to cause millions of lost revenue for many industries.
Im dead in the water and I cant complain on twitter :-(
Im dead in the water and I cant complain on twitter :-(
I found out Twitter was down because I wanted to go complain about PayPal, lol.
change your name server at your registrar to something else, add all necessary entries in the new DNS and be up before DDoS is stopped.
Sadly we're too interconnected. Every company that relies on that DNS should do what you suggest, but the control is definitely not in our ( users ) hands.
Yeah, users are screwed. Unless they have a little more experience with how unreliable cloud can be, and they made a local copy of everything* their work depends on, just in case.
*Everything that can be local.
*Everything that can be local.
nah our corp dns is fine, its all the cloud services we and everyone else uses. Thank Sergey and Larry their stuff still works
git still works on your local! Get back to work
Reddit is out -- my productivity is up, and I'm not happy about it.
Working here. Try a non-US VPN if you want to stop doing work again.
Ran my VPN ansible script on a London-based VPS. I'm back to not working!
Reddit is up for me, but Github is down. Goodbye productivity.
[deleted]
DNS came back for a while and just started flaking again.
$ nslookup github.com
;; Got SERVFAIL reply from 8.8.8.8, trying next server
Server: 10.0.0.1
Address: 10.0.0.1#53
Non-authoritative answer:
Name: github.com
Address: 192.30.253.112
$ nslookup twitter.com
;; Got SERVFAIL reply from 8.8.8.8, trying next server
;; Got SERVFAIL reply from 8.8.4.4, trying next server
Server: 10.0.0.1
Address: 10.0.0.1#53
Non-authoritative answer:
Name: twitter.com
Address: 199.16.156.102
Name: twitter.com
Address: 199.16.156.230
Name: twitter.com
Address: 199.16.156.198
Name: twitter.com
Address: 199.16.156.70In CA - Spotify starting going in and out around 9:15a, then completely shut down at 9:20a (no offline playlists were available). As of 9:49a my offline playlists are back up, but nothing else.
Twitter was working from my mobile about 15 min ago, but not on my desktop. Now both are down. Can't believe Twitter is down. On my mobile I'm getting notifications, but can't view them. The people who are sending updates are mostly on the East Coast and Canada, so maybe it's back up for them.
Twitter was working from my mobile about 15 min ago, but not on my desktop. Now both are down. Can't believe Twitter is down. On my mobile I'm getting notifications, but can't view them. The people who are sending updates are mostly on the East Coast and Canada, so maybe it's back up for them.
I doubt that your offline playlists were affected ...
Not sure why they were, but I tried 6 different offline playlists and each of them said "can't play the current song", then cited internet detection... which hadn't been an issue.
Now (10:04a) all of my playlists are working again.
Now (10:04a) all of my playlists are working again.
Spotify has a terrible network connection algorithm that will sit there and spin forever trying to open the app if it thinks you're on the network but you're really not.
Put your phone in airplane mode, then re-open Spotify, and magically, all of your offline playlists will play.
Put your phone in airplane mode, then re-open Spotify, and magically, all of your offline playlists will play.
PayPal isn't working for me. Could cost a lot of people a lot of money.
Now even the DYN Status page is inaccessible from the Eastern US https://www.dynstatus.com/incidents/nlr4yrr162t8 (Github still down from the Eastern US.)
bigcommerce, volusion, new relic, optimizely, wistia, volusion, aweber, cnn, campaign monitor, all down for me. The biggest thing is seeing that ALL shopify stores are offline, so much $$$ being lost right now.
My Uber partner app crashed at 8am I was trying to complete a trip and it frozen my phone. It took about 5min for me to be able to enter back but it asked me for my SSN and permission to do a background check which is standard by Uber but I had already done so. Should I be concern that my personal data has been compromised? I contacted Uber but their idiots support people don't seem to have a clue and third fix is super basic like restart your phone, turn airplane mode or data on and off
Sendgrid support page is down as well
https://support.sendgrid.com/hc/en-us
SendGrid's support is hosted on Zendesk, which is (unfortunately) still experiencing the outage. My company's doc site (also on Zendesk) is down too.
It appears that people (domain name administrators) are switching away from Dyn as their DNS provider, as per https://status.heroku.com/incidents/965 (issued 19:21 UTC). Does this mean that other DNS providers (Amazon Route 53, CDNetworks, CloudFlare, DNSimple, easyDNS, Google DNS, Verisign, OpenDNS) could potentially become targets?
-Jason (5:13 PM EDT; 21:13 UTC)
-Jason (5:13 PM EDT; 21:13 UTC)
And aws:
6:13 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.
Amazon DNS must be getting loads of new customers today :)
Even though their status page says everything is basically fine -- oh just a blip with API dns this morning no biggie -- they have had major problems today too. Like, can't connect to dynamodb, sqs, kinesis.
Unfortunately they lie through their status page pretty regularly. I am not sure what it would take for them to admit they had a major problem.
Unfortunately they lie through their status page pretty regularly. I am not sure what it would take for them to admit they had a major problem.
Apparently us-east-1 was the only region that exclusively used dyndns (all other regions have failover strategies to other providers and that's why there wasn't downtime anywhere else but us-east and eu-west). They resolved the problem pretty quickly, but it drove our NOC insane...fortunately downtime for us was only about half an hour or so.
Yeah, thats about a half hour of down time for every AWS East based company... who knows how much lost revenue -- and yet they just say the service is green with a little info tag.
If our service is down for any customer (much less an entire segment of customers) we at least publish that we had a partial outage.
If our service is down for any customer (much less an entire segment of customers) we at least publish that we had a partial outage.
AthenaNet.athenahealth.com , one of Americas largest health care medical software providers is down, effecting 80,000 physicians and their patients.
GitHub pretty slow in Brazil
A lot of the bigger sites are very slow from Brazil today :(
And SP is also suffering from a lack of power in some areas. Happy Friday!
yeah, i thought it was my internet but it seems like a ton of people are having slow internet issues.
Brazucas unite :-)
Alguém teve algum outro problema fora os citados aqui?
Alguém teve algum outro problema fora os citados aqui?
same here. getting the "page taking way too long to load" page every time.
It seems incredibly misguided to plug an IP address into hosts from a forum post. Is there an official source from GitHub?
[deleted]
Chiming in from sunny Los Angeles
Up as of 5:30AM PST: Twitter, Etsy, Github, Soundcloud, Spotify, Dyn DNS
Down: Heroku, Pagerduty
Might want to use a VPN to another area
Up as of 5:30AM PST: Twitter, Etsy, Github, Soundcloud, Spotify, Dyn DNS
Down: Heroku, Pagerduty
Might want to use a VPN to another area
Is anyone noticing significant delays visiting CNN, ebay, orbitz? It took 15 seconds for the Orbitz page to load (tried from Google Chrome and Safari). It seems like outside of the core group of sites effected (GitHub, Twitter, PayPal), other major websites are having serious delays loading..
The DDOS also hit small business websites in North America. I know atleast three companies who experienced the DDOS attacks. The IPs that were port scanning prior to the attack are 173.254.216.66, 128.52.128.105 (MIT Edu), 172.98.67.39. I hope there is some action against these hackers.
Easy way to get around it for Github: https://www.apptic.me/blog/adding-github-to-your-hosts-file....
Same idea, just add the IPs to your hosts file.
Same idea, just add the IPs to your hosts file.
Change your DNS settings and some of these websites will now work.
http://www.dailydot.com/layer8/how-to-change-dns-settings-ma...
http://www.dailydot.com/layer8/how-to-change-dns-settings-ma...
http://okta.com is not working
Seems to be fine from the UK, so those of you with suitable VPNs might like to try that.
Many articles are now coming out about this outage this morning: http://www.techmeme.com/161021/p3#a161021p3
Whatsapp too:
# host -tns whatsapp.com
whatsapp.com name server ns2.p13.dynect.net.
whatsapp.com name server ns1.p13.dynect.net.
whatsapp.com name server ns4.p13.dynect.net.
whatsapp.com name server ns3.p13.dynect.net.
Thanks
Anyone has any idea which hosts desktop spotify app is trying to reach?
`netstat -punta | grep spotify` tells me they are on those IP:port :
* 151.101.120.246:80
* 194.132.198.50:4070
Problem is, I don't know what hosts this is supposed to be, and `dig -x` is not very helpful...
`netstat -punta | grep spotify` tells me they are on those IP:port :
* 151.101.120.246:80
* 194.132.198.50:4070
Problem is, I don't know what hosts this is supposed to be, and `dig -x` is not very helpful...
[deleted]
Why did they have to ddos the EA servers too I just wanna play battlefield 1
hosts file settings:
- to deploy to Heroku
Just remember to rollback your hosts file after the outage finishes.
- to deploy to Heroku
174.129.22.147 git.heroku.com
104.244.42.65 twitter.com
93.184.220.70 pbs.twimg.com
104.244.43.7 o.twimg.com
192.30.253.113 github.com
151.101.4.133 assets-cdn.github.com
Just remember to rollback your hosts file after the outage finishes.
CircleCI is also having issues viewing and running tests, viewing from Amsterdam: https://status.circleci.com/
This is not a dyn error, this is all the companies failed to prevent this to happen, is an old practice having different NS Servers for your domain, but it can prevent this to happen.
We are seeing inconsistent resolutions on fppaco.org. sometimes it resolves to a 4.x address and sometimes a 50.x address. According to whois.net, their DNS servers are DYNect
All sites look good for me in Europe using 8.8.8.8.
Edit: It actually looks like most of the sites are loading faster then normal.
Edit2: I have cleared the cache's on my machine and router. Still works.
Edit: It actually looks like most of the sites are loading faster then normal.
Edit2: I have cleared the cache's on my machine and router. Still works.
[deleted]
From Ireland all of the above are resolving (with no cache).
Is this another IP webcam etc. attack? Does anyone know of a write-up from a researcher in possession of one of these currently exploited bits of kits?
8.8.8.8 seems to know about github.com and github.io at least -- so I can work. (Maybe 8.8.8.8 built to resist censorship and therefor also attack?)
If you add these sites ip addresses to your /etc/hosts file locally then you don't have to worry about this for the most part.
[deleted]
https://github.com/ is up for me..
NorthEast USA
maybe I missed something
maybe I missed something
looks like they are all back online now.
[deleted]
I had issues with netflix as well. Netflix.com was resolving but lot of the resources from sub/other domains didn't load.
It looks like Verizon customers continue to have issues resolving Dyn DNS. Wondering if Verizon has actually blocked Dyn requests
Dyn's status page still says they are working the problem. Why would Verizon block access to a downed service?
Not entirely sure. Perhaps some of the DDOS was flowing through to them but that's complete speculation.
I mostly wanted to throw this out there for others who are trying to segment the impact of the issue.
I mostly wanted to throw this out there for others who are trying to segment the impact of the issue.
Sounds like IETF needs to update the DNS standards with 'intelligent' features, so that we can have some consistency.
[deleted]
https://github.com/ is up for me.
North East
github.com itself isn't resolving in some parts of the world. status.github.com is a beautiful default nginx 404 page.
[deleted]
confirmed three (3) coordinated attacks on Dyn's infrastructure today: 11:10 UTC, 13:52 GTM, and the third time is unknown -- Dyn has not specified the time of the 3rd attack today on their dynstatus.com page (likely because the whole thing can be considered ongoing). Can anyone speak to when the 3rd attack began today?
https://ease.apperian.com is down over here
Does anyone have heroku's ip's?
I managed to access twitter from Ecuador using Opera, activating the built-in VPN and setting it to Singapure.
Twitter/GitHub/Soundcloud/etc all just went down for me (UK), also The Verge and The Next Web.
Lots of sites down for me in the UK. I was using my ISPs DNS (Sky), switched to 8.8.8.8 and not much better.
Add 1800 contacts, ynab. Maybe not important to some people but it was how I realized something was up.
Dyn seems down
time to take a look at git-ssb p2p git+"github"
https://git-ssb.celehner.com/%25n92DiQh7ietE%2BR%2BX%2FI403L...
[deleted]
Pulling on AWS using github.com as origin failed had to add the ip to /etc/hosts.
you can watch BGP routes changing (as we speak) here https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16
[deleted]
All of the sites listed are back up for me, and still using Dyn DNS name servers.
ditto for me
I cant get into the PlayStation Network either, I guess its time to go outside.
On the West Coast, Twitter/SoundCloud/GitHub just went down for me...
The outage is attributed to routing 576 Autonomous systems and 3431 prefixes.
Remind me to populate my /etc/hosts file after this blows over.
I have a shop on etsy.com. It is functional at this time (1418 PDT).
No parts of Github are up for me (Ottawa). Over to you, Bitbucket.
Anyone experiencing problems with servers on AWS North Virginia?
Linkedin doesn't seem to load page's contents as well.
Amazon is having an outage. Okta is also down because of this.
[deleted]
10:02 PST in CA and Netflix (and, yes, still Twitter) is down.
Correction: Netflix is "half" loading.
It's 9:58am PST in California and Twitter is still down.
Need to setup a round robin dns - geographically dispersed
Include all website that use Github to sign-in or sign-up
Must be the Russians.
Airbn is also down :(
For us here in Europe/Germany Paymill is also down!
We switched from Dyn to Route53 a few weeks ago … lucky.
CircleCI is also down. As well as github hooks to slack.
Zoho.com, one of my email providers is down as well
gitter as well (https://twitter.com/gitchat)
I have a shop on etsy, and it is functional now.
Need round robin dns - geographically dispersed
That's not the part of the system that's under attack
Braintree is also experiencing these problems.
I have a shop on etsy. It is functional now.
Anyone with issues on AWS in North Virginia?
We've seen zero problems (standard stuff: EC2, S3, RDS, Redshift). I expected that with 1/3 of the internet down something would be broken.
It has been affecting a few offerings all day...specifically WorkMail for me.
Pagerduty down, which is not very helpful.
Critical infrastructure was outsourced and now they've gone down! Yay, the "cloud" and all it's "glory" where some joker with an API thinks that magically means always available.
GitHub, Twitter down for me in Pakistan.
down in Romania too
Sentry (at least web dashboard) is down.
[deleted]
Github still down for me in New York.
U.S DDOS to block Assange cable drop
Github down, can't deploy code!
Quora.com cannot be resolved either.
just add the open DNS you are allset
When will this outage be repaired??
Looks like unsplash.com is down too
npmjs.com seems to be down as well.
Mongo Cloud is having trouble too!
When will outage be restored??
groupme.com is down in nyc... i should memorize friends' phone numbers...
Paypal
Twitter still down - Ohio
Also down:
sie.sps.columbia.edu/enroll
sie.sps.columbia.edu/enroll
GitHub seems fine for me.
Twitter still down Ohio
Namecheap is also down.
Heroku is down as well.
thenextweb.com seems to be down as well (connecting from DC)
Runescape is also down
Okta seems to be down.
my.wpengine.com is also down. The public site is still up.
launchpad.net seems to be down.
edit: Using Google public DNS fixes things.
edit: Using Google public DNS fixes things.
Twitter down again...
Intercom as well :(
www.sonatype.com also down (at least from Germany)
a bunch of services using fastly are also impacted
zendesk admin panel is down here in brazil too
Ticketfly.com. Damn I really wanted to buy die Antwoord tickets
www.sonatype.com also down (from Germany)
PubNub is down.
python.org and cpan.org are down.
i have highly available biscuits
roblox.com is offline aswell.
box.com seems to be down too.
LinkedIn?
roblox.com is offline too.
twitter down again now....
[deleted]
It's Hillary's fault...
poop.bike is still up
deck.news also down
Rollbar!!!! noooooo!!!!!
dot.
[deleted]
[deleted]
Twitter, Github, Quora, Soundcloud, Etsy, PayPal, Heroku, The Guardian, WhatsApp and numerous others still down as of 1:00 pm PT.
Amazon and Spotify are still up.
Amazon and Spotify are still up.
Playstation network down last 4 hours. This is in the UK don't know about the rest of the world.
Status:
https://status.playstation.com/en-gb/
Works fine from Asia.
(When asking e.g. ns1.p34.dynect.net directly.)
(When asking e.g. ns1.p34.dynect.net directly.)
add moz.com
[deleted]
Same thing happened with hover last week... Same attacker perhaps? Maybe hover was just a test drive of their DNS DDoS machinery?
https://twitter.com/HoverStatus/status/786296893013766144
https://twitter.com/HoverStatus/status/786296893013766144
[deleted]
- DYN
- Twitter
- Etsy
- Github
- soundcloud
- spotify
- heroku
- pagerduty
- shopify
- intercom (app, not landing page)
Note that if these sites seem to be up to you, it's likely that your machine has cached the DNS response for these sites.
Some of these sites seem to work when using a UK VPN