Skype Outage – I run a supernode and here’s what I think happened(blogs.voxeo.com)
blogs.voxeo.com
Skype Outage – I run a supernode and here’s what I think happened
http://blogs.voxeo.com/voxeotalks/2010/12/23/skype-outage-i-run-a-supernode-and-heres-what-i-think-happened/
44 comments
Or, maybe this will cause users to take Jabber/XMPP more seriously and stop using proprietary technology for corporate IM.
I've worked at places where management is completely gaga over Skype and would push me to support it despite the fact that I had no ability to block spam, troubleshoot messaging problems or integrate our IM system into existing Asterisk, SSO, monitoring and collaboration solutions.
IMHO, Openfire is far more flexible, extensible, secure, reliable and most importantly - manageable as a service.
http://www.igniterealtime.org/projects/openfire/
disclaimer: I do not work for Ignite and have no vested interest in their business.
I've worked at places where management is completely gaga over Skype and would push me to support it despite the fact that I had no ability to block spam, troubleshoot messaging problems or integrate our IM system into existing Asterisk, SSO, monitoring and collaboration solutions.
IMHO, Openfire is far more flexible, extensible, secure, reliable and most importantly - manageable as a service.
http://www.igniterealtime.org/projects/openfire/
disclaimer: I do not work for Ignite and have no vested interest in their business.
When an XMPP solution that's as easy to use as Skype comes out I'll be all for this. As it stands, even Google's video chat is much more difficult for end-users to install and use.
I used to use Ekiga for video chat and it was much shoddier than Skype, constantly dropping calls, refusing to release the audio or video device so that we couldn't call back, and other serious bugs. Skype "just works".
I really hope that someone comes up with a decent free software competitor, but it doesn't really exist at this point. The fastest way to solve this problem would be for Skype to become free software.
I used to use Ekiga for video chat and it was much shoddier than Skype, constantly dropping calls, refusing to release the audio or video device so that we couldn't call back, and other serious bugs. Skype "just works".
I really hope that someone comes up with a decent free software competitor, but it doesn't really exist at this point. The fastest way to solve this problem would be for Skype to become free software.
Agreed, I've always thought that Skype would one day roll out an enterprise type "Skype Server" that would explicitly make performance better within corporations.
Why would they want to fragment the largest botnet in history? :)
They've already said that they're going to open source the client (http://blogs.skype.com/linux/2009/11/skype_open_source.html)... they just haven't actually done anything about it yet.
They'd better get a move on though, otherwise someone else (http://www.enrupt.com/index.php) is going to do it for them: http://www.enrupt.com/index.php/2010/07/07/skype-biggest-sec...
The trouble is that Skype is so closed and has been seemingly uninterested in open source and Linux, it's hard to find open source dev's who are interested in developing Skype related stuff.
They'd better get a move on though, otherwise someone else (http://www.enrupt.com/index.php) is going to do it for them: http://www.enrupt.com/index.php/2010/07/07/skype-biggest-sec...
The trouble is that Skype is so closed and has been seemingly uninterested in open source and Linux, it's hard to find open source dev's who are interested in developing Skype related stuff.
Their opening of the client is more of an opening of the interface -- all of the real work will still be performed by a proprietary blob that the open interface interacts with. I don't think that will be very useful. I'd prefer if they released something that was _really_ open-source.
It would be more useful then the current state of affairs. With a proprietary blob to talk to, it would be possible for people to do things like write text-based clients in curses, or the suggested 'server' version of Skype specifically for running as a super node or relay node.
We don't know how flexible the blob they give us will be. Since the current interface has no provision for toggling or dealing with one's status on the P2P network, there's no reason to believe that that will necessarily be a feature that the blob exports; Skype may still keep the network communications totally opaque, all handled behind the black magic of the blob, which would simply export the basics like contact list, etc.
Not necessarily. We already know that things like supernodes are selected based on network conditions. The 'server' interface could just be a reduction of dependencies so that you could run it as a daemon process on a server. The Skype Network would then select it as a super node so long as the server had the right conditions (no NAT, no firewall, fat pipe, etc)
It presumably would still need a Skype login, but that could also be used to control it. Accepting commands from a white-list of Skype Ids.
It presumably would still need a Skype login, but that could also be used to control it. Accepting commands from a white-list of Skype Ids.
Just checked my Windows event logs... as I also run a supernode and yup, one Application Error at 2010-12-22 15:51:10 from Skype.
Tech data shows version 5:
Tech data shows version 5:
Faulting application name: Skype.exe, version: 5.0.0.152, time stamp: 0x4cb31516
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdbdf
Exception code: 0xe0fafafa
Fault offset: 0x0000b727
Faulting process id: 0xd20
Faulting application start time: 0x01cba1ea3b70b5ef
Faulting application path: C:\Program Files (x86)\Skype\Phone\Skype.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 4b2781d7-0de3-11e0-bbdc-005056c00008
Nice to see the CEO of a large company that knows how to walk the talk. I've also seen him post on HN. Does Steve Ballmer know the difference between a micro and monolithic kernel?
I was thinking the same thing. What other big company CEO's would even think of investigating why their Skype was more jittery in some hotels, let alone actually know how to do it.
I am also surprised that a company as large as his, uses Skype instant messaging for their communication.
We're a very distributed company, with some projects being worked on by people in a dozen locations, so some form of IM is needed.
I'm a big XMPP proponent (and Voxeo acquired my xmpp-based company) and we do use XMPP for a lot of things inside Voxeo (like Phono, our jquery softphone: http://phono.com/).
Every voxeo employee has an XMPP ID, but they don't tend to get used. People gravitate to Skype naturally. It's the user experience on the client side of XMPP that keeps people from using it extensively for internal communications. At any given time I've got 40+ group chats going on and we create and destroy group chats many times a week for specific needs. The ability to make a voice call to one or more participants of the chat is also a winner.
The fact that everyone on the planet has a Skype ID also helps. We can easily pull partners and customers into chats as needed. They're already on Skype, so we don't need to teach them anything new.
It's a lesson in usability, certainly. Create a client that allows users, without any assistance, to create groups, move seamlessly between text, voice, and video, and has a foolproof signup and setup process, and folks will use it.
I'm a big XMPP proponent (and Voxeo acquired my xmpp-based company) and we do use XMPP for a lot of things inside Voxeo (like Phono, our jquery softphone: http://phono.com/).
Every voxeo employee has an XMPP ID, but they don't tend to get used. People gravitate to Skype naturally. It's the user experience on the client side of XMPP that keeps people from using it extensively for internal communications. At any given time I've got 40+ group chats going on and we create and destroy group chats many times a week for specific needs. The ability to make a voice call to one or more participants of the chat is also a winner.
The fact that everyone on the planet has a Skype ID also helps. We can easily pull partners and customers into chats as needed. They're already on Skype, so we don't need to teach them anything new.
It's a lesson in usability, certainly. Create a client that allows users, without any assistance, to create groups, move seamlessly between text, voice, and video, and has a foolproof signup and setup process, and folks will use it.
tl,dr;
a forced software upgrade kept crashing all skype windows clients, taking down supernodes as well.
[deleted]
Well, the big question is not if Skype went down becase lots of supernodes crashed. The question is why they crashed, and if the trigger was external. There are messages on p2p-hackers list arguing just that:
I have evidence that suggests this is being done
by a ddos attack on the supernodes' object list
cmd parameter.I had the same issue: Shortly before Skype went completely offline, my Windows Skype client suddenly started crashing every few minutes. The Mac client didn't have that problem.
Some bug in the new version is probably the likeliest explanation, but if someone deliberately attacked the Skype network, it would probably look similar, right?
Some bug in the new version is probably the likeliest explanation, but if someone deliberately attacked the Skype network, it would probably look similar, right?
...or vice versa. a bug could easily look like an attack.
You're not going to get full client crashes without a major bug. Some amount of attack is possible but I've been getting crashes for a couple weeks now and skype should have fixed things.
Nice write up, pretty interesting that Skype has not come out and said anything about WTF happened. Also ironic that their security blog's last posting it titled "The importance of updating" :) ( http://blogs.skype.com/garage/ )
-John
-John
A wonderful reason to have pre-release testers for every piece of software that communicates with other software, any time you plan to push updates. With pulls, your early updaters are your test beds; with push, you have to create your testing groups.
The problem with a big adaptive p2p network is that it is close to impossible to test under real production loads.
My guess is that the average supernode is on something like 6M/1.5M DSL. This is easy to simulate.
Now, if most of your nodes are on machines directly connected to 10GbE, then that's a problem. But most Skype nodes aren't. (I do imagine there are a few connected to 100M+ connections. But only a few.)
Now, if most of your nodes are on machines directly connected to 10GbE, then that's a problem. But most Skype nodes aren't. (I do imagine there are a few connected to 100M+ connections. But only a few.)
I imagine quite a lot in countries like Sweden. I had 100Mbit in my apartment for cheaper than my shitty comcast which barely breaks 5.
They could have setup their 'mega-supernodes' running the new client in a production environment, so that if it crashed horribly, it wasn't a client machine that failed (or their entire network).
Upgrading (relatively) untested software network-wide on what is essentially their critical infrastructure is bad news. If this didn't happen now, it would happen sometime. It was just a question of when.
Upgrading (relatively) untested software network-wide on what is essentially their critical infrastructure is bad news. If this didn't happen now, it would happen sometime. It was just a question of when.
Push software is even better about testing: you can forcibly upgrade 1% of installs, monitor them intensely, and let the (non-upgraded) system executive roll back the version if something bad happens. The system executive can even automatically roll back if it loses contact with the vendor, in case the upgrade goes so horribly wrong that it brings down networking.
Are Skype relying on third parties with no contractual relationship for an essential component of the network?
Yup. But they do mention it when installing. You can also disable it.
If you are not firewalled or NATd you are a supernode automatically - on the plus side you get better sound quality for voice calls.
Normally the bandwidth used is pretty low since you are mostly forwarding text messages.
When they said mega-supernodes they meant machines controlled by them that do nothing else, and are on high bandwidth connections. (My bet is lots of amazon instances.)
If you are not firewalled or NATd you are a supernode automatically - on the plus side you get better sound quality for voice calls.
Normally the bandwidth used is pretty low since you are mostly forwarding text messages.
When they said mega-supernodes they meant machines controlled by them that do nothing else, and are on high bandwidth connections. (My bet is lots of amazon instances.)
I've looked around for the ability to turn off supernode on Linux and Mac clients for a while but have come up empty. Do you know which section of the preferences/settings it's supposed to be in?
I don't think there's a UI element for it, but you have to add a registry key, see here:
https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...
https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...
Skype's EULA pretty clearly states that this is the case:
3.3 Utilization of Your Computer: Skype Software may utilize the processor and bandwidth of the computer (or other applicable device) You are utilizing, for the limited purpose of facilitating the communication between Skype Software users.
3.3 Utilization of Your Computer: Skype Software may utilize the processor and bandwidth of the computer (or other applicable device) You are utilizing, for the limited purpose of facilitating the communication between Skype Software users.
I, too, run a Skype supernode - completely unvoluntarily, barring the possibility of something hidden behind an asterisk in the EULA. Does anyone know if there is a way to "opt out" of this? Everytime I start skype up it makes a million connections to everywhere and without even having a call on the line my measly 2.5 mbps upstream is permanently choked.
https://www.cs.uwaterloo.ca/twiki/view/CF/SkypeConfiguration...
Found this link lower down :-)
Found this link lower down :-)
Sorry, I should've mentioned that I'm an OS X user :)
With that said, I've had a look-see in Skype's .plist and its various Application Support files, but oddly there's nothing "grep'able" to be found anywhere...
With that said, I've had a look-see in Skype's .plist and its various Application Support files, but oddly there's nothing "grep'able" to be found anywhere...
I seriously doubt the Skype supernodes actually run in regular clients on people's desktops. I'm pretty sure these are special servers placed strategically on the net.
If you see thousands of connections on your Skype at home then there probably is some weird P2P problem going on.
If you see thousands of connections on your Skype at home then there probably is some weird P2P problem going on.
I'm sorry to be the one who has to point this out, but you have no idea what you are talking about.
http://www.skype.com/intl/en-us/security/universities/
http://www.skype.com/intl/en-us/security/universities/
Also, hopefully this will teach Skype to do more with open-source. I really hope they open the client up. This bug may have been caught, and things would definitely have turned out differently if Skype ran freely on other platforms. Maybe someone could even factor out a "Skype server" instead of an exclusive policy of client supernodes. Even serious torrenters rent a server somewhere to host their torrents -- P2P doesn't have to be strictly consumer-level connection, and really shouldn't be.