My server's been hacked(serverfault.com)
serverfault.com
My server's been hacked
http://serverfault.com/questions/218005/my-servers-been-hacked-emergency
13 comments
"Carefully explaining your problem is half of the solution."
I basically agree with all the advice given, but the easy-fix-guy in me realizes that it's most likely just replacing a weak password and deleting an easily found botnetscript, and he'll can have the box up and running again.
I guess that's the difference between fixing my private boxes with no sensitive informastion and fixing this guy's 600-site serving box:)
I guess that's the difference between fixing my private boxes with no sensitive informastion and fixing this guy's 600-site serving box:)
just a question from someone who probably also would've asked this "incomplete" question (e.g. my server's been hacked what do i do): what kind of other information should be provided? sys logs apache logs ?
From the article... I'm assuming you've understood all the issues that led to the successful intrusion in the first place before you even start this section. I don't want to overstate the case but if you haven't done that first then you really do need to. Sorry.
It would be better if you learned to read your logs on a regular basis, to know what the baseline activities and patterns for your servers look like, configured locally-appropriate log settings for your environment (and quite possibly with more detailed settings than the logging defaults), and to then look for oddities or patterns or connections or unusual events on a daily or weekly basis, or unusual logins, or whatever particular hole(s) the attackers have exercised.
Then you look for file system modifications or unexpected server activity patterns, etc. And match these against the log activity and firewall traffic.
Posting everything from your logs is going to drown folks in data. And may well end up exposing more about your security configuration than you had intended.
Follow what the article suggests here... What steps can you take to reduce the probability of an attack being successful?
Going backwards to where a server configuration might be defensible and debuggable and recoverable. Look to a move to certificate-based authentication (or passphrases and certificates) and away from traditional passwords, to VPNs rather than open protocols, to sftp and away from ftp, to locked-down web and service directories for services rather than writable directories, to various options with firewall blocks or zen-based or other blacklists combined with tools such as mod_security, to staying current on your software versions, to configuring defenses in depth against common attacks, and to getting your logs where you want it, and the one that tends to get lost...
You want... Good backups of your data.
And preferably with some of these backup copies entirely disconnected from the security domain of your server. Push-based copies or pull-based copies involving remote hosts, for instance.
It would be better if you learned to read your logs on a regular basis, to know what the baseline activities and patterns for your servers look like, configured locally-appropriate log settings for your environment (and quite possibly with more detailed settings than the logging defaults), and to then look for oddities or patterns or connections or unusual events on a daily or weekly basis, or unusual logins, or whatever particular hole(s) the attackers have exercised.
Then you look for file system modifications or unexpected server activity patterns, etc. And match these against the log activity and firewall traffic.
Posting everything from your logs is going to drown folks in data. And may well end up exposing more about your security configuration than you had intended.
Follow what the article suggests here... What steps can you take to reduce the probability of an attack being successful?
Going backwards to where a server configuration might be defensible and debuggable and recoverable. Look to a move to certificate-based authentication (or passphrases and certificates) and away from traditional passwords, to VPNs rather than open protocols, to sftp and away from ftp, to locked-down web and service directories for services rather than writable directories, to various options with firewall blocks or zen-based or other blacklists combined with tools such as mod_security, to staying current on your software versions, to configuring defenses in depth against common attacks, and to getting your logs where you want it, and the one that tends to get lost...
You want... Good backups of your data.
And preferably with some of these backup copies entirely disconnected from the security domain of your server. Push-based copies or pull-based copies involving remote hosts, for instance.
Amazing answer this question has spawned. I particulary like this statement, it somehow puts things into perspective:
"First: understand that the disaster has already happened. This is not the time for denial; it is the time to accept what has happened, to be realistic about it, and to take steps to manage the consequences of the impact."
"First: understand that the disaster has already happened. This is not the time for denial; it is the time to accept what has happened, to be realistic about it, and to take steps to manage the consequences of the impact."
600 sites on a single non-redundant box? Ouch.
For all we know there's a hot spare sitting next to it, but it wouldn't help much to bring up another infected box.
For every person in this guy's situation, I wonder how many others have hacked servers and are totally unaware?