We got hacked(plumbr.eu)
plumbr.eu
We got hacked
http://www.plumbr.eu/blog/we-got-hacked
24 comments
This whole post reads like "How Not to Use AWS" instruction guide.
If you're running anything important on a single EC2 instance, you're doing it wrong. If you're logging in and manually configuring an EC2 instance, you're also doing it wrong.
If you're running anything important on a single EC2 instance, you're doing it wrong. If you're logging in and manually configuring an EC2 instance, you're also doing it wrong.
That's helpful, cheers.
How about letting us lesser plebs know how to do it properly then?
How about letting us lesser plebs know how to do it properly then?
This is probably beyond the scope of such a thread, but configuration automation via Chef or Puppet is where you'd start.
You really think that's the best place to start?
I would begin with a figuring out a more secure, desired configuration before trying to automate it. (Especially given, IMHO, the very steep learning curve for Chef & Puppet)
I would begin with a figuring out a more secure, desired configuration before trying to automate it. (Especially given, IMHO, the very steep learning curve for Chef & Puppet)
Puppet and Chef are great tools, but my first entry into AWS management was a little more attainable without having to learn anything new.
I simply set up the instance to automatically download the current codebase from our 'production' repo on every boot, automatically install requirements, run database migrations, etc. Then freeze that AMI as the production AMI.
If you migrate your databases off of the instance to RDS, don't use EBS, and manage user uploads and static assets to go straight to S3, then what you end up with a very easily load balanceable configuration.
To set up load balancing, just spin up two of those instances, attach them to an elastic load balancer, attach the load balancer to your elastic IP, then you ought to be more redundant and management free. (Note, this doesn't fix your security issues, though you could very easily bake a nightly apt-get run for security updates into that image).
After that, you want to look into Fabric (or language-specific alternatives if they exist) to allow for remote management of your machines for things like bouncing web services, pulling new code deploys, etc.
If you want to sidestep all this management altogether, I'd strongly suggest looking into something like dotcloud - http://dotcloud.com/ - which effectively does all this for you.
I simply set up the instance to automatically download the current codebase from our 'production' repo on every boot, automatically install requirements, run database migrations, etc. Then freeze that AMI as the production AMI.
If you migrate your databases off of the instance to RDS, don't use EBS, and manage user uploads and static assets to go straight to S3, then what you end up with a very easily load balanceable configuration.
To set up load balancing, just spin up two of those instances, attach them to an elastic load balancer, attach the load balancer to your elastic IP, then you ought to be more redundant and management free. (Note, this doesn't fix your security issues, though you could very easily bake a nightly apt-get run for security updates into that image).
After that, you want to look into Fabric (or language-specific alternatives if they exist) to allow for remote management of your machines for things like bouncing web services, pulling new code deploys, etc.
If you want to sidestep all this management altogether, I'd strongly suggest looking into something like dotcloud - http://dotcloud.com/ - which effectively does all this for you.
That may be true from a scalability and maintenance perspective. I don't see how those criticisms relate to security. Automatically-maintained hosts are no less subject to exploit...
Clear and open about it. Good.
Asked my team to review their Jenkins passwords and Jenkins user rights...
I don't know if you're doing this, but I think it's a bad idea to leave Jenkins publicly accessible. Indeed, IMHO, it's a bad idea to leave stuff that should not be accessible by the general public publicly accessible. Especially things that have access to your code.
Do ask your team to review passwords and user rights, but also put this service and others like it behind a VPN. Then both the VPN server and Jenkins will have to have holes simultaneously before you get hacked.
Do ask your team to review passwords and user rights, but also put this service and others like it behind a VPN. Then both the VPN server and Jenkins will have to have holes simultaneously before you get hacked.
As the main hero of that story, I can assure you that we are working on VPN setup right now. I guess this is not in top-10 checklist for a startup.
Should've coded the whole thing in Makumba and this would've never happened!!one
"C source named backdoor.h"
Why would an attacker leave a file with that name? Seems like misdirection or a mistake. How are you going to verify they did nothing else?
Was the original attack via jenkins? all it says some vague privilege escalation was used to upload c file. what?
I believe the issue was related to the fact that the user running Jenkins was a full passwordless sudo user.
Maybe the attacker used the groovy console too...
Maybe the attacker used the groovy console too...
No the author seems to indicate that it was on their application code and an attacker was able to get OS access, and the attacker subsequently replaced the ssh service with one that instead ignores login attempts and harvests the username/password pairs.
It seems to have been a coincidence that their Jenkins service was not secured.
It seems to have been a coincidence that their Jenkins service was not secured.
First of all, your stupid copy/paste script has completely broken copy-paste. When I copy, I get nothing except the spam you're trying to inject, so congrats on managing to piss me off right off the bat.
Second, why even list A, B, or C? Whoever thought running Jenkins as a passwordless sudo user shouldn't be doing sysops. Why was Jenkins even public facing? At worst, put it behind a VPN.
Second, why even list A, B, or C? Whoever thought running Jenkins as a passwordless sudo user shouldn't be doing sysops. Why was Jenkins even public facing? At worst, put it behind a VPN.
So... if the attacker would not have had deactivated SSH by accident they would not have noticed?
[deleted]
I see irony in the "getting hacked" part. But seriously, if he is more experienced, then why do they have machines unpatched like that for 6 months?
"it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks"
It is rather bold to be posting about how you just got hacked by not applying simple patches, and to then bash the "kid" who couldn't install a proper rootkit.