Large scale Internet SSH brute force attacks seem to have stopped here(utcc.utoronto.ca)
utcc.utoronto.ca
Large scale Internet SSH brute force attacks seem to have stopped here
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/SSHBruteForceAttacksNoMoreHere
126 comments
I found disabling aes ciphers alone killed 90% of the bots with an occasional ECC only bot sneaking through. I only have 2-3 options for MAC, KEX and cipher. Certain mobile clients won't work (older embedded ssh library) but others support more modern configs. But I started my journey with [1] and have slowly tweaked and slimmed it down over the years.
https://stribika.github.io/2015/01/04/secure-secure-shell.ht...
https://stribika.github.io/2015/01/04/secure-secure-shell.ht...
> [after] hardening steps [...] most of the bots can't even negotiate a connection
Yep, same here, except I'm using [tinyssh], which organically does not support password-based auth, or any ciphers other than ssh-ed25519, curve25519-sha256, and [email protected].
[tinyssh] https://tinyssh.org/
Yep, same here, except I'm using [tinyssh], which organically does not support password-based auth, or any ciphers other than ssh-ed25519, curve25519-sha256, and [email protected].
[tinyssh] https://tinyssh.org/
>which organically does not support password-based auth
This is the way. Everything else is just gravy imo.
This is the way. Everything else is just gravy imo.
I agree with your post and just want to add that TinySSH also supports [email protected] for post-quantum security.
I'm hoping that the folks running the bots use their bots' failure to negotiate a connection as a filter in a similar, but opposite, way that Nigerian scammers use ridiculously scammy, all caps, emails as a filter. It would significantly reduce log spam if each bot only ever connected once.
Anyone who responds to a Nigerian scam email is much more likely to be able to be successfully scammed. It is more efficient, for the scammer, that less credulous people get filtered out in the first step, so as not to waste time on them.
Similarly, any site where an ssh bot fails connection because the site operator removed e.g., weaker default ciphers is much less likely to support password auth for it to even be possible to have a password brute forced. It is more efficient for the bot operator to just move along to a more likely target.
Anyone who responds to a Nigerian scam email is much more likely to be able to be successfully scammed. It is more efficient, for the scammer, that less credulous people get filtered out in the first step, so as not to waste time on them.
Similarly, any site where an ssh bot fails connection because the site operator removed e.g., weaker default ciphers is much less likely to support password auth for it to even be possible to have a password brute forced. It is more efficient for the bot operator to just move along to a more likely target.
Same for www. Ratchet up the TLS/SSL - https://ssl-config.mozilla.org/ - go for modern and you'll see a lot of failed connections from bots and scanners.
Also, if you don't use any other IP block list, do use DROP from Spamhaus: https://www.spamhaus.org/drop/ - that is small enough that you can run it on the webserver if you don't have much control over your connection to the outside world.
Also, if you don't use any other IP block list, do use DROP from Spamhaus: https://www.spamhaus.org/drop/ - that is small enough that you can run it on the webserver if you don't have much control over your connection to the outside world.
I also do that and even take it a step further for my silly hobby sites. I only accept HTTP/2.0 which blocks 99% of the bots and most search engines. Bing is the only search engine that appears to support H2 in their crawler bots.
It is a violation of the old mantra exemplified by Sendmail - "be kind in what you accept but be strict in what you send" (or words to that effect).
It also looks a bit like a nod to security through obscurity.
However the world is a different place nowadays and rather nasty! I think we should insist on the best standards we have and not simply allow old stuff to carry on working through some form of misplaced altruism.
I have come across some criticism of h2 but I don't know enough about the context and haven't researched that so can't comment but if dumping h1 works for you then I say crack on.
I quietly enjoy watching my HA Proxy logs live and watching things bounce off. I have had an on prem MS Exchange (currently 2016) system that passes PCI-DSS for several years. Quite a few URLs will run really slowly from the outside world and then fail to work. Password testing on the OWA will quietly get handed off to a fake page that not only never lets you login, but gradually gets slower and slower and uses very few actual resources. The real OWA page is firing up all sorts of things in anticipation of you authenticating but the HA Proxy fake login is a C process and tiny.
You call your sites "silly hobby" but they are still interweb facing and if you don't protect them they will cease to be just yours and instead be part of a botnet that's trying to hack me and my customers or stealing your electricity to run cryptocoin generation or used as part of the war against Ukraine.
There is a good chance that a determined and skilled haxxor will run rings around both of us but let's crack on and make ourselves a bit trickier to crack than the average. If we make ourselves tough enough to crack then unless you are a valuable target then they will pass eventually but you will have slowed them down a bit.
It also looks a bit like a nod to security through obscurity.
However the world is a different place nowadays and rather nasty! I think we should insist on the best standards we have and not simply allow old stuff to carry on working through some form of misplaced altruism.
I have come across some criticism of h2 but I don't know enough about the context and haven't researched that so can't comment but if dumping h1 works for you then I say crack on.
I quietly enjoy watching my HA Proxy logs live and watching things bounce off. I have had an on prem MS Exchange (currently 2016) system that passes PCI-DSS for several years. Quite a few URLs will run really slowly from the outside world and then fail to work. Password testing on the OWA will quietly get handed off to a fake page that not only never lets you login, but gradually gets slower and slower and uses very few actual resources. The real OWA page is firing up all sorts of things in anticipation of you authenticating but the HA Proxy fake login is a C process and tiny.
You call your sites "silly hobby" but they are still interweb facing and if you don't protect them they will cease to be just yours and instead be part of a botnet that's trying to hack me and my customers or stealing your electricity to run cryptocoin generation or used as part of the war against Ukraine.
There is a good chance that a determined and skilled haxxor will run rings around both of us but let's crack on and make ourselves a bit trickier to crack than the average. If we make ourselves tough enough to crack then unless you are a valuable target then they will pass eventually but you will have slowed them down a bit.
There is a good chance that a determined and skilled haxxor will run rings around both of us
I agree. I do silly things like this to get rid of the bot noise and to keep snooping bots like Discord/Steam off my files. I doubt any corporation would mimic my practices. Most of my hobby sites these days are static pre-compressed content and block anything other than HEAD and GET requests that match a regex patterns.
All of that said I think that you are right. A targeted attack of my server providers could net console access and I would be very surprised if that console access didn't already provide unfettered access to assorted agencies in the name of lawful intercept. At least that was the case when I was in the wireless industry.
I agree. I do silly things like this to get rid of the bot noise and to keep snooping bots like Discord/Steam off my files. I doubt any corporation would mimic my practices. Most of my hobby sites these days are static pre-compressed content and block anything other than HEAD and GET requests that match a regex patterns.
All of that said I think that you are right. A targeted attack of my server providers could net console access and I would be very surprised if that console access didn't already provide unfettered access to assorted agencies in the name of lawful intercept. At least that was the case when I was in the wireless industry.
"I do silly things like this"
I know what you mean but it really isn't silly, it is socially responsible. You are already doing the right thing but thought of it as a hobby but it really isn't.
The thing about the internet is that you can live in the UK or the US or wherever and have someone from the Russian Federation or the PRC off of China or a NORK or simply a common criminal from anywhere rock up and knock on your website's door and try to behave like a parasite.
Think of these little darlings as parasites and it becomes obvious why you should put some effort in to keeping the buggers out.
I know what you mean but it really isn't silly, it is socially responsible. You are already doing the right thing but thought of it as a hobby but it really isn't.
The thing about the internet is that you can live in the UK or the US or wherever and have someone from the Russian Federation or the PRC off of China or a NORK or simply a common criminal from anywhere rock up and knock on your website's door and try to behave like a parasite.
Think of these little darlings as parasites and it becomes obvious why you should put some effort in to keeping the buggers out.
Filtering by client works wonders in servers that support it. Nobody recompiles libssh.
...be sure to do this locally where one has console access...
I found I was able to restart sshd without bouncing my ssh connection. I think this is because sshd forks off children? Of course I made sure everything was working before killing the existing connection!
Thanks for the link to the audit service! We went from "F" to "A+"...
I found I was able to restart sshd without bouncing my ssh connection. I think this is because sshd forks off children? Of course I made sure everything was working before killing the existing connection!
Thanks for the link to the audit service! We went from "F" to "A+"...
> Another interesting side effect is that my ssh negotiation is much faster on high latency connections after the hardening.
why?
why?
Watching the connection in verbose it appears the cipher list from both the client and server side being significantly shorter and matching on both sides speeds up the negotiation. There may also be other factors but that is what sticks out to me. I've not dug deep enough into this to see if utilizing different ciphers is also part of the speed-up.
RFC 4253 says about the key exchange:
> Each side MAY guess which algorithm the other side is using, and MAY send an initial key exchange packet according to the algorithm, if appropriate for the preferred method.
If you only support one method, that guess will always be correct, and you save a round trip.
> Each side MAY guess which algorithm the other side is using, and MAY send an initial key exchange packet according to the algorithm, if appropriate for the preferred method.
If you only support one method, that guess will always be correct, and you save a round trip.
[deleted]
Initially it sounded implausible that a small size increase could noticably raise connection establishment, but it's possible the small size increase causes MTU to be exceeded, and thus an additional roundtrip.
Is it a slow bandwidth connection as well as low latency? It could be because curve25519/ed25519 are smaller packet sizes than other algorithms (and also just the length of the list of algorithms in the kexinit packet).
I would have expected the round trip count at the SSH level to be the same, but there could be more TCP round trips for window adjustment.
Plugging my own software, Dropbear's dbclient is a bit quicker at high latency - "ssh [email protected]" is 3 seconds (from Perth, Australia) vs 1.5 secs with dbclient. It makes use of the "guess kex" feature that most clients don't, and sends other packets sooner. Of course dbclient doesn't have all the useful features of OpenSSH, and isn't as performant at high bandwidth.
I would have expected the round trip count at the SSH level to be the same, but there could be more TCP round trips for window adjustment.
Plugging my own software, Dropbear's dbclient is a bit quicker at high latency - "ssh [email protected]" is 3 seconds (from Perth, Australia) vs 1.5 secs with dbclient. It makes use of the "guess kex" feature that most clients don't, and sends other packets sooner. Of course dbclient doesn't have all the useful features of OpenSSH, and isn't as performant at high bandwidth.
Fewer roundtrips to exchange less data, multiplied by the latency per round trip => lower total time.
You are correct about the libraries not supporting modern ssh :)
I ran a bunch of honeypots for a while and grabbed hassh (like ja3 but for ssh) signatures, most bots are using old fucking libssh/libssh2/paramiko that simply can't talk to modern hosts.
I ran a bunch of honeypots for a while and grabbed hassh (like ja3 but for ssh) signatures, most bots are using old fucking libssh/libssh2/paramiko that simply can't talk to modern hosts.
I tried that tool against my Raspberry Pi which should have a reasonably-secure SSH configuration and I got a lot of warnings about:
> NIST P-curves are possibly back-doored by the U.S. National Security Agency
Is there any evidence of this? This seems extremely paranoid.
> NIST P-curves are possibly back-doored by the U.S. National Security Agency
Is there any evidence of this? This seems extremely paranoid.
Is there any evidence of this? This seems extremely paranoid.
I have not seen any evidence, just discussions. [1][2]
[1] - https://crypto.stackexchange.com/questions/10263/should-we-t...
[2] - https://arstechnica.com/information-technology/2015/01/nsa-o...
I have not seen any evidence, just discussions. [1][2]
[1] - https://crypto.stackexchange.com/questions/10263/should-we-t...
[2] - https://arstechnica.com/information-technology/2015/01/nsa-o...
No smoking gun. But beyond Snowden's allegations, (a) the NSA has a history of very similar shenanigans and (b) there are readily available alternatives, so why risk it?
Fail2ban blocked 1087 ip addresses in the last week, which seems normal.
I reset it and it has blocked eleven ip addresses in the last hour, mainly China and Digital Ocean as usual.
Just to see what happens, I'v tried sending abuse reports about ssh brute force, vnc brute force and phishing sites, by the standard method of doing a whois lookup on the ip for the abuse email address.
Some server and web hosting companies take the inconvenient approach of having an email auto-reply that says "we ignore all emailed abuse reports, you must use this web form", sometimes requiring a captcha.
When I reported a load of boxes attempting brute force logins, most complaints disappeared into the void.
I got a few responses from virtual server providers saying "no response from the customer after two weeks so we shut down the box" and one CC:ed email that appeared to be from an end user saying "we have reinstalled the box and changed the password."
I reset it and it has blocked eleven ip addresses in the last hour, mainly China and Digital Ocean as usual.
Just to see what happens, I'v tried sending abuse reports about ssh brute force, vnc brute force and phishing sites, by the standard method of doing a whois lookup on the ip for the abuse email address.
Some server and web hosting companies take the inconvenient approach of having an email auto-reply that says "we ignore all emailed abuse reports, you must use this web form", sometimes requiring a captcha.
When I reported a load of boxes attempting brute force logins, most complaints disappeared into the void.
I got a few responses from virtual server providers saying "no response from the customer after two weeks so we shut down the box" and one CC:ed email that appeared to be from an end user saying "we have reinstalled the box and changed the password."
I had a DigitalOcean droplet running Selenium grid that I didn't secure properly. I got an email that Sony Entertainment had reported my IP for botting login attempts in the Playstation store. I think I had 48 hours to respond to DigitalOcean.
Shut down Selenium right away, checked the logs, yep someone had taken over my selenium grid. It was just a small personal project on a sever I had forgot was still running, but at least they took reports seriously when my account was causing things.
Shut down Selenium right away, checked the logs, yep someone had taken over my selenium grid. It was just a small personal project on a sever I had forgot was still running, but at least they took reports seriously when my account was causing things.
Honestly, I don't understand why people make reporting abuse so hard/labour-intensive.
It is trivial to record netflow data (and most networks do that already), and then verify incoming abuse reports against those records.
It is trivial to record netflow data (and most networks do that already), and then verify incoming abuse reports against those records.
Because processing abuse tickets is hard and labor intensive. If reporting abuse to a host lead to shutdowns as swiftly as YouTube DMCA it would become a denial of service attack method. Or the abuse ticket queue would be flooded and it would take even longer to cut off bad actors. Cloud services are notorious for abuse traffic. If they were required to raise KYC requirements it might mean the end of $5/mo. VPS. Finally hosting services have an inherent conflict of interest between making money selling service and cancelling someone's account for abuse.
[deleted]
the counterpoint is that the antagonists can abuse the reporting mechanisms, so some friction is necessary
[deleted]
is there a chance of forming some kind of a community fail2ban blocklist? I guess trusting the contributors and admins is the hard part here and that’s why spam lists are a double edged sword?
As someone else mentions, CrowdSec can do just that. It's FOSS and can act as a modern Fail2Ban replacement that can detect all sorts of attacks - in this case ssh bruteforce/slow brute force attacks - and shares very basic information about those attacks (source ip, timestamp, which attack) with everyone else.
So in that way everybody using CrowdSec are helping each other out. More information at https://crowdsec.net.
Disclaimer: I am head of community at CrowdSec so feel free to ask me any questions you may have here or join our Discord at https://dicord.gg/crowdsec.
Thank you! super cool! how do you solve the trust issue? (e.g. someone reporting their competitor ips as attackers, or whitelisting false positives/appeals)
The CrowdSec folks have something similar to that:
https://crowdsec.net/ https://github.com/crowdsecurity/crowdsec
https://crowdsec.net/ https://github.com/crowdsecurity/crowdsec
peter hessler had an interesting system where the blacklists were distributed via bgp. It sounds weird at first but the more I think about it the more it makes sense. delivering routes(or in this case anti routes) is bgp's core mission.
Unfortunately he shut down his bgp spam route sender last year.
http://www.bgp-spamd.net/
Unfortunately he shut down his bgp spam route sender last year.
http://www.bgp-spamd.net/
That's basically what DNS RBLs are.
Well that and sane people just disable password auth. fail2ban is a 90s sysadmin solution to a nonexistent problem.
Fail2Ban can do more than SSH. Any log that can be parsed and has a useful remote IP can work.
I have it scanning my Ubiquiti NVR logs, I modified Tomcat to log the remote IP from my reverse proxy. If anyone tries to log into my NVR three times then Fail2Ban adds the IP to a permanent blocklist on my OpnSense firewall and then HAProxy kills the TCP connection. They can't even ping after that.
I have it scanning my Ubiquiti NVR logs, I modified Tomcat to log the remote IP from my reverse proxy. If anyone tries to log into my NVR three times then Fail2Ban adds the IP to a permanent blocklist on my OpnSense firewall and then HAProxy kills the TCP connection. They can't even ping after that.
IP reputation is a problem wider then just SSH password brute-forcing - see HTTP(S) brute-force example elsewhere in this thread.
Perhaps VPS providers should monitor for things like huge volumes of outgoing SSH login attempts, the same way they monitor for crypto mining.
[deleted]
I am genuinely surprised that the virtual server providers responded meaningfully to the abuse complaints and took action. Good job reporting the scum.
[deleted]
Doesnt fail2ban have an option that will automatically send out abuse complaints for you?
There's something ironic about that.
A bot tries to mass ssh login on entire subnets.
A bot on one machine blocks the first bot and notified their hosting provider.
A bot on the hosting provider responds with a web form for reporting abuse.
A bot tries to fill out the web form, but is hit with a captcha.
A bot tries to mass ssh login on entire subnets.
A bot on one machine blocks the first bot and notified their hosting provider.
A bot on the hosting provider responds with a web form for reporting abuse.
A bot tries to fill out the web form, but is hit with a captcha.
Its bots all the way down!
Same kind of numbers on my main host - 1325 the last 7 days, 1125 the 7 before that. Bloody annoying.
On a related note, set up a website a few days ago. Brand new domain, newly opened port. But I've rented this VPS for a while so I guess its IP is already on a few lists.
But anyways, it's interesting watching the logs for what I assume are tests of known exploits.
But anyways, it's interesting watching the logs for what I assume are tests of known exploits.
GET /.gitconfig HTTP/1.0" 422 Unprocessable Entity
GET /.git/config HTTP/1.0" 404 Not Found
GET /owa/auth/x.js HTTP/1.0" 404 Not Found
GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.0" 404 Not Found
GET /owa/auth/logon.aspx?url=https%3a%2f%2f1%2fecp%2f HTTP/1.0" 404 Not Found
GET /system_api.php HTTP/1.0" 404 Not Found
GET /c/version.js HTTP/1.0" 404 Not Found
GET /streaming/clients_live.php HTTP/1.0" 404 Not Found
GET /stalker_portal/c/version.js HTTP/1.0" 404 Not Found
GET /stream/live.php HTTP/1.0" 404 Not Found
GET /flu/403.html HTTP/1.0" 404 Not Found
POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.0" 404 Not Found
GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.0" 404 Not Found
GET /backups-dup-lite/dup-installer/main.installer.php HTTP/1.0" 404 Not Found
GET /%24%7B%28%23a%3D%40org.apache.commons.io.IOUtils%40toString%28%40java.lang.Runtime%40getRuntime%28%29.exec%28%22whoami%22%29.getInputStream%28%29%2C%22utf-8%22%29%29.%28%40com.opensymphony.webwork.ServletActionContext%40getResponse%28%29.setHeader%28%22X-Cmd-Response%22%2C%23a%29%29%7D/
That's why I recommend always pointing the default virtual host in the apache or nginx configuration to an empty static site, and making the real site visible only as a named virtual host (requiring the correct Host header), even when the server will be used only for a single site. Most of these automated exploit attempts will never send the correct Host header, and therefore will only see the default virtual host.
With nginx I also set the return code to 444 on the default virtual host, this is not a real status code but instead tells nginx to kill any connections to this vhost at the TCP level.
I have used a default host with a self signed certificate and 444 for while. One advice was to make it support only the NULL cipher, but I did not succeed to do that, don't remember the details now.
However, many scanners still end with a full 400. Either their implemenations are so bad or they intentionally send corrupted requests to try to exploit some vulnerability. I have not digged any deeper.
However, many scanners still end with a full 400. Either their implemenations are so bad or they intentionally send corrupted requests to try to exploit some vulnerability. I have not digged any deeper.
for https, since 1.19.4 you can reject the tls handshake early https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_...
Could you give a pointer to more info on that? I couldn't find more details on such a setup while googling it.
When configuring apache or nginx, one of the virtual hosts you configure is the "default" virtual host, used when the Host header didn't match any of the virtual hosts. IIRC, on nginx, you explicitly say on the virtual host "this is the default", while on apache, it's either the first or the last (forgot which one).
How to configure that virtual host is up to you; the simplest configuration would be to point its document root to an empty directory, and add a couple of access control directives to deny access to it from all IP addresses.
How to configure that virtual host is up to you; the simplest configuration would be to point its document root to an empty directory, and add a couple of access control directives to deny access to it from all IP addresses.
I think this is the (only) place in the Nginx docs that mentions 444: https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#...
Really? I see plenty of automated exploit attempts against my vhosts.
Fail2van with some good nginx filters to ban scans works really well
I've had this, too, and it later turned out that it was the VPS provider itself scanning my new instance for vulnerabilities.
Was it actually the provider scanning, or just a compromised VPS host using that same provider?
It does seem proactive to scan customers' hosts then notify them if exploits are found.
It does seem proactive to scan customers' hosts then notify them if exploits are found.
[deleted]
Yep, I run a barely popular static site, but people do love trying all the attacks on it:
https://gist.github.com/Q726kbXuN/85c947a5d37cb01f72f82318d0...
This is two weeks of 404s
https://gist.github.com/Q726kbXuN/85c947a5d37cb01f72f82318d0...
This is two weeks of 404s
Instead of 404, possibly redirect them to an ad page and retire early? Half joking...
Naw, advertisers are pretty good at detecting bots. Your CPM will quickly drop to 0.
Better redirect to a multi GB Linux ISO download.
Could you somehow stream an infinite stream? Like taping a circle of black paper together in a fax machine and faxing a spammer.
This gives me an idea. Maybe redirect to some old big corp website -- HP, Oracle, IBM? -- where the page they're being redirected to also redirects several times?
have you thought about serving them eicar?
I have served up various memes and annoying graphics on the off chance someone will investigate what they got.
Though, I assume if their scanner doesn't see the response it wants, it just automatically moves on.
Though, I assume if their scanner doesn't see the response it wants, it just automatically moves on.
I have found nginx bad bot blocker [1] very handy for this. I have it setup to respond with 444 and it allows me to add my own rules very easily.
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...
All of these HTTP/1.0 requests look just like a scanner that the security team would run again my internal site. Every day.
Just shows how regularly Microsoft exchange is attacked that this small list has three different exchange attacks.
I run a honeypot network to produce a threatfeed. Over the last month I saw an increase in daily numbers from Aug 7th to 19th. About 30% increase. From the 19th until yesterday I saw a 20% decrease from the peak. Still above average right now slightly.
In fact, it's interesting. The time it takes to open a port and wait for a random attacker isn't measured in minutes. It's seconds. You can do this at home.
>For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything. Besides root, only five login names had more than 100 attempts (and none of them exist here): 'admin', 'user', 'ubuntu', 'debian', and 'pi'. And only three machines saw more than 1,000 attempts (across all targeted login names).
Here's a public threatfeed. http://charles.the-haleys.org/ssh_dico_attack_with_timestamp...
He has had a fair number of attackers in the last week.
I feel like 'stopped' isn't the right word.
In fact, it's interesting. The time it takes to open a port and wait for a random attacker isn't measured in minutes. It's seconds. You can do this at home.
>For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything. Besides root, only five login names had more than 100 attempts (and none of them exist here): 'admin', 'user', 'ubuntu', 'debian', and 'pi'. And only three machines saw more than 1,000 attempts (across all targeted login names).
Here's a public threatfeed. http://charles.the-haleys.org/ssh_dico_attack_with_timestamp...
He has had a fair number of attackers in the last week.
I feel like 'stopped' isn't the right word.
May I humbly suggest https://github.com/skeeto/endlessh for both fun and profit
Still being attacked constantly. I gotta admit that I am a bit confused with the user names chosen:
CISCO, User, adempiere, admin1, adminstrator, alarm, alem, amanda, ansible, apache2, apc, arma3server, as, assembla, azure, azureuser, bamboo, bilbomeakine, bill, blackvoid, carlos, cds, centos, chinochan, cloud, cloudera, codeship, contributor, csgo, csgoserver, csserver, debian, default, demo, deployer, dev, device, devops, docker, dominion, ec2, ec2-user, ecs, elastic, elasticsearch, elsearch, engineer, es, esuser, eurek, for, ftp, ftp_admin, ftp_user, ftpadmin, ftpserver, ftptest, ftpuser, git, gitlab, glassfish, gmodserver, gpadmin, grav, grid, guest, hadoop, hduser, hostmetrics, jboss, jenkins, jira, john, joomla, junkbust, kafka, kevin, kibana, kubernetes, lighthouse, linkl, linkxess, localadmin, mail, marketing, mc, mcserv, mcserver, michael, mike, minecraft, momo, mongodb, netgear, netscreen, nexus, odoo, office, opc, oper, oracle, osm, osmc, pi, port, postgres, pvm, r00t, redmine, rust, rustserver, sanlang, secscan, service, spark, sphinx, squid, steam, steve, suhelper, super, support, svn, svpilot, systemd, systems, systemx, tbnet, teamspeak3, telecomadmin, test, test2, test3, test4, test6, test7, testftp, testuser, tomcat, ts3, ts3bot, ts3server, ubnt, ubuntu, uftp, upload, uploader, user, usuario, uucp, vagrant, vpn, vpnssh, web, webadmin, weblogic, wordpress, wp, wy, xbmc, xinyi, z, zabbix, zerotier-one, zyfwp
I get the basic service names, but what the heck is going on with "z" and "kevin"?
CISCO, User, adempiere, admin1, adminstrator, alarm, alem, amanda, ansible, apache2, apc, arma3server, as, assembla, azure, azureuser, bamboo, bilbomeakine, bill, blackvoid, carlos, cds, centos, chinochan, cloud, cloudera, codeship, contributor, csgo, csgoserver, csserver, debian, default, demo, deployer, dev, device, devops, docker, dominion, ec2, ec2-user, ecs, elastic, elasticsearch, elsearch, engineer, es, esuser, eurek, for, ftp, ftp_admin, ftp_user, ftpadmin, ftpserver, ftptest, ftpuser, git, gitlab, glassfish, gmodserver, gpadmin, grav, grid, guest, hadoop, hduser, hostmetrics, jboss, jenkins, jira, john, joomla, junkbust, kafka, kevin, kibana, kubernetes, lighthouse, linkl, linkxess, localadmin, mail, marketing, mc, mcserv, mcserver, michael, mike, minecraft, momo, mongodb, netgear, netscreen, nexus, odoo, office, opc, oper, oracle, osm, osmc, pi, port, postgres, pvm, r00t, redmine, rust, rustserver, sanlang, secscan, service, spark, sphinx, squid, steam, steve, suhelper, super, support, svn, svpilot, systemd, systems, systemx, tbnet, teamspeak3, telecomadmin, test, test2, test3, test4, test6, test7, testftp, testuser, tomcat, ts3, ts3bot, ts3server, ubnt, ubuntu, uftp, upload, uploader, user, usuario, uucp, vagrant, vpn, vpnssh, web, webadmin, weblogic, wordpress, wp, wy, xbmc, xinyi, z, zabbix, zerotier-one, zyfwp
I get the basic service names, but what the heck is going on with "z" and "kevin"?
Today someone tried violetta, admin123, rian, phoebe, carlos, calla, es, weiguo, bzrx1098ui and many more on one of my servers.
I guess attackers has gotten a hacked password file and hope that some of the users have used the same username and password combinations on other servers.
Searching for bzrx1098ui showed that attackers try with the same logins on many servers. E.g., https://dataplane.org/signals/sshidpw.txt
Most of the passwords on that list are silly, but not all of them E.g.: )w%WLq^3UAwn 75afaf6480ca5f9c214fabb6e3663813 7h4a5n9d0a2oiang@))* 960c3dac4fa81b4204779fd16ad7c954f95942876b9c4fb1a255667a9dbe389d
The last one is used at: https://github.com/tlaverdure/laravel-echo-server/issues/273
and https://www.digitalocean.com/community/tutorials/how-to-secu...
Which mentions that is can be generated as: echo "digital-ocean" | sha256sum
Apparently Digital Ocean was telling people in 2014, that feeding a weak password to a hash function would result in a longer and therefore very strong password.
I am sure there are plenty of people that would use sha256sum("password")= 6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e as a password in a redis-file. But it is not something you would type in every time you ran SSH.
So maybe someone is just trying to use a search engine to find passwords made public.
I guess attackers has gotten a hacked password file and hope that some of the users have used the same username and password combinations on other servers.
Searching for bzrx1098ui showed that attackers try with the same logins on many servers. E.g., https://dataplane.org/signals/sshidpw.txt
Most of the passwords on that list are silly, but not all of them E.g.: )w%WLq^3UAwn 75afaf6480ca5f9c214fabb6e3663813 7h4a5n9d0a2oiang@))* 960c3dac4fa81b4204779fd16ad7c954f95942876b9c4fb1a255667a9dbe389d
The last one is used at: https://github.com/tlaverdure/laravel-echo-server/issues/273
and https://www.digitalocean.com/community/tutorials/how-to-secu...
Which mentions that is can be generated as: echo "digital-ocean" | sha256sum
Apparently Digital Ocean was telling people in 2014, that feeding a weak password to a hash function would result in a longer and therefore very strong password.
I am sure there are plenty of people that would use sha256sum("password")= 6b3a55e0261b0304143f805a24924d0c1c44524821305f31d9277843b8a10f4e as a password in a redis-file. But it is not something you would type in every time you ran SSH.
So maybe someone is just trying to use a search engine to find passwords made public.
Z? Maybe from the russians with their Z symbol[0]? It depends on how recent that Z logging is. And kevin could be a homage to Kevin Mitnick[0]. Or maybe it's common in a certain region? Just my 2 cents.
[0] - https://en.wikipedia.org/wiki/Z_(military_symbol)
[1] - https://en.wikipedia.org/wiki/Kevin_Mitnick
[0] - https://en.wikipedia.org/wiki/Z_(military_symbol)
[1] - https://en.wikipedia.org/wiki/Kevin_Mitnick
ah yes, the elders of the internet:
amanda, bill, carlos, eurek, john, kevin, michael (mike), steve, and zyfwp
amanda, bill, carlos, eurek, john, kevin, michael (mike), steve, and zyfwp
It probably worked once
I've been interested in showing others this type of data related to SSH honeypots. I'm currently running a cowrie honeypot and I put together this Flask web app wrapped over cowrie's SQLite db to display the data it collects.
So far, ~1/4 of the 85,000 or so SSH attempts have been from the same address in Brazil.
If anyone's interested -- https://live-honeypot.com
So far, ~1/4 of the 85,000 or so SSH attempts have been from the same address in Brazil.
If anyone's interested -- https://live-honeypot.com
What happened is they have gotten better at identifying worthy targets and they spray instead of bruteforce. I have dealt with compromised cloud VMs recently enough due to bad ssh password, it was a crypto mining worm, the cloud indeed is a better place to spend your resources as an attacker because most people are lazy enough to allow entire /8s of their cloud provider and because the internet can't reach it, who cares?
We have take the approach that the only ports visible to internet traffic are un-authenticated ones... everything else must traverse ipsec or wireguard. This gives us completely silent ssh ports.
We found strongswan examples pretty useful... it's basically pick your config and copy/paste. Wireguard only has 'one way' to set it up which is a whole 'nother level of simplicity.
This has completely silenced our firewall and ssh logs. Nobody is out blasting ipsec or wireguard connection probes around the internet (for now).
We found strongswan examples pretty useful... it's basically pick your config and copy/paste. Wireguard only has 'one way' to set it up which is a whole 'nother level of simplicity.
This has completely silenced our firewall and ssh logs. Nobody is out blasting ipsec or wireguard connection probes around the internet (for now).
Alternatively, you can hide your ssh port behind port knocking… which I find to be simple and elegant - and requires almost no configuration or tooling.
Funny - I had noticed something similar in the last couple of months, but hadn't though to look in to it more. I was seeing 30,000 to 50,000 attempts a day, but over the last two weeks, it's around 10,000.
Perhaps it's like mining cryptocurrency - all the low hanging fruit has been found, so there's hardly much sense in continuing to waste resources looking for more.
Perhaps it's like mining cryptocurrency - all the low hanging fruit has been found, so there's hardly much sense in continuing to waste resources looking for more.
Is it possible the majority of attempts were actually being run by a single botnet, and it either shut down or the owner changed the attack payload? Perhaps it's just been running for the last few years, with the botmaster apathetic, forgetting about it or maybe even had lost control.
I think you're right about the low-hanging fruit part: it seems unlikely anyone would bother to put serious effort or money (eg: the resources of a botnet) into this type of attack anymore, and the OP seems to support this idea:
> For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything.
I think you're right about the low-hanging fruit part: it seems unlikely anyone would bother to put serious effort or money (eg: the resources of a botnet) into this type of attack anymore, and the OP seems to support this idea:
> For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything.
> Is it possible the majority of attempts were actually being run by a single botnet, and it either shut down or the owner changed the attack payload?
Russia is preoccupied.
Russia is preoccupied.
This is why you should only be using SSH keys instead of a user/pw.
Yes, it may be a sign that SSH users have almost all migrated to keys.
Amazon and other cloud providers using key-based ssh as the entrance to their instances probably pushed it to mainstream.
Amazon and other cloud providers using key-based ssh as the entrance to their instances probably pushed it to mainstream.
If you setup Fail2Ban and/or switch SSH from the default port 22 to something in the low 5 digits, then you should get very few attacks.
Someone above talked about using layered/defense-in-depth approach to avoid zero-days and that's also a good idea, but it's more involved / sometimes impractical.
Someone above talked about using layered/defense-in-depth approach to avoid zero-days and that's also a good idea, but it's more involved / sometimes impractical.
I did a small study on brute force ssh attacks a couple of years back. Switching the port does a really good job dodging brute force attacks, as long as you don't use 2222. Paradoxically, port 2200 appeared to be a safe choice.
I run a port in the high 4 digits, have for a decade at least, and get...zero attacks.
I still see them. They come and go, but are always present at some level.
2022-08-29T17:35:12.617Z [DEBUG] sshlog gen 113.61.219.237 admin admin SSH-2.0-HELLOWORLD
2022-08-29T17:48:17.879Z [DEBUG] sshlog gen 218.92.0.190 root poohbear SSH-2.0-PUTTY
2022-08-29T17:48:18.041Z [DEBUG] sshlog gen 218.92.0.190 root p@ssw0rd3 SSH-2.0-PUTTY
2022-08-29T17:48:18.2Z [DEBUG] sshlog gen 218.92.0.190 root p@ssword! SSH-2.0-PUTTY
2022-08-29T17:50:13.507Z [DEBUG] sshlog gen 185.191.205.92 hl hl SSH-2.0-libssh-0.6.3
2022-08-29T17:52:57.28Z [DEBUG] sshlog gen 138.68.91.192 victoria abc123 SSH-2.0-libssh-0.6.3May I ask how to configure the sshd to generate logs like yours? I searched for it and could not find much information.
It's called sshlog. It's a patch that logs username/password.
https://github.com/62726164/sshlog
https://github.com/62726164/sshlog
Thank you very much for the kind reply. I will try it out on one of my vps servers. Looking to have some fun when performing analysis afterwards. Thanks!
> [...] blocking of only a few IPs is disproportionately effective at stopping brute force SSH attacks here. Also, since we already block Internet logins to 'root', we're in almost no danger. No matter how many times they try, they have literally no chance of success.
This assumes OpenSSH, OpenSSL and other things in between the client and the server don't have ready to exploit security vulnerabilities. These large-scale scanners are a long term investment when there's a zero-day vulnerability to exploit in OpenSSL etc. That's why a layered/defense-in-depth approach is preferable.
This assumes OpenSSH, OpenSSL and other things in between the client and the server don't have ready to exploit security vulnerabilities. These large-scale scanners are a long term investment when there's a zero-day vulnerability to exploit in OpenSSL etc. That's why a layered/defense-in-depth approach is preferable.
I stopped opening port 22 when I moved to Tailscale. Won't go back.
Since wireguard is a thing now wouldn't it be inadvisable for new deployments to put ssh on the bare internet anyway?
Why exactly is wireguard safer than ssh with keys?
This is an interesting question.
I would say at the very least it offers an extra layer of authentication over ssh, so it would require a wireguard bug and ssh bug to cause damage. That alone provides extra security!
Another helpful thing, is that wireguard doesn't respond to invalid connection attempts at all. An invalid connection attempt appears as if you are accessing a port that nothing is listening on! Because of this, you can't detect what port wireguard is listening on, or even if it's running at all, so mass scanning the internet for wireguard listening ports and attempting connections is extremely ineffective.
There are probably other aspects that make it safer. You can read a lot about wireguard here https://www.wireguard.com/papers/wireguard.pdf
I would say at the very least it offers an extra layer of authentication over ssh, so it would require a wireguard bug and ssh bug to cause damage. That alone provides extra security!
Another helpful thing, is that wireguard doesn't respond to invalid connection attempts at all. An invalid connection attempt appears as if you are accessing a port that nothing is listening on! Because of this, you can't detect what port wireguard is listening on, or even if it's running at all, so mass scanning the internet for wireguard listening ports and attempting connections is extremely ineffective.
There are probably other aspects that make it safer. You can read a lot about wireguard here https://www.wireguard.com/papers/wireguard.pdf
Little offtopic:
"One of the things I've learned from this is that targeted blocking of only a few IPs is disproportionately effective at stopping brute force"
This is very much true also for other types of attacks/scan/etc.
Are you blocking Azure/OVH range on firewall ?
"One of the things I've learned from this is that targeted blocking of only a few IPs is disproportionately effective at stopping brute force"
This is very much true also for other types of attacks/scan/etc.
Are you blocking Azure/OVH range on firewall ?
The recall for these is also terrible. it's like going after a mosquito with a bazooka. case in point, we've accidentally blocked entire universities on a shared IP and the entire AWS West region doing this
I still see a good amount, but I think I am better at blocking them through distributed firewall sensors and such. Also, I think there is an opportunity cost. When you hear how lucrative other hacking avenues are, why bother with crappy little accounts that probably dont monetize for that much vs activities that are far more lucrative
https://thecyberwire.com/podcasts/research-saturday/247/note...
The numbers in that story (even at the low end) are eye popping :(
https://thecyberwire.com/podcasts/research-saturday/247/note...
The numbers in that story (even at the low end) are eye popping :(
This isn't my experience at all. Its a constant stream, people try common first names, and they're from lots of different people. Certain cloud websites in particular host the bots. A weird amount of requests come from IP addresses associated with security companies - just enough to be a nuissance for a website that doesn't actually get any traffic and is just for me to test things out on.
To have some fun simply make SSH tarpit to listen on port 22. It slowly sends an endless banner hahaha
https://github.com/Snawoot/ssh-tarpit
Did someone record which username/passwords those bots try? I never bothered with that.
Logging passwords for failed logins is an awful security anti-pattern; please don't do that.
Hint: you'll get quite a few of real passwords with 1-letter typo, written to disk in plan text, and probably copied to 5 different places, possibly some of them external.
I mean, you shouldn't, because you shouldn't have anything, anywhere, using password auth.
Which doesn't mean it's OK to log passwords if you do have password auth, it means it's not OK to have password auth.
Which doesn't mean it's OK to log passwords if you do have password auth, it means it's not OK to have password auth.
Using passwords for SSH is also bad security.
[deleted]
I used to do that on my honeypots.
I'll do it again when I set up new ones, it was neat seeing a new credential pair show up from one attacker only for a while, then show up from more over time as knowledge of some appliances default creds spread across the hacker world.
I'll do it again when I set up new ones, it was neat seeing a new credential pair show up from one attacker only for a while, then show up from more over time as knowledge of some appliances default creds spread across the hacker world.
Hm, that's probably pretty run-of-the-mill default user/password combinations, unless someone specifically targets you?
Back when I ran cowrie and ran statistics my data was pretty similar. Root was getting blown up in the list because of the mirai botnet.
Just yesterday I recently exposed SSH on a machine that had never had it exposed, on a router that had port 22 closed. Over the few hours it was open, my access log had about a hundred attempts against it.
It's not hard to scan the entire IPv4 address space, I doubt there is much use for lists of accessible servers.
"Not hard", to say the least, yeah:
https://github.com/robertdavidgraham/masscan
https://github.com/robertdavidgraham/masscan
Oh yes! It's not a surprise, just adding a bit to the anecdata. Internet SSH brute force attacks do not seem to have stopped here. :)
I’ve moved to tailscale ssh for all personal servers. I sleep better knowing there aren’t any port 22s facing the internet.
If you only allow key auth they mostly go away.
shifted port, not seeing it. Not to deny the other side benefits of speeding up the algorithm selection, and disabling simple password access.
[Edit]: I forgot to add that if testing the ssh hardening from that site, be sure to do this locally where one has console access and the same version of sshd or if on remote nodes make sure that remote consoles are working first to avoid being locked out of the remote nodes. Modifying the client ciphers first and testing remote connections prior to modifying the servers is the lowest risk.
[1] - https://www.ssh-audit.com/