Fail2ban – Remote Code Execution(research.securitum.com)
research.securitum.com
Fail2ban – Remote Code Execution
https://research.securitum.com/fail2ban-remote-code-execution/
64 comments
Damn, this is amazing. Even as bare access goes. Find is indeed a severity red, unsure who is gonna patch up mailutils
I'm going to guess "noone". This is not the first security hole like this caused by piping to mail. See CVE-2000-0703, a trivial local root via suidperl. Unfortunately backwards compatability often wins over prevention of future security holes.
This is the kind of thing that is never going to get patched on a lot of systems due to the obscurity of the tool in question. Really really bad.
At least higher profile stuff usually gets patched quickly.
At least higher profile stuff usually gets patched quickly.
fail2ban is surely not obscure, see for instance
https://qa.debian.org/popcon.php?package=fail2ban
It's not like this is a top100 package, but I reckon it's fairly popular on servers. I'm sure Debian will have a fix for this quickly.
https://qa.debian.org/popcon.php?package=fail2ban
It's not like this is a top100 package, but I reckon it's fairly popular on servers. I'm sure Debian will have a fix for this quickly.
It comes pre-installed in one of DigitalOcean's main images; I think the LAMP one.
also used by yunohost by default.
Is fail2ban obscure? I was under the impression it was a must have for any server exposed to the internet.
My thinking is that good security configuration makes fail2ban redundant. With password authentication disabled and strong keys, it’s not clear to me what the threat is that fail2ban offers to protect against.
I use it mostly because it simply declutters the logfiles, and it's super easy to set up and has practically no maintenance, so why not?
EDIT: as others have noted here, fail2ban can do much more than just ssh. I also use it for Exim to block all these open-relay-scanners which are polluting the logs.
EDIT: as others have noted here, fail2ban can do much more than just ssh. I also use it for Exim to block all these open-relay-scanners which are polluting the logs.
No maintenance except the security patches I guess :)
fail2ban is not only for SSH, but also for HTTP. You have a php website with apache and logs enabled? Fail2ban can ban people if they try to brute force your login page. It's actually quite powerful, but I see it in use less and less.
This. I host a very simple website at home on my Raspberry Pi; maybe 20 regular users. But, I receive a ton of traffic trying to login to PHP admin, nginx scripts, ssh brute force attempts, and on and on... I use fail2ban to ban individual IPs and if I see a lot coming from a particular CIDR range, I'll just block that whole range.
There are a ton of example jails out on GitHub and elsewhere that are easily dropped into your configs.
There are a ton of example jails out on GitHub and elsewhere that are easily dropped into your configs.
I mostly see it on smaller systems configured by people or teams who doesn't really manage servers professionally.
Mostly we just don't allow SSH via the internet, you have to be on the office network or on a VPN and AWS instances can be accessed using Amazon Systems Manager. For those few systems that absolutely most be accessible via SSH on the internet: SSH keys and/or multi-factor authentication is required.
Mostly we just don't allow SSH via the internet, you have to be on the office network or on a VPN and AWS instances can be accessed using Amazon Systems Manager. For those few systems that absolutely most be accessible via SSH on the internet: SSH keys and/or multi-factor authentication is required.
Why is it a must have? Just disable password logins on Internet-facing systems.
I should have been more explicit, you're right it's not necessary for SSH with keys.
I was thinking more about preventing spam on web servers.
I was thinking more about preventing spam on web servers.
This is why we enable port knocking (SPA) on all of our Internet-facing systems: it protects against zero-days and keeps the logs pristine. As a matter fact, any failed login of any kind creates a security alert.
I like to use it just to make logs more readable.
[deleted]
There's less ducktape solutions such as sshguard.
How popular is this feature to send emails from fail2ban? I haven't heard about it until now, and I think I'd have a very noisy inbox if my fail2ban setup sent me mails about every blocked IP.
Feeding the fail2ban log to Papertrail or another logging service should give enough visibility.
I've seen fail2ban block the IPv4 of a company's headquarters because a new member of staff setup their SSH config wrong and made too many attempts to connect to a production system. In this sort of situation, having federated logs on external infrastructure is a real saviour.
I've seen fail2ban block the IPv4 of a company's headquarters because a new member of staff setup their SSH config wrong and made too many attempts to connect to a production system. In this sort of situation, having federated logs on external infrastructure is a real saviour.
I do it on my personal servers for ssh.
I move SSH to a non-standard port, which cuts down a ton. I typically only get a couple emails a year.
I move SSH to a non-standard port, which cuts down a ton. I typically only get a couple emails a year.
Serious question: What do you hope to achieve with all that?
Fail2Ban has caused more problems for me than it has ever solved, and non-standard ports is confusing if you're a large team. Personally I feel like you're better of just requiring ssh keys, don't install fail2ban and stay on port 22.
Fail2Ban has caused more problems for me than it has ever solved, and non-standard ports is confusing if you're a large team. Personally I feel like you're better of just requiring ssh keys, don't install fail2ban and stay on port 22.
> non-standard ports is confusing if you're a large team
I think this is the key part. For me, this is for very small teams or for my personal server, which I need to be able to access with a password from possibly anywhere.
For a larger team, I think I agree - emails would be too low of a signal-to-noise ratio. Although I would still use something like fail2ban, but use it just to log to somewhere more appropriate just in case I need it.
I think this is the key part. For me, this is for very small teams or for my personal server, which I need to be able to access with a password from possibly anywhere.
For a larger team, I think I agree - emails would be too low of a signal-to-noise ratio. Although I would still use something like fail2ban, but use it just to log to somewhere more appropriate just in case I need it.
Gentle reminder that you do not need fail2ban to block repeat connection attempts. Two IPTables lines will block any IPs connecting over a given rate in a given time. A third line logs it.
For pure ssh, this is true, but note that fail2ban can also automatically unblock after a certain amount of time, which is nice when valid users shut themselves out by mistake over the weekend... For other services, depending on their type, the false-positive rate will often be way too high.
Pam_shield works well for this, and doesn't have to troll logs to keep counts.
FWIW this can also be done with ipsets. [1] ipset hash tables are quite useful if you are blocking a large number of hosts. The CPU load is much lower than having individual rules for each host blocked. You can set any time limit on a ipset or have no time limit. You can also dynamically insert ip's into ipsets using iptables rules.
[1] - https://superuser.com/questions/1412286/time-limited-whiteli...
[1] - https://superuser.com/questions/1412286/time-limited-whiteli...
Fail2ban can protect any kind of application, not just SSH.
All they need to do is emit a structured log line format after a failed login attempt, or any other kind of undesirable behaviour.
All they need to do is emit a structured log line format after a failed login attempt, or any other kind of undesirable behaviour.
Here's an example: https://www.rackaid.com/blog/how-to-block-ssh-brute-force-at...
You need fail2ban to block repeat failed login attempts. If you just use iptables, you can't distinguish between a failed SSH login and a short-lived successful one, and there's a lot of legitimate reasons you might have several short-lived successful ones in a row.
This is why we enable port knocking (SPA) on all of our Internet-facing systems: it protects against zero-days, keeps the logs pristine, and stops repeat connection attempts. As a matter fact, any failed login of any kind creates a security alert.
Care to provide those 3 lines?
Start with the basics:
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i lo -m comment --comment "Allow loopback connections" -j ACCEPT
iptables -A INPUT -p icmp -m comment --comment "Allow ICMP packets" -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --syn --dport 22 -m connlimit --connlimit-above 5 --connlimit-mask 32 -j REJECT --reject-with tcp-reset
iptables -t filter -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 100 -j DROP
iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set
iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 5 --name SSH --rsource -j DROP
iptables -A INPUT -i eth0 -p tcp --dport 443 -m state --state RELATED,ESTABLISHED -m limit --limit 150/second --limit-burst 300 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -m hashlimit --hashlimit-name HTTPS --hashlimit-mode srcip --hashlimit 1/s --hashlimit-burst 300 -j ACCEPT
iptables -A INPUT -i eth0 -j DROPNice, thanks!
Since 0.8.1 Fail2ban uses Sendmail MTA for mailing so this is worrying only if you purposefully reverted to mail or upgraded from an older version. Recent deployments SEEM like they'd be fine out of the box.
The patches just update the action files to add escaping if you happen to be using mail.
The patches just update the action files to add escaping if you happen to be using mail.
This exploit is for a combination of fail2ban and `mail`. Reading how it works, it seems much more of a vulnerability for anything using the `mail` command than in fail2ban per se.
That ~! escape is really dangerous. What percentage of sysadmins are even aware of its existence? I can see how it can be useful, but there is a lot of potential for exploit if you aren't extremely careful.
The `mail` command shouldn't so easily accommodate executing arbitrary commands from input. The ~! escape should probably be either removed from `mail` entirely, or enabled only if you pass it a flag. It seems like a vestige from an earlier, more innocent time.
This isn't to absolve sysadmins who fail to sanitize their inputs, but let's not make their job so difficult.
That ~! escape is really dangerous. What percentage of sysadmins are even aware of its existence? I can see how it can be useful, but there is a lot of potential for exploit if you aren't extremely careful.
The `mail` command shouldn't so easily accommodate executing arbitrary commands from input. The ~! escape should probably be either removed from `mail` entirely, or enabled only if you pass it a flag. It seems like a vestige from an earlier, more innocent time.
This isn't to absolve sysadmins who fail to sanitize their inputs, but let's not make their job so difficult.
This is a typical old world vs new world vulnerability. Old tools have lots of hacker features that are often not a good idea anymore in this (century's) world. See also cryptographic agility and such.
The Unix philosophy of combining small, specialised tools into a larger whole to accomplish a task is an important concept that really shouldn't be forgotten, but at the same time one should remember not all tools are meant to withstand malicious input from a hostile internet, in particular those that predate the explosion of the internet.
The Unix philosophy of combining small, specialised tools into a larger whole to accomplish a task is an important concept that really shouldn't be forgotten, but at the same time one should remember not all tools are meant to withstand malicious input from a hostile internet, in particular those that predate the explosion of the internet.
The mail command has been around for a while, and was intended mainly for human use, not scripts.
For example, just run mail alone with no arguments. It'll display the messages in your local mail box, and prompt you which ones to read, keep, delete, reply to, etc.
The mail command is an MUA, just like mutt or Thunderbird, but much older.
For example, just run mail alone with no arguments. It'll display the messages in your local mail box, and prompt you which ones to read, keep, delete, reply to, etc.
The mail command is an MUA, just like mutt or Thunderbird, but much older.
But it's been used for sending emails via scripts since forever, too.
Use sendmail for that.
For extra fun, there are (or at least were) multiple implementations of the mail command. The arguments were similar enough, but an old (and replaced) system at a previous employer required Heirloom mail/snail, not BSD because it actually intentionally used escape sequences like this.
In particular, it used one to add attachments (by giving the path).
We replaced it with Perl, getting rid of the shell script entirely (the whole stack was Perl).
Shell scripts really ought to use the sendmail command to send mail, but then you have to remember those obscure options to pass and generate the mail headers yourself, so it's understandable why no one does. (And probably handle dot-doubling).
In particular, it used one to add attachments (by giving the path).
We replaced it with Perl, getting rid of the shell script entirely (the whole stack was Perl).
Shell scripts really ought to use the sendmail command to send mail, but then you have to remember those obscure options to pass and generate the mail headers yourself, so it's understandable why no one does. (And probably handle dot-doubling).
AFAIK the email alert feature is not enabled by default in fail2ban.
Is there any chance a default config of fail2ban in a typical Linux distribution would be vulnerable to this?
Is there any chance a default config of fail2ban in a typical Linux distribution would be vulnerable to this?
it would have to generate an e-mail by default, which means you need mail setup.
on top of this you need to MITM the whois command (which is unencrypted, but still). This is not easily exploitable.
Is there a flag to disable tilde escapes, or a similar command that doesn't have this feature? The mail command is a really convenient way of sending mail in scripts.
This makes me think to put something like
drop table prefixes;
in my ARIN WHOIS for a single /24 and see what happens to people scraping the database for marketing purposes
drop table prefixes;
in my ARIN WHOIS for a single /24 and see what happens to people scraping the database for marketing purposes
Does this remind anyone else of the exploit in The Cuckoo's Egg[0] whereby hackers escalate privilege by using GNU Emacs' mailutils?
[0] https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)
[0] https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)
Looking at the patches, it seems running 'grep -rnw "mail -s" <fail2ban installation folder>, and replacing all found with "mail -E 'set escape' -s" is all that is needed.
Well the "-s" is denoting the subject but yes "-E 'set escape'" is the mitigation fail2ban are employing though it should be noted that this is specific to Mailutils. Many (most?) other mail(1)'s use the -E option to mean don't send messages with an empty body which left me very confused when I first read your comment so had to do some digging to understand it.
I have never understood the point of Fail2ban other than feeling kinda smug.
If your SSH server is more at risk because an attacker simply has more attempts, surely your SSH server is not secure?
It is just adding another attack surface.
If your SSH server is more at risk because an attacker simply has more attempts, surely your SSH server is not secure?
It is just adding another attack surface.
> If your SSH server is more at risk
You can have it block all ports for the attacking host, so it might help other protocols. Aside from just reducing load on the host. By dropping the TCP SYN packet, you also introduce delays for the attacker.
You can have it block all ports for the attacking host, so it might help other protocols. Aside from just reducing load on the host. By dropping the TCP SYN packet, you also introduce delays for the attacker.
The main benefit for me is protecting password services like HTTP basic auth, ftp, mailserver, etc. It’s pretty pointless for SSH when keys are involved
I have seen it used for legacy SFTP installations where password auth is still a thing (customers uploading CSV files and such).
It's one of the many things designed to make lousy passwords become actually ok-ish.
Specifically for ssh, I've also had my vps eating a surprising amount of cpu just to verify and reject bad logins when some scanner or other spent a couple days on it before moving on.
Specifically for ssh, I've also had my vps eating a surprising amount of cpu just to verify and reject bad logins when some scanner or other spent a couple days on it before moving on.
You can also just use a firewall rule to rate limit syn packets from a source. It won't affect legit traffic, and doesn't require running a userland process to watch the logs.
If nothing else, fail2ban reduces the staggering amount of logs that result in bots hitting services with useless password attempts.
Defense in depth.
We can't ever prove that something is "secure", we can only take precautions. Is your SSH server secure? You can't tell, by definition, in a world where 0 day exploits exist.
If you know where your requests are coming from, you can restrict to those IP addresses. It doesn't mean you think your server is insecure - but it's making the lives of people trying to get to it that much harder. If you don't know, then maybe fail2ban is a worthwhile alternative. It doesn't even necessarily mean you are trying to prevent brute forcing (password logins should always be disabled anyway) - but some exploits require multiple attempts.
In the past I was against measures like changing the default port (although it should still be <1024). But these measures are helpful the moment some new exploit is in the wild that could twart naive scanning scripts. Same thing goes for hiding the banner.
The best thing is still to automate things so that SSH access becomes a 'break the glass' moment. If you are always logging in via SSH(and it's a server, not a workstation that you work on), it means that there's something missing in the automation (or observability) that needs to be fixed.
We can't ever prove that something is "secure", we can only take precautions. Is your SSH server secure? You can't tell, by definition, in a world where 0 day exploits exist.
If you know where your requests are coming from, you can restrict to those IP addresses. It doesn't mean you think your server is insecure - but it's making the lives of people trying to get to it that much harder. If you don't know, then maybe fail2ban is a worthwhile alternative. It doesn't even necessarily mean you are trying to prevent brute forcing (password logins should always be disabled anyway) - but some exploits require multiple attempts.
In the past I was against measures like changing the default port (although it should still be <1024). But these measures are helpful the moment some new exploit is in the wild that could twart naive scanning scripts. Same thing goes for hiding the banner.
The best thing is still to automate things so that SSH access becomes a 'break the glass' moment. If you are always logging in via SSH(and it's a server, not a workstation that you work on), it means that there's something missing in the automation (or observability) that needs to be fixed.
Adding more stuff isn't necessarily defense in depth.
It's also an opportunity to make mistakes, which in worst case could provide free root shells to people with funny host names.
It's also an opportunity to make mistakes, which in worst case could provide free root shells to people with funny host names.
Upstream has changed the behavior as a result.
https://savannah.gnu.org/bugs/?60937
Kind of surprising as many *nix utilities that can shell out will offer "secure" modes to disable such an ability.
https://savannah.gnu.org/bugs/?60937
Kind of surprising as many *nix utilities that can shell out will offer "secure" modes to disable such an ability.
Interesting. This doesn't sound like that much of a vulnerability itself, but it does help clarify my thinking about SSH security.
Basically, SSH with proper configuration banning password auth is just fine and okay to be exposed to the internet. Extra logs from some failed attempts aren't really a big deal. If you want to make access more secure for it, that's okay, but I'd resist using complex on-server software for that which is likely to be less battle-tested and expose more attack surface. If you must do so, do things that are simple and/or off-server, like run on an alternate port or block network access for control ports at the firewall or security group level from any IP range but the ones you expect to be connecting from.
Basically, SSH with proper configuration banning password auth is just fine and okay to be exposed to the internet. Extra logs from some failed attempts aren't really a big deal. If you want to make access more secure for it, that's okay, but I'd resist using complex on-server software for that which is likely to be less battle-tested and expose more attack surface. If you must do so, do things that are simple and/or off-server, like run on an alternate port or block network access for control ports at the firewall or security group level from any IP range but the ones you expect to be connecting from.