tickets #152707
open
Improve Pagure public SSH security
Added by crameleon 7 months ago.
Updated about 1 month ago.
Description
As the health checks from the proxy servers (which do not authenticate and merely check if the SSH server responds) caused the proxy IP addresses to be banned repeatedly, these addresses were whitelisted.
Since the machine sees all SSH connections from the internet to originate from the atlas1/atlas2 proxy IP addresses, this now means that every connection from the internet is whitelisted, which practically causes fail2ban to be not banning any failed connection attempts.
This should be remediated.
Investigation shows that since there is no support for the PROXY protocol in SSH, something like TPROXY will need to be implemented on the HAProxy level to pass the real source address for the TCP proxy used for SSH.
- Private changed from Yes to No
- Category changed from Git(lab|hub) to Pagure
Alternative, and seemingly easier, solution is to use MMProxy, or the more modern go-mmproxy. Instead of special firewall rules it only requires a loopback route.
I made a package for testing: https://build.opensuse.org/package/show/home:crameleon:misc/go-mmproxy. I built the master branch as it contains a patch removing all external dependencies, which is a security bonus.
Along with this we should also split sshd into two isolated instances - one for our regular internal management access, and one accepting the public HAProxy -> MMProxy traffic confined to only be able to execute the Pagure related commands.
- % Done changed from 0 to 20
- Status changed from New to In Progress
- Assignee set to crameleon
- % Done changed from 20 to 30
- Subject changed from Facilitate proxy in pagure01 fail2ban setup to Improve Pagure public SSH security
- % Done changed from 30 to 50
Deployed, but a bit stuck with fail2ban. The source address through transparent proxying is populated on the loopback interface, meaning banning needs to happen there. However creating a firewalld zone containing the loopback interface seems to have no effect, since firewalld injects an early accept rule for lo into the underlying nftables:
chain filter_INPUT {
type filter hook input priority filter + 10; policy accept;
ct state { established, related } accept
ct status dnat accept
iifname "lo" accept
jump filter_INPUT_POLICIES_pre
jump filter_INPUT_ZONES
jump filter_INPUT_POLICIES_post
ct state { invalid } drop
reject with icmpx type admin-prohibited
}
"filter_INPUT_ZONES" is the chain containing the custom zone. The iifname "lo" accept seems to always be placed beforehand, without a setting to override it. :-(
We might need to migrate this machine to a custom nftables setup. Or write a fail2ban action which injects the bans in our perimeter firewall (which would technically be better than banning on a host firewall anyways, but is more involved to implement ...).
I started developing a API to ban IP addresses on the perimeter firewall. The only existing software I could find was for iptables, I forked it and have initial functionality adapted to nftables - it still needs some work but is probably the more sustainable approach than working around the firewalld issue and it will help us with implementing fail2ban/sshguard for other services in a central fashion.
Also available in: Atom
PDF