It’s every sysadmin’s nightmare: you’ve got several concurrent connections going, making configuration changes and troubleshooting issues – and then it happens: everything stops responding.
How do you troubleshoot the issue, not only to fix it but also help prevent it in the future?
I was using a DigitalOcean one-click WordPress installation to see what kind of optimizations they do out of the box. It’s easy to see “one-click WordPress” and just move on, but knowing what their automation accomplishes is important. After all, you’ll be the one troubleshooting it later! DO’s one-click WordPress does a few things:
- Spools up a new Ubuntu server
- Installs and configures Apache, MySQL, PHP and WordPress
- Configures iptables (standard linux firewall)
- Installs UFW (front-end to iptables for easier management),
- Installs fail2ban (automatically updates the firewall to block active attackers)
- Installs Let’s Encrypt
The problem I ran into had to do with #5.
Fail2ban does a great job of actively banning IP addresses of hosts who are flooding traffic. The WordPress plugin that DigitalOcean includes in this one-click installation also includes a hard IP ban for failed WordPress login attempts.
After getting my password wrong 3 times, I noticed the entire website was no longer accessible from my PC. It also affected browsing on my phone which was running on WiFi, but not using mobile data.
That was the first hint that it was IP-specific: my entire office network connection was being blocked.
The second hint was that I was also no longer able to SSH into the server. This is significantly more serious, because in order to unblock myself I need access to the server.
Thankfully, DigitalOcean provides console access through their dashboard, and I was able to unban myself through there:
~# fail2ban-client set sshd unbanip ww.xx.yy.zz
With SSH access restored, running a tail on /var/log/fail2ban.log shows that indeed my IP was banned after 3 WordPress login attempts, thanks to their aptly named wordpress-hard filter:
fail2ban.filter [1683]: INFO [wordpress-hard] Found ww.xx.yy.zz fail2ban.filter [1683]: INFO [wordpress-hard] Found ww.xx.yy.zz fail2ban.filter [1683]: INFO [wordpress-hard] Found ww.xx.yy.zz fail2ban.actions [1683]: NOTICE [wordpress-hard] Ban ww.xx.yy.zz
If you want to see what other “Jails” (filters) are in configured in your fail2ban, use the command:
~# fail2ban-client status
To see the IPs banned by a specific Jail:
~# fail2ban-client status wordpress-hard
In order to unban a specific IP:
~# fail2ban-client set wordpress-hard unbanip ww.xx.yy.zz
Once my access to the WordPress site was restored, I wanted to make sure I wouldn’t get stuck in a similar way in the future.
Since I don’t have the time tonight to thoroughly redo the firewall policies, I set out to bring the max failed attempts up. To see what the fail count is currently set to for this Jail:
~# fail2ban-client get wordpress-hard maxretry 3
And to change it to something a little higher:
~# fail2ban-client set wordpress-hard maxretry 5
I hope you don’t find yourself in this kind of predicament, but I’m guessing a lot of users are actively using the DigitalOcean WordPress one-click installer and may eventually fall into this trap.
Image source: pexels.com