I posted a question on the wordpress.org site about using systemd with wp-fail2ban. Would it be better to post it here?
Yes, please do!
With the release of v4.3 later this month I’ll be moving all support here - there’ll be a blog post about that soon.
OK, here it is:
Can you help with configuring fail2ban for systemd logging? I can see that wp-fail2ban is adding lines that are visible with journalctl, but fail2ban isn’t picking them up.
My config in jail.local:
[wordpress-hard]
enabled = true
filter = wordpress-hard
maxretry = 1
port = http,https
backend = systemd
The wp-fail2ban journal lines look like this:
Mar 05 21:55:26 SERVER.FQDN wordpress(VIRTUAL.FQDN)[246821]: Authentication attempt for unknown user USERNAME from IP_ADDRESS
fail2ban-regex does match that line against the wordpress-hard filter.
The weird thing is that the phpMyAdmin configuration is basically identical but fail2ban picks up the phpMyAdmin lines from the journal.
Have you run fail2ban-regex against the systemd journal directly? Something like:
fail2ban-regex systemd-journal filter.d/wordpress-hard.conf
If that works I’d suggest increasing the verbosity of fail2ban and see if that gives any clues.
I avoid systemd as far as possible for exactly these kinds of reasons; replacing init was one thing, but there’s nothing about syslog that needs “fixing”…
I figured out the problem. Fail2ban is picking up lines for system users but not for unprivileged users.
I verified with both WordPress and PHPMyAdmin. In both cases, fail2ban caught invalid logins at the root level but not at the VirtualHost level. (VirtualHosts are running php-fpm as unprivileged users.)
Example: WordPress installed at root, php-fpm running as http. Fail2ban catches wp-fail2ban log lines:
journalctl _PID= 441047 -o verbose -n1 -e
MESSAGE=Authentication attempt for unknown user USER from HOST
PRIORITY=5
SYSLOG_FACILITY=4
SYSLOG_IDENTIFIER=wordpress(FQDN)
SYSLOG_PID=441047
SYSLOG_TIMESTAMP=Mar 7 20:04:05
_BOOT_ID=75655e1741af4c57846925fda379f1a4
_CAP_EFFECTIVE=0
_CMDLINE=php-fpm: pool POOL_NAME
_COMM=php-fpm
_EXE=/usr/bin/php-fpm
_GID=33
_HOSTNAME=FQDN
_MACHINE_ID=fcc8857ae69b4766be3bcc41ae0a7ce0
_PID=441047
_SOURCE_REALTIME_TIMESTAMP=1583629445488039
_SYSTEMD_CGROUP=/system.slice/php-fpm.service
_SYSTEMD_INVOCATION_ID=036cd7e82f614da6878b90cb475996ed
_SYSTEMD_SLICE=system.slice
_SYSTEMD_UNIT=php-fpm.service
_TRANSPORT=syslog
_UID=33
WordPress installed in VirtualHost, php-fpm running as user id 1001. Fail2ban ignores wp-fail2ban log lines:
journalctl _PID=441785 -o verbose -n1 -e
MESSAGE=Authentication attempt for unknown user USER from HOST
PRIORITY=5
SYSLOG_FACILITY=4
SYSLOG_IDENTIFIER=wordpress(FQDN)
SYSLOG_PID=441785
SYSLOG_TIMESTAMP=Mar 7 20:06:36
_BOOT_ID=75655e1741af4c57846925fda379f1a4
_CAP_EFFECTIVE=0
_CMDLINE=php-fpm: pool POOL_NAME
_COMM=php-fpm
_EXE=/usr/bin/php-fpm
_GID=1001
_HOSTNAME=FQDN
_MACHINE_ID=fcc8857ae69b4766be3bcc41ae0a7ce0
_PID=441785
_SOURCE_REALTIME_TIMESTAMP=1583629596550194
_SYSTEMD_CGROUP=/system.slice/php-fpm.service
_SYSTEMD_INVOCATION_ID=036cd7e82f614da6878b90cb475996ed
_SYSTEMD_SLICE=system.slice
_SYSTEMD_UNIT=php-fpm.service
_TRANSPORT=syslog
_UID=1001
The only important difference that I can see is one UID/GID is 33, the other 1001.
I guess this is an issue I should ask about at the failban github?
Did you try increasing the fail2ban verbosity? The fail2ban log usually gives a good idea of why entries are being ignored, or if they’re being seen at all.
Yes. The virtualhost errors are not hitting the fail2ban log, even with loglevel set to DEBUG.
OK, then I think this is firmly in the “weird stuff systemd does” category and beyond what I can sensibly help with.
I’d be very interested in what the fail2ban folks come up with though - I may disagree with systemd but it can’t hurt to know more about how it “works”.
I agree, it’s a fail2ban/systemd issue, not a wp-fail2ban issue.
There is one thing that you might consider doing that would make using systemd easier for a setup like mine with different sites using wp-fail2ban. When you use systemd, fail2ban wants you to set a journalmatch so that it doesn’t have to check all journal lines. The way the wp-fail2ban lines read now, you can’t use the obvious match (SYSLOG_IDENTIFIER=wordpress) because the identifier varies by site (i.e., wordpress(example.com)), and SYSLOG_IDENTIFIER can’t match patterns.
Would you consider moving the domain name to the MESSAGE part of the log line and making the SYSLOG_IDENTIFIER just be wordpress?
(This is assuming that I can get the virtualhost issue resolved.)
Firstly, systemd is wrong: the identifier is the bit before the brackets - the brackets are an “extra”. However, I’d strongly recommend you don’t waste your time trying to get them to fix it unless you really have nothing better to do with your life and all paint within sight is already dry.
As for adding another workaround to support systemd, in principle, yes, but only if you’re willing to be a guinea-pig during development. This is very much an edge case - at this point it’s not worth serious development time as 99.999% of users just install syslog-ng.
One idea I just thought of while writing this: what if you set a custom facility? Surely systemd can filter by e.g. LOG_LOCAL5?
I googled, but I couldn’t find anything useful about custom facilities with systemd. Do you know anything about doing that?
The journalmatch I’m currently using is _COMM=php-fpm. That limits it fairly well, so it’s not a big deal. It’s just not as elegant as using “wordpress” would be. As I understand it, I’m limited to the identifiers listed above in the journalctl verbose output.
I am willing to be a guinea-pig if this is something you would like to pursue. I, too, have always used syslog-ng, but I am setting up a new server and decided to try to use systemd for as much as possible this time.
I did test by hacking your code and removing the domain name from the log output, and then I could find the wp-fail2ban lines using journalctl -t wordpress, which means that journalmatch = SYSLOG_IDENTIFIER=wordpress would work.
My assumption is that if you were to say:
define('WP_FAIL2BAN_AUTH_LOG', LOG_LOCAL0);
that number would change, and in theory you could filter that way.
That does work, but again only for the WordPress install at the root level, not for the unprivileged user’s WordPress. I will submit this to fail2ban and let you know what happens.
It comes down to the user that is set in the php-fpm pool configuration. Fail2ban catches failed logins in this case:
user = http
group = http
but not in this case:
user = rhodie
group = rhodie
(Again, not caused by wp-fail2ban, just updating the thread with more info.)
(I used phpMyAdmin because it has the same issue and comes with fail2ban.)
Got a response from the fail2ban developer. The issue is that by default fail2ban only reads the main systemd journal, but the unprivileged user has a different journal.
The fix is either to configure the relevant fail2ban jails to look in all journals with backend = systemd[journalflags=1] or to configure php-fpm to log to the main journal with
[Journal]
SplitMode=none
in its journald.conf.
Thanks - that’s good to know.
I imagine your best bet is to use the main journal with something like LOG_LOCAL0 for all the WPf2b events - that should save having to deal with systemd's idea of what an identifier is.