I posted a question on the wordpress.org site about using systemd with wp-fail2ban. Would it be better to post it here?
Yes, please do!
With the release of v4.3 later this month I’ll be moving all support here - there’ll be a blog post about that soon.
OK, here it is:
Can you help with configuring fail2ban for systemd logging? I can see that wp-fail2ban is adding lines that are visible with journalctl, but fail2ban isn’t picking them up.
My config in jail.local:
[wordpress-hard] enabled = true filter = wordpress-hard maxretry = 1 port = http,https backend = systemd
The wp-fail2ban journal lines look like this:
Mar 05 21:55:26 SERVER.FQDN wordpress(VIRTUAL.FQDN): Authentication attempt for unknown user USERNAME from IP_ADDRESS
fail2ban-regex does match that line against the wordpress-hard filter.
The weird thing is that the phpMyAdmin configuration is basically identical but fail2ban picks up the phpMyAdmin lines from the journal.
Have you run
fail2ban-regex against the
systemd journal directly? Something like:
fail2ban-regex systemd-journal filter.d/wordpress-hard.conf
If that works I’d suggest increasing the verbosity of
fail2ban and see if that gives any clues.
systemd as far as possible for exactly these kinds of reasons; replacing
init was one thing, but there’s nothing about
syslog that needs “fixing”…
I figured out the problem. Fail2ban is picking up lines for system users but not for unprivileged users.
I verified with both WordPress and PHPMyAdmin. In both cases, fail2ban caught invalid logins at the root level but not at the VirtualHost level. (VirtualHosts are running php-fpm as unprivileged users.)
Example: WordPress installed at root, php-fpm running as http. Fail2ban catches wp-fail2ban log lines:
journalctl _PID= 441047 -o verbose -n1 -e
MESSAGE=Authentication attempt for unknown user USER from HOST PRIORITY=5 SYSLOG_FACILITY=4 SYSLOG_IDENTIFIER=wordpress(FQDN) SYSLOG_PID=441047 SYSLOG_TIMESTAMP=Mar 7 20:04:05 _BOOT_ID=75655e1741af4c57846925fda379f1a4 _CAP_EFFECTIVE=0 _CMDLINE=php-fpm: pool POOL_NAME _COMM=php-fpm _EXE=/usr/bin/php-fpm _GID=33 _HOSTNAME=FQDN _MACHINE_ID=fcc8857ae69b4766be3bcc41ae0a7ce0 _PID=441047 _SOURCE_REALTIME_TIMESTAMP=1583629445488039 _SYSTEMD_CGROUP=/system.slice/php-fpm.service _SYSTEMD_INVOCATION_ID=036cd7e82f614da6878b90cb475996ed _SYSTEMD_SLICE=system.slice _SYSTEMD_UNIT=php-fpm.service _TRANSPORT=syslog _UID=33
WordPress installed in VirtualHost, php-fpm running as user id 1001. Fail2ban ignores wp-fail2ban log lines:
journalctl _PID=441785 -o verbose -n1 -e
MESSAGE=Authentication attempt for unknown user USER from HOST PRIORITY=5 SYSLOG_FACILITY=4 SYSLOG_IDENTIFIER=wordpress(FQDN) SYSLOG_PID=441785 SYSLOG_TIMESTAMP=Mar 7 20:06:36 _BOOT_ID=75655e1741af4c57846925fda379f1a4 _CAP_EFFECTIVE=0 _CMDLINE=php-fpm: pool POOL_NAME _COMM=php-fpm _EXE=/usr/bin/php-fpm _GID=1001 _HOSTNAME=FQDN _MACHINE_ID=fcc8857ae69b4766be3bcc41ae0a7ce0 _PID=441785 _SOURCE_REALTIME_TIMESTAMP=1583629596550194 _SYSTEMD_CGROUP=/system.slice/php-fpm.service _SYSTEMD_INVOCATION_ID=036cd7e82f614da6878b90cb475996ed _SYSTEMD_SLICE=system.slice _SYSTEMD_UNIT=php-fpm.service _TRANSPORT=syslog _UID=1001
The only important difference that I can see is one UID/GID is 33, the other 1001.
I guess this is an issue I should ask about at the failban github?
Did you try increasing the
fail2ban verbosity? The
fail2ban log usually gives a good idea of why entries are being ignored, or if they’re being seen at all.
Yes. The virtualhost errors are not hitting the fail2ban log, even with loglevel set to DEBUG.
OK, then I think this is firmly in the “weird stuff
systemd does” category and beyond what I can sensibly help with.
I’d be very interested in what the
fail2ban folks come up with though - I may disagree with
systemd but it can’t hurt to know more about how it “works”.
I agree, it’s a fail2ban/systemd issue, not a wp-fail2ban issue.
There is one thing that you might consider doing that would make using systemd easier for a setup like mine with different sites using wp-fail2ban. When you use systemd, fail2ban wants you to set a journalmatch so that it doesn’t have to check all journal lines. The way the wp-fail2ban lines read now, you can’t use the obvious match (SYSLOG_IDENTIFIER=wordpress) because the identifier varies by site (i.e., wordpress(example.com)), and SYSLOG_IDENTIFIER can’t match patterns.
Would you consider moving the domain name to the MESSAGE part of the log line and making the SYSLOG_IDENTIFIER just be wordpress?
(This is assuming that I can get the virtualhost issue resolved.)
systemd is wrong: the identifier is the bit before the brackets - the brackets are an “extra”. However, I’d strongly recommend you don’t waste your time trying to get them to fix it unless you really have nothing better to do with your life and all paint within sight is already dry.
As for adding another workaround to support
systemd, in principle, yes, but only if you’re willing to be a guinea-pig during development. This is very much an edge case - at this point it’s not worth serious development time as 99.999% of users just install
One idea I just thought of while writing this: what if you set a custom facility? Surely
systemd can filter by e.g.
I googled, but I couldn’t find anything useful about custom facilities with systemd. Do you know anything about doing that?
The journalmatch I’m currently using is _COMM=php-fpm. That limits it fairly well, so it’s not a big deal. It’s just not as elegant as using “wordpress” would be. As I understand it, I’m limited to the identifiers listed above in the journalctl verbose output.
I am willing to be a guinea-pig if this is something you would like to pursue. I, too, have always used syslog-ng, but I am setting up a new server and decided to try to use systemd for as much as possible this time.
I did test by hacking your code and removing the domain name from the log output, and then I could find the wp-fail2ban lines using journalctl -t wordpress, which means that journalmatch = SYSLOG_IDENTIFIER=wordpress would work.
My assumption is that if you were to say:
that number would change, and in theory you could filter that way.
That does work, but again only for the WordPress install at the root level, not for the unprivileged user’s WordPress. I will submit this to fail2ban and let you know what happens.
It comes down to the user that is set in the php-fpm pool configuration. Fail2ban catches failed logins in this case:
user = http group = http
but not in this case:
user = rhodie group = rhodie
(Again, not caused by wp-fail2ban, just updating the thread with more info.)
(I used phpMyAdmin because it has the same issue and comes with fail2ban.)
Got a response from the fail2ban developer. The issue is that by default fail2ban only reads the main systemd journal, but the unprivileged user has a different journal.
The fix is either to configure the relevant fail2ban jails to look in all journals with
backend = systemd[journalflags=1] or to configure php-fpm to log to the main journal with
in its journald.conf.
Thanks - that’s good to know.
I imagine your best bet is to use the main journal with something like
LOG_LOCAL0 for all the WPf2b events - that should save having to deal with
systemd's idea of what an identifier is.