invis.net

Using systemd with wp-fail2ban

I posted a question on the wordpress.org site about using systemd with wp-fail2ban. Would it be better to post it here?

Yes, please do!

With the release of v4.3 later this month I’ll be moving all support here - there’ll be a blog post about that soon.

OK, here it is:

Can you help with configuring fail2ban for systemd logging? I can see that wp-fail2ban is adding lines that are visible with journalctl, but fail2ban isn’t picking them up.

My config in jail.local:

[wordpress-hard]
enabled = true
filter = wordpress-hard
maxretry = 1
port = http,https
backend = systemd

The wp-fail2ban journal lines look like this:
Mar 05 21:55:26 SERVER.FQDN wordpress(VIRTUAL.FQDN)[246821]: Authentication attempt for unknown user USERNAME from IP_ADDRESS

fail2ban-regex does match that line against the wordpress-hard filter.

The weird thing is that the phpMyAdmin configuration is basically identical but fail2ban picks up the phpMyAdmin lines from the journal.

Have you run fail2ban-regex against the systemd journal directly? Something like:

fail2ban-regex systemd-journal filter.d/wordpress-hard.conf

If that works I’d suggest increasing the verbosity of fail2ban and see if that gives any clues.

I avoid systemd as far as possible for exactly these kinds of reasons; replacing init was one thing, but there’s nothing about syslog that needs “fixing”…

I figured out the problem. Fail2ban is picking up lines for system users but not for unprivileged users.

I verified with both WordPress and PHPMyAdmin. In both cases, fail2ban caught invalid logins at the root level but not at the VirtualHost level. (VirtualHosts are running php-fpm as unprivileged users.)

Example: WordPress installed at root, php-fpm running as http. Fail2ban catches wp-fail2ban log lines:

journalctl _PID= 441047 -o verbose -n1 -e

     MESSAGE=Authentication attempt for unknown user USER from HOST
     PRIORITY=5
     SYSLOG_FACILITY=4
     SYSLOG_IDENTIFIER=wordpress(FQDN)                                                                                             
     SYSLOG_PID=441047                                                                                                             
     SYSLOG_TIMESTAMP=Mar  7 20:04:05                                                                                              
     _BOOT_ID=75655e1741af4c57846925fda379f1a4
     _CAP_EFFECTIVE=0
     _CMDLINE=php-fpm: pool POOL_NAME
     _COMM=php-fpm
     _EXE=/usr/bin/php-fpm
     _GID=33                                                                                                                       
     _HOSTNAME=FQDN
     _MACHINE_ID=fcc8857ae69b4766be3bcc41ae0a7ce0
     _PID=441047                                                                                                                   
     _SOURCE_REALTIME_TIMESTAMP=1583629445488039                                                                                   
     _SYSTEMD_CGROUP=/system.slice/php-fpm.service
     _SYSTEMD_INVOCATION_ID=036cd7e82f614da6878b90cb475996ed
     _SYSTEMD_SLICE=system.slice
     _SYSTEMD_UNIT=php-fpm.service
     _TRANSPORT=syslog
     _UID=33                                                                                                                       

WordPress installed in VirtualHost, php-fpm running as user id 1001. Fail2ban ignores wp-fail2ban log lines:

journalctl _PID=441785 -o verbose -n1 -e

      MESSAGE=Authentication attempt for unknown user USER from HOST
      PRIORITY=5
      SYSLOG_FACILITY=4
      SYSLOG_IDENTIFIER=wordpress(FQDN)                                                                                             
      SYSLOG_PID=441785                                                                                                             
      SYSLOG_TIMESTAMP=Mar  7 20:06:36                                                                                              
      _BOOT_ID=75655e1741af4c57846925fda379f1a4
      _CAP_EFFECTIVE=0
      _CMDLINE=php-fpm: pool POOL_NAME
      _COMM=php-fpm
      _EXE=/usr/bin/php-fpm
      _GID=1001                                                                                                                     
      _HOSTNAME=FQDN
      _MACHINE_ID=fcc8857ae69b4766be3bcc41ae0a7ce0
      _PID=441785                                                                                                                   
      _SOURCE_REALTIME_TIMESTAMP=1583629596550194                                                                                   
      _SYSTEMD_CGROUP=/system.slice/php-fpm.service
      _SYSTEMD_INVOCATION_ID=036cd7e82f614da6878b90cb475996ed
      _SYSTEMD_SLICE=system.slice
      _SYSTEMD_UNIT=php-fpm.service
      _TRANSPORT=syslog
      _UID=1001                                                                                                                     

The only important difference that I can see is one UID/GID is 33, the other 1001.

I guess this is an issue I should ask about at the failban github?

Did you try increasing the fail2ban verbosity? The fail2ban log usually gives a good idea of why entries are being ignored, or if they’re being seen at all.

Yes. The virtualhost errors are not hitting the fail2ban log, even with loglevel set to DEBUG.

OK, then I think this is firmly in the “weird stuff systemd does” category and beyond what I can sensibly help with.

I’d be very interested in what the fail2ban folks come up with though - I may disagree with systemd but it can’t hurt to know more about how it “works”.

I agree, it’s a fail2ban/systemd issue, not a wp-fail2ban issue.

There is one thing that you might consider doing that would make using systemd easier for a setup like mine with different sites using wp-fail2ban. When you use systemd, fail2ban wants you to set a journalmatch so that it doesn’t have to check all journal lines. The way the wp-fail2ban lines read now, you can’t use the obvious match (SYSLOG_IDENTIFIER=wordpress) because the identifier varies by site (i.e., wordpress(example.com)), and SYSLOG_IDENTIFIER can’t match patterns.

Would you consider moving the domain name to the MESSAGE part of the log line and making the SYSLOG_IDENTIFIER just be wordpress?

(This is assuming that I can get the virtualhost issue resolved.)

Firstly, systemd is wrong: the identifier is the bit before the brackets - the brackets are an “extra”. However, I’d strongly recommend you don’t waste your time trying to get them to fix it unless you really have nothing better to do with your life and all paint within sight is already dry.

As for adding another workaround to support systemd, in principle, yes, but only if you’re willing to be a guinea-pig during development. This is very much an edge case - at this point it’s not worth serious development time as 99.999% of users just install syslog-ng.

One idea I just thought of while writing this: what if you set a custom facility? Surely systemd can filter by e.g. LOG_LOCAL5?

I googled, but I couldn’t find anything useful about custom facilities with systemd. Do you know anything about doing that?

The journalmatch I’m currently using is _COMM=php-fpm. That limits it fairly well, so it’s not a big deal. It’s just not as elegant as using “wordpress” would be. As I understand it, I’m limited to the identifiers listed above in the journalctl verbose output.

I am willing to be a guinea-pig if this is something you would like to pursue. I, too, have always used syslog-ng, but I am setting up a new server and decided to try to use systemd for as much as possible this time.

I did test by hacking your code and removing the domain name from the log output, and then I could find the wp-fail2ban lines using journalctl -t wordpress, which means that journalmatch = SYSLOG_IDENTIFIER=wordpress would work.

My assumption is that if you were to say:

define('WP_FAIL2BAN_AUTH_LOG', LOG_LOCAL0);

that number would change, and in theory you could filter that way.

That does work, but again only for the WordPress install at the root level, not for the unprivileged user’s WordPress. I will submit this to fail2ban and let you know what happens.

It comes down to the user that is set in the php-fpm pool configuration. Fail2ban catches failed logins in this case:

user = http
group = http

but not in this case:

user = rhodie
group = rhodie

(Again, not caused by wp-fail2ban, just updating the thread with more info.)

Fail2ban bug report

(I used phpMyAdmin because it has the same issue and comes with fail2ban.)

Got a response from the fail2ban developer. The issue is that by default fail2ban only reads the main systemd journal, but the unprivileged user has a different journal.

The fix is either to configure the relevant fail2ban jails to look in all journals with backend = systemd[journalflags=1] or to configure php-fpm to log to the main journal with

[Journal]
SplitMode=none

in its journald.conf.

Thanks - that’s good to know.

I imagine your best bet is to use the main journal with something like LOG_LOCAL0 for all the WPf2b events - that should save having to deal with systemd's idea of what an identifier is.