tl;dr: Add PATH="${PATH}:/bin:/usr/bin:/sbin:/usr/sbin" to /etc/default/firehol when using backported version 3 of firehol on Ubuntu.

firehol – an easy to use but powerful iptables stateful firewall

man firehol

With this out of the way: When installing firehol on aging Xenial (Ubuntu 16.04) you want the backported packages by Andrey Galkin to get version 3 of firehol over version 2 in universe – especially when working with IPv6: https://launchpad.net/~andvgal

When done setting up your rules you may find out after a reboot that the systemd job involved will claim to have started firehol but eventually discover that your iptables are empty despite systemd claiming otherwhise and having set START_FIREHOL=YES in /etc/default/firehol:

● firehol.service - LSB: firehol firewall configuration
   Loaded: loaded (/etc/init.d/firehol; bad; vendor preset: enabled)
   Active: active (exited) since Fr 2020-11-27 15:43:51 CET; 2h 8min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 31555 ExecStop=/etc/init.d/firehol stop (code=exited, status=0/SUCCESS)
  Process: 31574 ExecStart=/etc/init.d/firehol start (code=exited, status=0/SUCCESS)

This is especially weird when you run the startup /sbin/firehol start command manually and it succeeds just fine.

I had to dig deep to find out where the script is in fact falling flat. This was mostly because of old init script /etc/init.d/firehol redirecting the output of the starting process to /dev/null not showing the errors at all:

do_start () {
        # return
        #  0 000 if firewall has been handled
        #  1 001 if firewall could not be activated
        #  4 100 if FireHOL is disabled via /etc/default/firehol
        [ "$START_FIREHOL" = "NO"  ] && return 4
        /sbin/firehol start "$@" > /dev/null 2>&1 || return 1

Now we finally get a result and with INIT_VERBOSE=yes set we do indeed get some useful output:

Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 33: dirname: command not found
Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 33: cd: HOME not set
Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 33: basename: command not found
Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 36: dirname: command not found
Nov 27 17:59:38 firehol[27095]: Cannot access /install.config
Nov 27 17:59:38 firehol[27095]:    ...fail!

And this is basically yelling at us that the PATH variable is not set because the script can not find and execute required commands. Sadly this fail is not catched or logged without verbose information and thanks to the /dev/null redirect at all.

At first glance I was going to blame systemd isolating the script from environment variables but that was too fast because setting it explicit changed nothing. To blame is the old set-up logic of the init script /etc/init.d/firehol right at the top not allowing /usr/bin where dirname or basename and others are found.

PATH=/bin:/sbin
NAME=firehol
DESC="firewall"
SCRIPTNAME=/etc/init.d/$NAME

test -x /sbin/firehol || exit 0

[ -r /etc/default/firehol ] && set -a && . /etc/default/firehol

I compared the /sbin/firehol script of version 2 with version 3 and there is a subtle difference at the start in version 2 that is missing in version 3:

# EXTERNAL/SYSTEM COMMANDS MANAGEMENT
#
# ------------------------------------------------------------------------------
# XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# ------------------------------------------------------------------------------

export PATH="${PATH}:/bin:/usr/bin:/sbin:/usr/sbin"

I’d argue that version 3 missing this is more correct because setting up the PATH is really the job of the system that is running the script. So basically SysVinit or systemd. Sadly that doesn’t help us here and fiddling with a maintainer provided file is a no go because this will be erased on the next update (if any). Luckily we can see from the init script /etc/init.d/firehol that it also sources the file /etc/default/firehol. This means we can set any additional environment variable here:

# FireHOL application default file
# sourced by the initscript `/etc/init.d/firehol'.

PATH="${PATH}:/bin:/usr/bin:/sbin:/usr/sbin"

# To enable firehol at startup set START_FIREHOL=YES (init script variable)
START_FIREHOL=YES

After editing this file we finally get some more information and our iptables are piling up with rules again.

● firehol.service - LSB: firehol firewall configuration
   Loaded: loaded (/etc/init.d/firehol; bad; vendor preset: enabled)
  Drop-In: /etc/systemd/system/firehol.service.d
   Active: active (exited) since Fr 2020-11-27 18:17:41 CET; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14337 ExecStop=/etc/init.d/firehol stop (code=exited, status=0/SUCCESS)
  Process: 14511 ExecStart=/etc/init.d/firehol start (code=exited, status=0/SUCCESS)

Nov 27 18:17:39 systemd[1]: Starting LSB: firehol firewall configuration...
Nov 27 18:17:39 firehol[14511]: Params
Nov 27 18:17:39 firehol[14511]: FireHOL: Saving active firewall to a temporary file...  OK
Nov 27 18:17:40 firehol[14511]: FireHOL: Processing file '//etc/firehol/firehol.conf'...  OK  (470 iptables rules)
Nov 27 18:17:41 firehol[14511]: FireHOL: Activating ipsets...  OK
Nov 27 18:17:41 firehol[14511]: FireHOL: Fast activating new firewall...  OK
Nov 27 18:17:41 firehol[14511]: FireHOL: Saving activated firewall to '//var/spool/firehol'...  OK
Nov 27 18:17:41 systemd[1]: Started LSB: firehol firewall configuration.

Personally I can’t wait for all init scripts to sink into oblivion because debugging this sort of errors is hard and a waste of time and usually revolves about problems solved already in many different ways before – each falling flat in some corner case.