Tag: systemd

I’m making heavy use of Hetzner storageboxes, a rather slow network storage solution perfectly suited for backup tasks. Sadly they come with hickups resulting sometimes in a failed automount status where the machine may not recover from on it’s own. That’s so common that I got tired of checking monitoring or logging in to each individual machine. and to my aid!

for i in {bob,alice,steam,punk,younameit}; do echo "$i:"; systemctl --host "$i" status storagebox.automount; done
So how are my storagebox mounts today? Ah yes, yes, seems like two need a little pep talk.
Bash: Import variables from a .env file by tokudantokudan (Random texts)
This sets and exports all variables set in a .env file that can also be used by systemd to setup environment variables. Read and export ...

Yw, yes set -a / set +a before and after sourcing a file in bash makes this easier. I found out when I needed this to quickly source an env file used by a script that is launched and configured by systemd like this:

EnvironmentFile=%h/.config/foo.env

Sometimes I’ve to run this script manually so I do this to source the same config:

set -a
. /path/to/.config/foo.env
set +a
foo --bar

Posted on
by

tl;dr: Add PATH="${PATH}:/bin:/usr/bin:/sbin:/usr/sbin" to /etc/default/firehol when using backported version 3 of firehol on Ubuntu.

firehol – an easy to use but powerful iptables stateful firewall

man firehol

With this out of the way: When installing firehol on aging Xenial (Ubuntu 16.04) you want the backported packages by Andrey Galkin to get version 3 of firehol over version 2 in universe – especially when working with IPv6: https://launchpad.net/~andvgal

When done setting up your rules you may find out after a reboot that the systemd job involved will claim to have started firehol but eventually discover that your iptables are empty despite systemd claiming otherwhise and having set START_FIREHOL=YES in /etc/default/firehol:

● firehol.service - LSB: firehol firewall configuration
   Loaded: loaded (/etc/init.d/firehol; bad; vendor preset: enabled)
   Active: active (exited) since Fr 2020-11-27 15:43:51 CET; 2h 8min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 31555 ExecStop=/etc/init.d/firehol stop (code=exited, status=0/SUCCESS)
  Process: 31574 ExecStart=/etc/init.d/firehol start (code=exited, status=0/SUCCESS)

This is especially weird when you run the startup /sbin/firehol start command manually and it succeeds just fine.

I had to dig deep to find out where the script is in fact falling flat. This was mostly because of old init script /etc/init.d/firehol redirecting the output of the starting process to /dev/null not showing the errors at all:

do_start () {
        # return
        #  0 000 if firewall has been handled
        #  1 001 if firewall could not be activated
        #  4 100 if FireHOL is disabled via /etc/default/firehol
        [ "$START_FIREHOL" = "NO"  ] && return 4
        /sbin/firehol start "$@" > /dev/null 2>&1 || return 1

Now we finally get a result and with INIT_VERBOSE=yes set we do indeed get some useful output:

Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 33: dirname: command not found
Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 33: cd: HOME not set
Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 33: basename: command not found
Nov 27 17:59:38 firehol[27095]: /sbin/firehol: line 36: dirname: command not found
Nov 27 17:59:38 firehol[27095]: Cannot access /install.config
Nov 27 17:59:38 firehol[27095]:    ...fail!

And this is basically yelling at us that the PATH variable is not set because the script can not find and execute required commands. Sadly this fail is not catched or logged without verbose information and thanks to the /dev/null redirect at all.

At first glance I was going to blame systemd isolating the script from environment variables but that was too fast because setting it explicit changed nothing. To blame is the old set-up logic of the init script /etc/init.d/firehol right at the top not allowing /usr/bin where dirname or basename and others are found.

PATH=/bin:/sbin
NAME=firehol
DESC="firewall"
SCRIPTNAME=/etc/init.d/$NAME

test -x /sbin/firehol || exit 0

[ -r /etc/default/firehol ] && set -a && . /etc/default/firehol

I compared the /sbin/firehol script of version 2 with version 3 and there is a subtle difference at the start in version 2 that is missing in version 3:

# EXTERNAL/SYSTEM COMMANDS MANAGEMENT
#
# ------------------------------------------------------------------------------
# XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# ------------------------------------------------------------------------------

export PATH="${PATH}:/bin:/usr/bin:/sbin:/usr/sbin"

I’d argue that version 3 missing this is more correct because setting up the PATH is really the job of the system that is running the script. So basically SysVinit or systemd. Sadly that doesn’t help us here and fiddling with a maintainer provided file is a no go because this will be erased on the next update (if any). Luckily we can see from the init script /etc/init.d/firehol that it also sources the file /etc/default/firehol. This means we can set any additional environment variable here:

# FireHOL application default file
# sourced by the initscript `/etc/init.d/firehol'.

PATH="${PATH}:/bin:/usr/bin:/sbin:/usr/sbin"

# To enable firehol at startup set START_FIREHOL=YES (init script variable)
START_FIREHOL=YES

After editing this file we finally get some more information and our iptables are piling up with rules again.

● firehol.service - LSB: firehol firewall configuration
   Loaded: loaded (/etc/init.d/firehol; bad; vendor preset: enabled)
  Drop-In: /etc/systemd/system/firehol.service.d
   Active: active (exited) since Fr 2020-11-27 18:17:41 CET; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14337 ExecStop=/etc/init.d/firehol stop (code=exited, status=0/SUCCESS)
  Process: 14511 ExecStart=/etc/init.d/firehol start (code=exited, status=0/SUCCESS)

Nov 27 18:17:39 systemd[1]: Starting LSB: firehol firewall configuration...
Nov 27 18:17:39 firehol[14511]: Params
Nov 27 18:17:39 firehol[14511]: FireHOL: Saving active firewall to a temporary file...  OK
Nov 27 18:17:40 firehol[14511]: FireHOL: Processing file '//etc/firehol/firehol.conf'...  OK  (470 iptables rules)
Nov 27 18:17:41 firehol[14511]: FireHOL: Activating ipsets...  OK
Nov 27 18:17:41 firehol[14511]: FireHOL: Fast activating new firewall...  OK
Nov 27 18:17:41 firehol[14511]: FireHOL: Saving activated firewall to '//var/spool/firehol'...  OK
Nov 27 18:17:41 systemd[1]: Started LSB: firehol firewall configuration.

Personally I can’t wait for all init scripts to sink into oblivion because debugging this sort of errors is hard and a waste of time and usually revolves about problems solved already in many different ways before – each falling flat in some corner case.

Posted on
by

Most Linux users know some systemd by now. There is systemd stop $someservice and systemd disable $someservice. If you think that’s all one has to know about systemd you’re missing out on a lof of features. That’s not the story for today tho.

Some services tend to come back even when disabled and stopped. Socket services come to mind. This is e.g. the case for rpcbind starting a portmapper service on tcp/111 that can be used for a DDoS reflection attack. There are various ways to deal with this from uninstall to firewalling or to mask this service. Since everbody on Linux gets to pick their own poision I decided to use mask to make sure this can simply no longer start without having to break dependencies by removing it or fiddle with the firewall:

systemctl stop rpcbind.service
systemctl mask rpcbind.service
Created symlink /etc/systemd/system/rpcbind.service → /dev/null.
systemctl daemon-reload

A masked service can not even be restarted manually any more. It’s dead in the water as long as this symlink exists.

Read more on this from Poettering himself:

http://0pointer.de/blog/projects/three-levels-of-off