unix / systemd

updated October 14, 2026

I run long-lived processes on Linux VMs as systemd services so they survive SSH logout and restart on crash or reboot.

A service unit

This unit runs one runner from cmd / sockeye:

# /etc/systemd/system/sockeye-runner.service
[Unit]
Description=sockeye runner
After=network-online.target
Wants=network-online.target

[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
ExecStart=/usr/local/bin/soc runner
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Install and start it:

sudo cp sockeye-runner.service /etc/systemd/system/
sudo systemctl enable --now sockeye-runner

enable starts it on every boot and --now starts it now. Restart=always restarts it after it exits.

The service runs as a dedicated system user, never a human account, so app state and credentials stay out of a person's home:

sudo useradd --system -d /var/lib/sockeye -m -s /usr/sbin/nologin sockeye

If I skip the useradd step, StateDirectory=sockeye makes systemd create /var/lib/sockeye owned by that user.

Template units

A template unit, marked by the @ suffix, runs several copies of one process. %i in the unit is the instance name:

# /etc/systemd/system/[email protected]
[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
Environment=RUNNER_SLOT=%i
ExecStart=/usr/local/bin/soc runner
Restart=always

sudo systemctl enable --now sockeye-runner@{1..4}

That starts sockeye-runner@1 through @4. Each runner keeps its own workspace, named for its instance.

The instance name can also be a port. The sockeye server runs two blue/green slots, sockeye-server@1994 and sockeye-server@1995, and its unit sets Environment=PORT=%i. A deploy boots the idle slot on the new build and stops the old one.

Environment variables

systemd gives a process a minimal environment, so I keep config and secrets in an EnvironmentFile. The path /etc/sockeye.env is my choice. Debian uses /etc/default/<name> and Red Hat uses /etc/sysconfig/<name>.

# /etc/sockeye.env
SERVER_URL=https://sockeye.example.com
RUNNER_KEY=/var/lib/sockeye/.ssh/runner_key
PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin

The default PATH is minimal, so I set it when the process shells out to go or git. The file holds secrets, so I restrict it: sudo chown root:root /etc/sockeye.env && sudo chmod 600 /etc/sockeye.env.

Everyday commands

sudo systemctl restart sockeye-server@1994
sudo systemctl stop sockeye-server@1994
systemctl is-active sockeye-server@1994
systemctl status sockeye-server@1994
systemctl show sockeye-server@1994 -p NRestarts --value

After editing a unit file, reload systemd's view before restarting:

sudo systemctl daemon-reload

Timers

A timer replaces a cron line. It is a second unit that starts a service. The service has Type=oneshot, so systemd runs the command once and waits for it to end:

# /etc/systemd/system/sports-ingest.service
[Service]
Type=oneshot
User=sports
ExecStart=/srv/sports/app/sports -ingest

# /etc/systemd/system/sports-ingest.timer
[Timer]
OnCalendar=*:0/30

[Install]
WantedBy=timers.target

The timer is the unit I enable:

sudo systemctl enable --now sports-ingest.timer
systemctl list-timers

A cron line writes no log unless I pipe it to systemd-cat. A timer's service logs to the journal under its own name, so journalctl -u sports-ingest shows every run. systemctl status sports-ingest shows the exit code of the last run. The service also takes the same hardening lines as any other unit, such as NoNewPrivileges=true.

Logs

Each service prints plain text to stdout and stderr. systemd captures it into its journal, so I read logs with journalctl. Every unit's output is addressable by name:

journalctl -u sockeye-server@1994         # everything for the unit
journalctl -u sockeye-server@1994 -n 50   # last 50 lines
journalctl -u sockeye-server@1994 -f      # follow live, like tail -f
journalctl -u sockeye-server@1994 -e      # jump to the end

A template unit is addressable by instance:

journalctl -u 'sockeye-runner@1'
journalctl -u 'sockeye-runner@*'     # all instances

By time:

journalctl -u sockeye-server@1994 --since "10 min ago"
journalctl -u sockeye-server@1994 --since today
journalctl -u sockeye-server@1994 --since "2026-07-19 17:00" --until "18:00"

By boot and severity:

journalctl -b        # since the current boot
journalctl -b -1     # the previous boot
journalctl -p err    # errors and worse, any unit

Filter with grep, or with the journal's own matching:

journalctl -u sockeye-server@1994 | grep error
journalctl -u sockeye-server@1994 -g connect   # journal-native regex

There are no log file paths to remember and no logrotate to configure. journalctl reads the same way for every service on the box.

A root relay for the journal

The sockeye server shows logs on its pages, but it runs as www-data, which cannot read the journal. If I added www-data to the systemd-journal group, a hijacked web process could read every unit.

So a small relay runs as root. It listens on a unix socket that only the www-data group can open. A request is one JSON line with a unit, a time range, and a search. The relay answers only for a unit listed in a file that only root can edit. It checks each field and builds the journalctl arguments itself, so no text from the caller becomes a flag.

The relay compares a unit name with the file as text. The name in soc service units must match a line in the allowlist exactly. A service that lists app@8081 gets no lines when the allowlist holds app@*. List app@* in both places.

← All articles