unix / systemd

updated November 05, 2026

I run long-lived processes on Linux VMs as systemd services so they survive SSH logout and restart on crash or reboot.

A service unit

This unit runs one runner from apps / sockeye:

# /etc/systemd/system/sockeye-runner.service
[Unit]
Description=sockeye runner
After=network-online.target
Wants=network-online.target

[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
ExecStart=/usr/local/bin/soc runner
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Install and start it:

sudo cp sockeye-runner.service /etc/systemd/system/
sudo systemctl enable --now sockeye-runner

enable starts it on every boot and --now starts it now. Restart=always restarts it after it exits.

The service runs as a dedicated system user, never a human account, so app state and credentials stay out of a person's home:

sudo useradd --system -d /var/lib/sockeye -m -s /usr/sbin/nologin sockeye

StateDirectory=sockeye creates the state directory and sets its owner. It does not create the user; User=sockeye requires an existing account.

Template units

A template unit, marked by the @ suffix, runs several copies of one process. %i in the unit is the instance name:

# /etc/systemd/system/[email protected]
[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
Environment=RUNNER_SLOT=%i
ExecStart=/usr/local/bin/soc runner
Restart=always

sudo systemctl enable --now sockeye-runner@{1..4}

That starts sockeye-runner@1 through @4. Each runner keeps its own workspace, named for its instance.

Environment variables

systemd gives a process a minimal environment, so I keep config and secrets in an EnvironmentFile. The path /etc/sockeye.env is my choice. Debian uses /etc/default/<name> and Red Hat uses /etc/sysconfig/<name>.

# /etc/sockeye.env
SERVER_URL=https://sockeye.example.com
RUNNER_KEY=/var/lib/sockeye/.ssh/runner_key
PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin

The default PATH is minimal, so I set it when the process shells out to go or git. The file holds secrets, so I restrict it: sudo chown root:root /etc/sockeye.env && sudo chmod 600 /etc/sockeye.env.

Drop-ins

A drop-in adds lines to a unit without editing the unit file. systemd reads every .conf file in the unit's .d directory after the unit, and a template's directory applies to every instance. A runner's private route to the server is a second environment file, which the deploy writes from the instance's config:

# /etc/systemd/system/[email protected]/route.conf
[Service]
EnvironmentFile=/etc/sockeye-runner-route.env

The unit file stays the one in the repo, the secrets stay in their own file, and the deploy owns only the file it writes. systemctl cat sockeye-runner@1 prints the unit with each drop-in after it.

Stopping

On stop, systemd sends SIGTERM to every process in the unit by default. The sockeye jobs process runs deploy commands as children. Its unit gives a running deploy time to finish before it stops:

[Service]
KillMode=mixed
TimeoutStopSec=11min

The jobs process stops taking work and waits for its active deploy. A deploy has a 10-minute limit. After 11 minutes, mixed sends SIGKILL to every process that remains.

Watchdog

Restart=always restarts a process that exits. It does not restart a process that runs and stops answering. The sockeye server unit sets a watchdog for that case:

[Service]
WatchdogSec=60s
TimeoutAbortSec=30s

The server sends WATCHDOG=1 to systemd every 15 seconds, and only after its own /healthz answers 200. When 60 seconds pass with no message, systemd sends SIGABRT. Go then writes the stack of each goroutine to the journal and exits, and Restart=always starts the server again. A process that is still alive 30 seconds after SIGABRT gets SIGKILL.

Transient units

systemd-run starts a command as a unit with no unit file. The sockeye runner starts each CI check this way, so code from a branch gets the sandbox of a service:

systemd-run --wait --pipe --collect --quiet \
  --property=User=sockeye-check-$slot \
  --property=ProtectSystem=strict \
  --property=ReadWritePaths=/var/lib/sockeye-jobs/$slot \
  --property=InaccessiblePaths=-/var/lib/sockeye \
  --property=PrivateTmp=yes \
  --property=NoNewPrivileges=yes \
  --property=KillMode=control-group \
  --property=RuntimeMaxSec=$timeout \
  --property=LoadCredential=pg-password:/var/lib/sockeye-pg/<repo> \
  /bin/bash -eo pipefail -c "$command"

--wait and --pipe connect the unit to the caller, so the runner reads the output and the exit code as for a child process. Each runner slot has its own user. The check can write only its job directory, and it cannot open the runner's key. KillMode=control-group and RuntimeMaxSec stop each process of the check, a double fork too, when it ends or times out.

LoadCredential gives the check its database password as a file under $CREDENTIALS_DIRECTORY. The password is not in the command line or in the unit's Environment property, which any local user can read with systemctl show.

Everyday commands

sudo systemctl restart sockeye-server
sudo systemctl stop sockeye-server
systemctl is-active sockeye-server
systemctl status sockeye-server
systemctl show sockeye-server -p NRestarts --value

After editing a unit file, reload systemd's view before restarting:

sudo systemctl daemon-reload

Timers

A timer replaces a cron line. It is a second unit that starts a service. The service has Type=oneshot, so systemd runs the command once and waits for it to end:

# /etc/systemd/system/sports-ingest.service
[Service]
Type=oneshot
User=sports
ExecStart=/srv/sports/app/sports -ingest

# /etc/systemd/system/sports-ingest.timer
[Timer]
OnCalendar=*:0/30

[Install]
WantedBy=timers.target

The timer is the unit I enable:

sudo systemctl enable --now sports-ingest.timer
systemctl list-timers

A cron line writes no log unless I pipe it to systemd-cat. A timer's service logs to the journal under its own name, so journalctl -u sports-ingest shows every run. systemctl status sports-ingest shows the exit code of the last run. The service also takes the same hardening lines as any other unit, such as NoNewPrivileges=true.

Logs

Each service prints plain text to stdout and stderr. systemd captures it into its journal, so I read logs with journalctl. Every unit's output is addressable by name:

journalctl -u sockeye-server         # everything for the unit
journalctl -u sockeye-server -n 50   # last 50 lines
journalctl -u sockeye-server -f      # follow live, like tail -f
journalctl -u sockeye-server -e      # jump to the end

A template unit is addressable by instance:

journalctl -u 'sockeye-runner@1'
journalctl -u 'sockeye-runner@*'     # all instances

By time:

journalctl -u sockeye-server --since "10 min ago"
journalctl -u sockeye-server --since today
journalctl -u sockeye-server --since "2026-07-19 17:00" --until "18:00"

By boot and severity:

journalctl -b        # since the current boot
journalctl -b -1     # the previous boot
journalctl -p err    # errors and worse, any unit

Filter with grep, or with the journal's own matching:

journalctl -u sockeye-server | grep error
journalctl -u sockeye-server -g connect   # journal-native regex

There are no log file paths to remember and no logrotate to configure. journalctl reads the same way for every service on the box.

A root relay for the journal

The sockeye server shows logs on its pages, but it runs as www-data, which cannot read the journal. If I added www-data to the systemd-journal group, a hijacked web process could read every unit.

So a small relay runs as root. It listens on a unix socket that only the www-data group can open. A request is one JSON line with a unit, a time range, and a search. The relay answers only for a unit listed in a file that only root can edit. It checks each field and builds the journalctl arguments itself, so no text from the caller becomes a flag.

The relay compares a unit name with the file as text. The name in soc service units must match a line in the allowlist exactly. A service that lists app@8081 gets no lines when the allowlist holds app@*. List app@* in both places.

← All articles