linux / systemd

updated October 03, 2026

I run long-lived processes on Linux VMs as systemd services so they survive SSH logout and restart on crash or reboot. systemd also captures each service's output, so journalctl reads it back.

A service unit

This unit runs one runner from cmd / sockeye:

# /etc/systemd/system/sockeye-runner.service
[Unit]
Description=sockeye runner
After=network-online.target
Wants=network-online.target

[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
ExecStart=/usr/local/bin/sockeye runner
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Install and start it:

sudo cp sockeye-runner.service /etc/systemd/system/
sudo systemctl enable --now sockeye-runner

enable starts it on every boot and --now starts it now. Restart=always restarts it after it exits.

The service runs as a dedicated system user, never a human account, so app state and credentials stay out of a person's home:

sudo useradd --system -d /var/lib/sockeye -m -s /usr/sbin/nologin sockeye

StateDirectory=sockeye makes systemd create /var/lib/sockeye owned by that user if the useradd step is skipped, so the unit carries the invariant either way.

Template units

A template unit, marked by the @ suffix, runs several copies of one process. %i in the unit is the instance name:

# /etc/systemd/system/[email protected]
[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
Environment=RUNNER_SLOT=%i
ExecStart=/usr/local/bin/sockeye runner
Restart=always

sudo systemctl enable --now sockeye-runner@{1..4}

That starts sockeye-runner@1 through @4. Each runner keeps its own workspace, named for its instance.

The instance name can also be a port. The sockeye server runs two blue/green slots, sockeye-server@1994 and sockeye-server@1995, and its unit sets Environment=PORT=%i. A deploy boots the idle slot on the new build and stops the old one.

Environment variables

systemd gives a process a minimal environment, so I keep config and secrets in an EnvironmentFile. The path /etc/sockeye.env is my choice. Debian uses /etc/default/<name> and Red Hat uses /etc/sysconfig/<name>.

# /etc/sockeye.env
SERVER_URL=https://sockeye.example.com
BOX_TOKEN=...
PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin

The default PATH is minimal, so I set it when the process shells out to go or git. The file holds secrets, so I restrict it: sudo chown root:root /etc/sockeye.env && sudo chmod 600 /etc/sockeye.env.

Everyday commands

sudo systemctl restart sockeye-server@1994
sudo systemctl stop sockeye-server@1994
systemctl is-active sockeye-server@1994
systemctl status sockeye-server@1994
systemctl show sockeye-server@1994 -p NRestarts --value

After editing a unit file, reload systemd's view before restarting:

sudo systemctl daemon-reload

Logs

systemd captures each service's stdout and stderr into its journal, so I read logs with journalctl rather than hunting for files. Every unit's output is addressable by name:

journalctl -u sockeye-server@1994         # everything for the unit
journalctl -u sockeye-server@1994 -n 50   # last 50 lines
journalctl -u sockeye-server@1994 -f      # follow live, like tail -f
journalctl -u sockeye-server@1994 -e      # jump to the end

A template unit is addressable by instance:

journalctl -u 'sockeye-runner@1'
journalctl -u 'sockeye-runner@*'     # all instances

By time:

journalctl -u sockeye-server@1994 --since "10 min ago"
journalctl -u sockeye-server@1994 --since today
journalctl -u sockeye-server@1994 --since "2026-07-19 17:00" --until "18:00"

By boot and severity:

journalctl -b        # since the current boot
journalctl -b -1     # the previous boot
journalctl -p err    # errors and worse, any unit

Filter with grep, or with the journal's own matching:

journalctl -u sockeye-server@1994 | grep error
journalctl -u sockeye-server@1994 -g connect   # journal-native regex

The service prints plain text and systemd stores it. There are no log file paths to remember and no logrotate to configure, and journalctl reads the same way for every service on the box.

← All articles