linux / systemd

updated October 09, 2026

I run long-lived processes on Linux VMs as systemd services so they survive SSH logout and restart on crash or reboot.

A service unit

This unit runs one runner from cmd / sockeye:

# /etc/systemd/system/sockeye-runner.service
[Unit]
Description=sockeye runner
After=network-online.target
Wants=network-online.target

[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
ExecStart=/usr/local/bin/soc runner
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Install and start it:

sudo cp sockeye-runner.service /etc/systemd/system/
sudo systemctl enable --now sockeye-runner

enable starts it on every boot and --now starts it now. Restart=always restarts it after it exits.

The service runs as a dedicated system user, never a human account, so app state and credentials stay out of a person's home:

sudo useradd --system -d /var/lib/sockeye -m -s /usr/sbin/nologin sockeye

If I skip the useradd step, StateDirectory=sockeye makes systemd create /var/lib/sockeye owned by that user.

Template units

A template unit, marked by the @ suffix, runs several copies of one process. %i in the unit is the instance name:

# /etc/systemd/system/[email protected]
[Service]
User=sockeye
StateDirectory=sockeye
EnvironmentFile=/etc/sockeye.env
Environment=RUNNER_SLOT=%i
ExecStart=/usr/local/bin/soc runner
Restart=always

sudo systemctl enable --now sockeye-runner@{1..4}

That starts sockeye-runner@1 through @4. Each runner keeps its own workspace, named for its instance.

The instance name can also be a port. The sockeye server runs two blue/green slots, sockeye-server@1994 and sockeye-server@1995, and its unit sets Environment=PORT=%i. A deploy boots the idle slot on the new build and stops the old one.

Environment variables

systemd gives a process a minimal environment, so I keep config and secrets in an EnvironmentFile. The path /etc/sockeye.env is my choice. Debian uses /etc/default/<name> and Red Hat uses /etc/sysconfig/<name>.

# /etc/sockeye.env
SERVER_URL=https://sockeye.example.com
RUNNER_KEY=/var/lib/sockeye/.ssh/runner_key
PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin

The default PATH is minimal, so I set it when the process shells out to go or git. The file holds secrets, so I restrict it: sudo chown root:root /etc/sockeye.env && sudo chmod 600 /etc/sockeye.env.

Everyday commands

sudo systemctl restart sockeye-server@1994
sudo systemctl stop sockeye-server@1994
systemctl is-active sockeye-server@1994
systemctl status sockeye-server@1994
systemctl show sockeye-server@1994 -p NRestarts --value

After editing a unit file, reload systemd's view before restarting:

sudo systemctl daemon-reload

Logs

Each service prints plain text to stdout and stderr. systemd captures it into its journal, so I read logs with journalctl. Every unit's output is addressable by name:

journalctl -u sockeye-server@1994         # everything for the unit
journalctl -u sockeye-server@1994 -n 50   # last 50 lines
journalctl -u sockeye-server@1994 -f      # follow live, like tail -f
journalctl -u sockeye-server@1994 -e      # jump to the end

A template unit is addressable by instance:

journalctl -u 'sockeye-runner@1'
journalctl -u 'sockeye-runner@*'     # all instances

By time:

journalctl -u sockeye-server@1994 --since "10 min ago"
journalctl -u sockeye-server@1994 --since today
journalctl -u sockeye-server@1994 --since "2026-07-19 17:00" --until "18:00"

By boot and severity:

journalctl -b        # since the current boot
journalctl -b -1     # the previous boot
journalctl -p err    # errors and worse, any unit

Filter with grep, or with the journal's own matching:

journalctl -u sockeye-server@1994 | grep error
journalctl -u sockeye-server@1994 -g connect   # journal-native regex

There are no log file paths to remember and no logrotate to configure. journalctl reads the same way for every service on the box.

← All articles