go / wakeups
Some parts of a web app must react to a database write soon after it happens. A dashboard shows old data after the next write. A command-line client that waits for a result has to guess how long to sleep. A background worker that waits for a job has to ask again and again.
I answer all three with a signal that carries no data. Something says "this changed". Go passes that signal to whoever waits. Each waiter re-reads what it cares about. The counter poll normally wakes a reader within a second. A periodic wake covers a signal lost during a process exit.
The signal comes from one of two places. In my largest web app, a
Postgres trigger wakes the job queue. In sockeye, my
continuous integration server, the code that wrote the row wakes the
waiters in its process, and a counter table carries the signal to the
other web process. sockeye's waiters are runner boxes, a
soc show --wait command, and every open dashboard page. This
article covers the server side.
web / live regions covers what a page does when
the signal arrives.
The Postgres signal
The job queue keeps each job as a row (see
go / job queues). A trigger fires pg_notify when a
pending job arrives, on a channel named for its queue:
CREATE FUNCTION notify_pending_job ()
RETURNS TRIGGER
AS $$
BEGIN
PERFORM pg_notify('jobs_' || NEW.queue, '');
RETURN NULL;
END;
$$
LANGUAGE plpgsql;
CREATE TRIGGER notify_pending_job
AFTER INSERT ON jobs
FOR EACH ROW
WHEN (NEW.status = 'pending')
EXECUTE FUNCTION notify_pending_job ();
The notification has no payload. A payload is a second schema, and it drifts from the table it describes. A poller that gets the bare signal reads the rows it needs, so it never reads an old copy. Postgres folds identical notifications in one transaction into one, so a bulk insert wakes a queue once. A spurious wakeup costs one small query, so I do not try to suppress them.
The trigger fires from Postgres rather than from the Go code that wrote
the row. Every write path then fires it: the handler, a psql
session, a migration, and code I write next year and forget about.
Postgres also delivers the notification at commit, so a poller that
wakes sees the row.
The listener connection
A connection that waits for a notification serves no query. So one
connection, outside the pool's normal use, runs LISTEN on every
queue's channel and calls back with the channel each notification
arrives on:
func (db *DB) Listen(ctx context.Context, channels []string, ready func(), onNotify func(channel string)) error
A loop around it listens again after a failure, with a pause so a
database that is down does not spin the loop. Postgres drops a
notification that nobody listens for. So ready runs once the
LISTEN stands, and each poller reads its queue then, before it
waits. The poller also reads on an interval, which covers a lost
notification. A connection that waits writes nothing, so the TCP
keepalive that Go's dialer turns on is what ends a wait on a peer that
is gone.
Fan-out
Inside a sockeye process, a subscriber is a channel with a buffer of
one. Each write path calls wake after its commit:
func (s *server) subscribe() chan struct{} {
ch := make(chan struct{}, 1)
s.subsMu.Lock()
s.subs[ch] = struct{}{}
s.subsMu.Unlock()
return ch
}
func (s *server) unsubscribe(ch chan struct{}) {
s.subsMu.Lock()
delete(s.subs, ch)
s.subsMu.Unlock()
}
func (s *server) wake() {
s.subsMu.Lock()
defer s.subsMu.Unlock()
for ch := range s.subs {
select {
case ch <- struct{}{}:
default:
}
}
}
The buffer of one and the non-blocking send do two jobs. A burst of writes coalesces into one pending wakeup, so a sweep of result rows wakes a subscriber once. A subscriber that is slow to read cannot block the listener, so one stuck request cannot stall every other waiter.
The subscriber gets no data from the channel. It re-reads.
A waiter subscribes before its first read:
wake := s.subscribe()
defer s.unsubscribe(wake)
if settled := s.read(); settled {
return
}
<-wake
In the other order there is a gap between the read and the subscribe. A write that lands in the gap wakes nobody, and the waiter sleeps until the write after it.
Across web processes
sockeye runs two web processes. A write wakes the subscribers in the
process that handled it, and a reader on the other process must hear
of it too. A wake_counter table carries the signal, with one row
for each repo. After its commit, a write adds one to its repo's row:
INSERT INTO wake_counter (repo, n)
VALUES ($1, 1)
ON CONFLICT (repo)
DO UPDATE SET n = wake_counter.n + 1
Each process reads the table once a second. It wakes its own subscribers for each repo whose count moved since its last read. A reader on the other process can wake late, but within a minute at most:
- The add is not part of the write's transaction, so no write waits on another's row. An add that fails is tried again on the next tick.
- Many writes to one repo between two ticks make one add.
- A process compares each count for difference, not order, so a missed tick catches up on the next one.
- A process's first read sets its baseline and wakes nobody, so a restart does not wake every reader.
- A process that stops between its commit and its add loses that wake. So each process wakes all of its readers once a minute.
A dashboard page asks only for its own repo, so a write to one repo does not make every open page fetch again.
Long polling
A runner that wants a job sends the state it last saw. The server answers when the state differs, or when a timeout ends. The answer is the current state either way:
func (s *server) longPoll(ctx context.Context, old State) State {
ctx, cancel := context.WithTimeout(ctx, PollWait)
defer cancel()
c := make(chan State, 1)
go func() { c <- s.waitNewState(old) }()
select {
case cur := <-c:
return cur
case <-ctx.Done():
return old
}
}
The state a runner waits on comes from the database, not from the process's memory. A process can be a second behind the other one, and a runner's poll can reach either.
The timeout answers with a state, not with an error. A client cannot tell a request held open forever from a server that stopped. A request that answers every 25 seconds with "no news" proves the server is up.
PollWait is one constant that both sides import. The client's HTTP
timeout is PollWait plus a margin. A client that gives up first
aborts every idle poll, throws the connection away, and logs a timeout
for a server that answered correctly.
Replacing sleep
A script that pushes a commit and then asks how the checks went has to guess how long to wait. A guess too long wastes time nobody needed. A guess too short reads a green answer about the commit before the one it pushed. No script gets the number right, because the number is the length of somebody's test suite.
The CLI asks the server to hold the request instead. The server subscribes, checks whether the head has settled, and waits for a wakeup or the deadline before it reads again:
func (s *server) waitForHead(ctx context.Context, c *change, sha string) (*change, error) {
wake := s.subscribe()
defer s.unsubscribe(wake)
deadline := time.NewTimer(PollWait)
defer deadline.Stop()
for {
settled, err := s.headSettled(ctx, c, sha)
if err != nil || settled {
return c, err
}
select {
case <-wake:
case <-deadline.C:
return c, nil
case <-ctx.Done():
return nil, ctx.Err()
}
if c, err = s.findChange(ctx, c.ID); err != nil {
return nil, err
}
}
}
The CLI loops on that request until the change settles or its own timeout runs out. An answer before the end still includes the checks that finished, so the CLI prints one line per check as it lands.
The sha is the commit the CLI waits on. A wait that reaches the
server before the push hook records the new head would read the old
head's checks: green, about the wrong code. The server treats a head
that is not sha yet as not settled. A sleep hid that mistake and
did not fix it.
Server-Sent Events
The same subscription sits behind a Server-Sent Events endpoint. Each
wakeup writes an empty event named wake, and a comment line every
15 seconds keeps a proxy from closing an idle connection:
func (s *server) wakeStream(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache, no-store, no-transform")
w.Header().Set("X-Accel-Buffering", "no")
flush := w.(http.Flusher).Flush
flush()
ch := s.subscribe()
defer s.unsubscribe(ch)
keepalive := time.NewTicker(15 * time.Second)
defer keepalive.Stop()
for {
select {
case <-ch:
io.WriteString(w, "event: wake\ndata:\n\n")
case <-keepalive.C:
io.WriteString(w, ": keepalive\n\n")
case <-req.Context().Done():
return
}
flush()
}
}
The event says nothing about what changed. The page decides what to re-read.
Fit
This fits a system whose state is small enough for each waiter to re-read on every wakeup, and whose writes all go through one Postgres database. The subscribers live in process memory, and each process wakes its own. Postgres carries the signal between processes, as a notification or as a counter row. There is no message broker and no schema for events.