go / wakeups

Some parts of a web app must react to a database write soon after it happens. A dashboard shows old data after the next write. A command-line client that waits for a result has to guess how long to sleep. A background worker that waits for a job has to ask again and again.

I answer all three with a signal that carries no data. Something says "this changed". Go passes that signal to whoever waits. Each waiter re-reads what it cares about. The counter poll normally wakes a reader within a second. A periodic wake covers a signal lost during a process exit.

The signal comes from one of two places. In my largest web app, a Postgres trigger wakes the job queue. In sockeye, my continuous integration server, the code that wrote the row wakes the waiters in its process, and a counter table carries the signal to the other web process. sockeye's waiters are runner boxes, a soc show --wait command, and every open dashboard page. This article covers the server side. web / live regions covers what a page does when the signal arrives.

The Postgres signal

The job queue keeps each job as a row (see go / job queues). A trigger fires pg_notify when a pending job arrives, on a channel named for its queue:

CREATE FUNCTION notify_pending_job ()
  RETURNS TRIGGER
  AS $$
BEGIN
  PERFORM pg_notify('jobs_' || NEW.queue, '');
  RETURN NULL;
END;
$$
LANGUAGE plpgsql;

CREATE TRIGGER notify_pending_job
  AFTER INSERT ON jobs
  FOR EACH ROW
  WHEN (NEW.status = 'pending')
  EXECUTE FUNCTION notify_pending_job ();

The notification has no payload. A payload is a second schema, and it drifts from the table it describes. A poller that gets the bare signal reads the rows it needs, so it never reads an old copy. Postgres folds identical notifications in one transaction into one, so a bulk insert wakes a queue once. A spurious wakeup costs one small query, so I do not try to suppress them.

The trigger fires from Postgres rather than from the Go code that wrote the row. Every write path then fires it: the handler, a psql session, a migration, and code I write next year and forget about. Postgres also delivers the notification at commit, so a poller that wakes sees the row.

The listener connection

A connection that waits for a notification serves no query. So one connection, outside the pool's normal use, runs LISTEN on every queue's channel and calls back with the channel each notification arrives on:

func (db *DB) Listen(ctx context.Context, channels []string, ready func(), onNotify func(channel string)) error

A loop around it listens again after a failure, with a pause so a database that is down does not spin the loop. Postgres drops a notification that nobody listens for. So ready runs once the LISTEN stands, and each poller reads its queue then, before it waits. The poller also reads on an interval, which covers a lost notification. A connection that waits writes nothing, so the TCP keepalive that Go's dialer turns on is what ends a wait on a peer that is gone.

Fan-out

Inside a sockeye process, a subscriber is a channel with a buffer of one. Each write path calls wake after its commit:

func (s *server) subscribe() chan struct{} {
	ch := make(chan struct{}, 1)
	s.subsMu.Lock()
	s.subs[ch] = struct{}{}
	s.subsMu.Unlock()
	return ch
}

func (s *server) unsubscribe(ch chan struct{}) {
	s.subsMu.Lock()
	delete(s.subs, ch)
	s.subsMu.Unlock()
}

func (s *server) wake() {
	s.subsMu.Lock()
	defer s.subsMu.Unlock()
	for ch := range s.subs {
		select {
		case ch <- struct{}{}:
		default:
		}
	}
}

The buffer of one and the non-blocking send do two jobs. A burst of writes coalesces into one pending wakeup, so a sweep of result rows wakes a subscriber once. A subscriber that is slow to read cannot block the listener, so one stuck request cannot stall every other waiter.

The subscriber gets no data from the channel. It re-reads.

A waiter subscribes before its first read:

wake := s.subscribe()
defer s.unsubscribe(wake)

if settled := s.read(); settled {
	return
}
<-wake

In the other order there is a gap between the read and the subscribe. A write that lands in the gap wakes nobody, and the waiter sleeps until the write after it.

Across web processes

sockeye runs two web processes. A write wakes the subscribers in the process that handled it, and a reader on the other process must hear of it too. A wake_counter table carries the signal, with one row for each repo. After its commit, a write adds one to its repo's row:

INSERT INTO wake_counter (repo, n)
VALUES ($1, 1)
ON CONFLICT (repo)
DO UPDATE SET n = wake_counter.n + 1

Each process reads the table once a second. It wakes its own subscribers for each repo whose count moved since its last read. A reader on the other process can wake late, but within a minute at most:

A dashboard page asks only for its own repo, so a write to one repo does not make every open page fetch again.

Long polling

A runner that wants a job sends the state it last saw. The server answers when the state differs, or when a timeout ends. The answer is the current state either way:

func (s *server) longPoll(ctx context.Context, old State) State {
	ctx, cancel := context.WithTimeout(ctx, PollWait)
	defer cancel()
	c := make(chan State, 1)
	go func() { c <- s.waitNewState(old) }()
	select {
	case cur := <-c:
		return cur
	case <-ctx.Done():
		return old
	}
}

The state a runner waits on comes from the database, not from the process's memory. A process can be a second behind the other one, and a runner's poll can reach either.

The timeout answers with a state, not with an error. A client cannot tell a request held open forever from a server that stopped. A request that answers every 25 seconds with "no news" proves the server is up.

PollWait is one constant that both sides import. The client's HTTP timeout is PollWait plus a margin. A client that gives up first aborts every idle poll, throws the connection away, and logs a timeout for a server that answered correctly.

Replacing sleep

A script that pushes a commit and then asks how the checks went has to guess how long to wait. A guess too long wastes time nobody needed. A guess too short reads a green answer about the commit before the one it pushed. No script gets the number right, because the number is the length of somebody's test suite.

The CLI asks the server to hold the request instead. The server subscribes, checks whether the head has settled, and waits for a wakeup or the deadline before it reads again:

func (s *server) waitForHead(ctx context.Context, c *change, sha string) (*change, error) {
	wake := s.subscribe()
	defer s.unsubscribe(wake)

	deadline := time.NewTimer(PollWait)
	defer deadline.Stop()
	for {
		settled, err := s.headSettled(ctx, c, sha)
		if err != nil || settled {
			return c, err
		}
		select {
		case <-wake:
		case <-deadline.C:
			return c, nil
		case <-ctx.Done():
			return nil, ctx.Err()
		}
		if c, err = s.findChange(ctx, c.ID); err != nil {
			return nil, err
		}
	}
}

The CLI loops on that request until the change settles or its own timeout runs out. An answer before the end still includes the checks that finished, so the CLI prints one line per check as it lands.

The sha is the commit the CLI waits on. A wait that reaches the server before the push hook records the new head would read the old head's checks: green, about the wrong code. The server treats a head that is not sha yet as not settled. A sleep hid that mistake and did not fix it.

Server-Sent Events

The same subscription sits behind a Server-Sent Events endpoint. Each wakeup writes an empty event named wake, and a comment line every 15 seconds keeps a proxy from closing an idle connection:

func (s *server) wakeStream(w http.ResponseWriter, req *http.Request) {
	w.Header().Set("Content-Type", "text/event-stream")
	w.Header().Set("Cache-Control", "no-cache, no-store, no-transform")
	w.Header().Set("X-Accel-Buffering", "no")
	flush := w.(http.Flusher).Flush
	flush()

	ch := s.subscribe()
	defer s.unsubscribe(ch)

	keepalive := time.NewTicker(15 * time.Second)
	defer keepalive.Stop()
	for {
		select {
		case <-ch:
			io.WriteString(w, "event: wake\ndata:\n\n")
		case <-keepalive.C:
			io.WriteString(w, ": keepalive\n\n")
		case <-req.Context().Done():
			return
		}
		flush()
	}
}

The event says nothing about what changed. The page decides what to re-read.

Fit

This fits a system whose state is small enough for each waiter to re-read on every wakeup, and whose writes all go through one Postgres database. The subscribers live in process memory, and each process wakes its own. Postgres carries the signal between processes, as a notification or as a counter row. There is no message broker and no schema for events.

← All articles