go / test
Assertions come from is. Database tests insert fixtures through a test helper. CI reuses the Go caches across runs, with two flags that keep the result cache warm.
Assertions
func TestRename(t *testing.T) {
is := is.New(t)
co, err := Rename(id, "Acme")
is.NoErr(err)
is.Eq(co.Name, "Acme")
}
A failure reads:
rename_test.go:9: co.Name = Beta, want Acme
The variable shadows the package so a call site reads as English.
Eq,NotEqcompare two values.NoErr,HasErrcheck errors.Nil,NotNilcheck nil.Truechecks a predicate with no want to name, such asis.True(strings.Contains(s, x)).
New(t) fails fatally on the first failure. NewRelaxed(t) records
and continues, for a test that reports every mismatch in a loop.
Eq(x, nil) fails for a nil *T, because a typed nil boxed into
any is a non-nil interface. Nil handles it.
True(got == want) compiles, prints false, want true, and loses
both values.
Eq takes any
A method cannot have type parameters, so Eq takes any and gives up
compile-time type checking:
is.Eq(gotInt64, 3) // builds, then fails: int64 3 against int 3
When the two values format the same, the failure prints %T too. The
fix is a typed literal, int64(3).
Eq uses reflect.Value.Equal where both values are comparable, so
pointers compare by identity. It uses reflect.DeepEqual elsewhere,
so slices and maps compare by contents.
Label from source
co.Name is the source text of the assertion's first argument, read
back from the test file. There is no message to write.
The values always print. A failed file read degrades to
got X, want Y. A literal argument gets no label, because
[]int{1, 2} = []int{1, 2} is noise.
Under -trimpath, which the cache needs, runtime.Caller
returns a module-relative path that does not open. filepath.Base of
it does open, because go test runs each binary in its package
directory.
The package parses the whole file and takes the call node that covers the line. So a call spread over several lines keeps its label. The node must be a call to the helper. This check rejects a file with the same name from another package.
gotest.tools/v3/assert also reads the source, but opens the path
runtime.Caller returns, so under go test -trimpath its parse
fails. matryer/is prints the trailing comment, which is still a
hand-written message.
Fixtures
I use is beside database test fixtures:
func TestRename(t *testing.T) {
t.Parallel()
is := is.New(t)
db := test.NewDB(t)
co := db.InsertCompany()
err := Rename(t.Context(), db.DB, co.ID, "Acme")
is.NoErr(err)
is.Eq(db.FindCompany(co.ID).Name, "Acme")
}
db wraps the database connection and holds fixture helpers such as
db.InsertCompany(). is holds the assertions.
I keep the two helpers separate. db.Eq reads like a database query
instead of a test assertion.
The test runs the function and checks the result with is. Each
fixture method inserts a row into a test transaction:
type CompanyOpts struct {
Name string
Status string
}
type CompanyRow struct {
ID int64 `db:"id"`
Name string `db:"name"`
}
func (db *DB) InsertCompany(opts ...CompanyOpts) CompanyRow {
db.t.Helper()
var o CompanyOpts
if len(opts) > 0 {
o = opts[0]
}
if o.Name == "" {
o.Name = "Test Co"
}
if o.Status == "" {
o.Status = "active"
}
var row CompanyRow
err := db.QueryRow(db.ctx, `
INSERT INTO companies (name, status)
VALUES ($1, $2)
RETURNING id, name
`, o.Name, o.Status).Scan(&row.ID, &row.Name)
if err != nil {
db.t.Fatalf("insert company: %v", err)
}
return row
}
The options struct sets defaults that satisfy database constraints. A test overrides only the fields it tests.
The method returns the inserted row with its generated ID. If the
insert fails, t.Fatalf stops the test immediately.
The test database runs on a separate cluster. See postgres / dev test clusters.
Shared rows
Each test runs in its own transaction, and the transaction rolls back at the end. So the tests run in parallel against one database. That isolates what a test writes. It does not isolate a row that every test can see, such as a seeded user.
An insert that references a row through a foreign key takes a key-share lock on that row until its transaction ends. A seeded row that many fixtures point at can hold locks from many parallel transactions at once.
One of my tests ran UPDATE users SET team = NULL to hide the seeded
users from the query under test. The update wrote the one seeded user
while about twenty parallel transactions held key-share locks on it.
Postgres failed it with an internal error:
new multixact has more than one updating member. The seeded user was
already off the team, so the update changed nothing the test read, and
I deleted it.
A test does not write a row it did not insert. To keep seeded rows out of a result, the test filters on its own fixtures.
Committed rows
One test must commit. It tests a trigger that sends NOTIFY, and
Postgres delivers a notification only when the transaction commits.
Its rows stand for about a second, and the package test binaries run
at the same time. A test in another package that read the jobs table
with no filter of its own read them. It scanned a NULL into an
*int64, or counted 1 where it wanted 0. Two tests failed on pushes
that touched neither, and passed on a rerun.
A failure that nobody trusts is a failure that nobody reads. So each
read of jobs in a test now names its own rows: by queue, by job
name, or by an id that its own fixture made. A read matches an id in
args by JSONB containment, not by a cast:
SELECT count(*) FROM jobs WHERE args @> jsonb_build_object('company_id', $1::bigint);
A cast such as (args->>'company_id')::bigint reads every row in the
table. A row from another test with a non-numeric id then raises an
error.
A test in the jobs package parses every other test file and fails on
a read of jobs whose WHERE clause names none of the three. An
allow list takes an exception, with one reason for each directory.
Tables
I write a table test when cases differ only in their inputs and wants. The table holds only the differences. A case with its own setup or its own output stays a separate test.
Every row calls t.Parallel():
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
is := is.New(t)
db := test.NewDB(t)
// ...
})
}
A row is a new test. Folding parallel tests into rows that are not parallel trades away concurrency that the suite needs as it grows.
Races
A test of http.TimeoutHandler used a 1 ms timeout. Its handler
returned when the request context was cancelled. That is the same
moment the timer fires. TimeoutHandler selects on the handler being
done and on the context being done. So both cases were ready, and
select picked one at random. On a loaded CI runner, the test read
200, not 503.
Now the handler blocks until the test ends, so the timer is the only outcome:
done := make(chan struct{})
t.Cleanup(func() { close(done) })
h := http.TimeoutHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
<-done
}), time.Millisecond, "timeout")
Cache
To reuse the go test cache across CI runs, I persist the cache
directories and make each test binary depend only on its source.
Go keeps two on-disk caches:
GOCACHE: compiled packages and passing test results.GOMODCACHE: downloaded module source.
On a CI VM I put both on the local disk and reuse them across runs:
export GOCACHE=/var/cache/ci/go-cache
export GOMODCACHE=/var/cache/ci/go-mod
The result cache keys on the inputs to each test binary. Two defaults change those inputs on every CI run:
go teststamps the binary with VCS info, so every commit produces a different binary.- The binary embeds the absolute source path, which differs per box.
Two flags remove both:
go test -buildvcs=false -trimpath ./...
Identical source then produces an identical binary, and go test
reuses cached results across commits and across boxes that share a
GOCACHE.
The first run populates the cache. The next run on the same source
reports (cached) for unchanged packages and finishes in seconds.
cmd / sockeye runs its CI cache this way.
A cached result includes what the test binary printed, and go test
replays it on a hit. So a check that reads the run's output costs
nothing when nothing changed. My template coverage check works this
way. Each render prints a line to stdout. A command reads the lines
back out of the -v output. See
go / html templates.
The cache also keys on the environment variables the binary read. So a run with the trace on and a run with it off are separate entries.