go / json

I handle JSON in Go with encoding/json/v2, which is stable in Go 1.27, and its token layer, encoding/json/jsontext. Its defaults are strict. I keep them for the JSON that my own programs write. For JSON that another program writes, I relax them in one named set of options per peer.

import (
	jsonv1 "encoding/json"
	"encoding/json/jsontext"
	"encoding/json/v2"
)

The v1 package stays in the import list for two of its options. In Go 1.27, v1 is a layer over v2, so a v1 option works in a v2 call.

Strict by default

These v2 defaults differ from v1:

A config file that I write gets these rules and one more:

if err := json.Unmarshal(b, &cfg, json.RejectUnknownMembers(true)); err != nil {
	return cfg, fmt.Errorf("reading %s: %w", path, err)
}

A key with a typo names itself as an "unknown object member name". In v1, the decoder ignored the key, and the field kept its zero value.

One option set per peer

Options are values, and json.JoinOptions combines them. My sports app reads feeds from several vendors, and one set holds the rules for all of them:

// externalJSON decodes JSON that another program wrote.
var externalJSON = json.JoinOptions(
	json.MatchCaseInsensitiveNames(true),
	jsonv1.MatchCaseSensitiveDelimiter(true),
	jsontext.AllowInvalidUTF8(true),
)
if err := json.Unmarshal(body, &board, externalJSON); err != nil {
	return err
}

A vendor sends balls and strikes in its own case, and some of my structs have no tags for them. Without the case rule, those fields decode as zero and the live line loses its count. A case-insensitive match in v2 also ignores _ and -, so image_url matches ImageURL. MatchCaseSensitiveDelimiter keeps the delimiters exact, as v1 did. One bad byte in a feed puts U+FFFD in its place, and the rest of the feed decodes. A duplicate name stays an error, because no vendor sends one.

The set has a name, and each call site passes it. A reader sees at the call which rules apply, and a grep for the name finds every peer decode. A struct that a person or a model writes into gets exact lowercase tags in place of a relaxed match. A model sends tool input as {"command":"ls"}, so the field tag says command.

Text I do not control

An encoder that carries build output, log lines, or model text also carries bytes that are not UTF-8. v2 refuses the whole value. An API answer that fails to encode ends in a broken body, and a log batch that fails to encode is lost. So those encoders replace the bad byte, as v1 did:

func Write(ctx context.Context, w http.ResponseWriter, status int, v any) {
	w.Header().Set("Content-Type", "application/json; charset=utf-8")
	w.WriteHeader(status)

	enc := jsontext.NewEncoder(w, jsontext.AllowInvalidUTF8(true))
	if err := json.MarshalEncode(enc, v); err != nil {
		log.Error(ctx, err)
	}
}

The decoder of browse uses the same option. A string that a page returns through the DevTools Protocol can hold a lone UTF-16 surrogate. A strict decoder drops that reply, and the call then waits until the run times out.

omitempty and omitzero

In v2, omitempty leaves out a field that encodes as an empty JSON value: null, "", [], or {}. It does not leave out false or 0. omitzero leaves out the Go zero value. The request that sand sends to its guest agent shows the difference:

type Request struct {
	Argv  []string `json:"argv"`
	Dir   string   `json:"dir,omitempty"`
	Env   []string `json:"env,omitempty"`
	Stdin bool     `json:"stdin,omitzero"`
}

With omitempty, each request line would carry "stdin":false. With omitzero, an agent in an older guest template reads the same line as before. I use omitzero on each number and bool field. omitempty still works for a string or a list, because their zero value encodes as an empty JSON value.

Empty lists

A nil slice encodes as []. A page script that reads the list needs no null guard. A peer that expects null gets it through json.FormatNilSliceAsNull(true) in its set. One model API reads "tools":null for a call with no tools, and its request set holds that option.

Durations

A time.Duration with no format is an encode error. errs sends a duration as integer nanoseconds, as v1 did. The rule is a marshaler function in its option set, not a method on a type:

var wireJSON = json.JoinOptions(
	jsontext.AllowInvalidUTF8(true),
	json.WithMarshalers(json.MarshalToFunc(func(enc *jsontext.Encoder, d time.Duration) error {
		return enc.WriteToken(jsontext.Int(int64(d)))
	})),
)

A server that reads such a field passes jsonv1.FormatDurationAsNano in its decode set.

Bytes that must not change

A JSON file under git must keep its bytes, because a change in bytes is a diff. The writer of one such file passes the options that match v1 output:

var persistOptions = json.JoinOptions(
	jsontext.WithIndent("  "),
	jsontext.EscapeForHTML(true),
	jsontext.EscapeForJS(true),
	json.Deterministic(true),
)

Deterministic sorts map keys. The escapes also matter where JSON goes into a <script> element or an HTML attribute. A company name such as </script> then cannot end the element.

Readers and writers

json.UnmarshalRead decodes from an io.Reader and json.MarshalWrite encodes to an io.Writer. They replace json.NewDecoder and json.NewEncoder for one value. UnmarshalRead rejects data after the value, so a request body with two values gets a 400. MarshalWrite writes no trailing newline.

A response that a vendor owns can hold more after its first value. That decode reads one value through a jsontext.Decoder and stops:

dec := jsontext.NewDecoder(resp.Body, jsontext.AllowInvalidUTF8(true))
if err := json.UnmarshalDecode(dec, &page, json.MatchCaseInsensitiveNames(true)); err != nil {
	return fmt.Errorf("invalid page data: %w", err)
}

Tests

Each relaxed rule has a test that fails without it. One test decodes a captured feed body that has untagged fields. Another sends a body with an invalid byte and checks for U+FFFD in the output. A third compares the CLI output with the v1 bytes. When I remove an option from a set, its test says which peer breaks.

← All articles