go / json
I handle JSON in Go with encoding/json/v2, which is stable in Go 1.27,
and its token layer, encoding/json/jsontext. Its defaults are strict.
I keep them for the JSON that my own programs write. For JSON that
another program writes, I relax them in one named set of options per
peer.
import (
jsonv1 "encoding/json"
"encoding/json/jsontext"
"encoding/json/v2"
)
The v1 package stays in the import list for two of its options. In Go 1.27, v1 is a layer over v2, so a v1 option works in a v2 call.
Strict by default
These v2 defaults differ from v1:
- A member name matches its field tag with the exact case.
- A duplicate member name is an error.
- Invalid UTF-8 is an error, in a decode and in an encode.
- Data after the JSON value is an error.
- A nil slice encodes as
[], and a nil map as{}. <,>, and&encode as they are, not as\u003cescapes.- Map keys encode in no fixed order.
- A
time.Durationhas no default format, so it is an error.
A config file that I write gets these rules and one more:
if err := json.Unmarshal(b, &cfg, json.RejectUnknownMembers(true)); err != nil {
return cfg, fmt.Errorf("reading %s: %w", path, err)
}
A key with a typo names itself as an "unknown object member name". In v1, the decoder ignored the key, and the field kept its zero value.
One option set per peer
Options are values, and json.JoinOptions combines them. My sports
app reads feeds from several vendors, and one set holds the rules for
all of them:
// externalJSON decodes JSON that another program wrote.
var externalJSON = json.JoinOptions(
json.MatchCaseInsensitiveNames(true),
jsonv1.MatchCaseSensitiveDelimiter(true),
jsontext.AllowInvalidUTF8(true),
)
if err := json.Unmarshal(body, &board, externalJSON); err != nil {
return err
}
A vendor sends balls and strikes in its own case, and some of my
structs have no tags for them. Without the case rule, those fields
decode as zero and the live line loses its count. A case-insensitive
match in v2 also ignores _ and -, so image_url matches
ImageURL. MatchCaseSensitiveDelimiter keeps the delimiters exact,
as v1 did. One bad byte in a feed puts U+FFFD in its place, and the
rest of the feed decodes. A duplicate name stays an error, because no
vendor sends one.
The set has a name, and each call site passes it. A reader sees at the
call which rules apply, and a grep for the name finds every peer
decode. A struct that a person or a model writes into gets exact
lowercase tags in place of a relaxed match. A model sends tool input
as {"command":"ls"}, so the field tag says command.
Text I do not control
An encoder that carries build output, log lines, or model text also carries bytes that are not UTF-8. v2 refuses the whole value. An API answer that fails to encode ends in a broken body, and a log batch that fails to encode is lost. So those encoders replace the bad byte, as v1 did:
func Write(ctx context.Context, w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
enc := jsontext.NewEncoder(w, jsontext.AllowInvalidUTF8(true))
if err := json.MarshalEncode(enc, v); err != nil {
log.Error(ctx, err)
}
}
The decoder of browse uses the same option. A string that a page returns through the DevTools Protocol can hold a lone UTF-16 surrogate. A strict decoder drops that reply, and the call then waits until the run times out.
omitempty and omitzero
In v2, omitempty leaves out a field that encodes as an empty JSON
value: null, "", [], or {}. It does not leave out false or
0. omitzero leaves out the Go zero value. The request that sand
sends to its guest agent shows the difference:
type Request struct {
Argv []string `json:"argv"`
Dir string `json:"dir,omitempty"`
Env []string `json:"env,omitempty"`
Stdin bool `json:"stdin,omitzero"`
}
With omitempty, each request line would carry "stdin":false. With
omitzero, an agent in an older guest template reads the same line as
before. I use omitzero on each number and bool field. omitempty
still works for a string or a list, because their zero value encodes
as an empty JSON value.
Empty lists
A nil slice encodes as []. A page script that reads the list needs
no null guard. A peer that
expects null gets it through json.FormatNilSliceAsNull(true) in
its set. One model API reads "tools":null for a call with no tools,
and its request set holds that option.
Durations
A time.Duration with no format is an encode error. errs
sends a duration as integer nanoseconds, as v1 did. The rule is a
marshaler function in its option set, not a method on a type:
var wireJSON = json.JoinOptions(
jsontext.AllowInvalidUTF8(true),
json.WithMarshalers(json.MarshalToFunc(func(enc *jsontext.Encoder, d time.Duration) error {
return enc.WriteToken(jsontext.Int(int64(d)))
})),
)
A server that reads such a field passes jsonv1.FormatDurationAsNano
in its decode set.
Bytes that must not change
A JSON file under git must keep its bytes, because a change in bytes is a diff. The writer of one such file passes the options that match v1 output:
var persistOptions = json.JoinOptions(
jsontext.WithIndent(" "),
jsontext.EscapeForHTML(true),
jsontext.EscapeForJS(true),
json.Deterministic(true),
)
Deterministic sorts map keys. The escapes also matter where JSON goes
into a <script> element or an HTML attribute. A company name such as
</script> then cannot end the element.
Readers and writers
json.UnmarshalRead decodes from an io.Reader and json.MarshalWrite
encodes to an io.Writer. They replace json.NewDecoder and
json.NewEncoder for one value. UnmarshalRead rejects data after the
value, so a request body with two values gets a 400. MarshalWrite
writes no trailing newline.
A response that a vendor owns can hold more after its first value.
That decode reads one value through a jsontext.Decoder and stops:
dec := jsontext.NewDecoder(resp.Body, jsontext.AllowInvalidUTF8(true))
if err := json.UnmarshalDecode(dec, &page, json.MatchCaseInsensitiveNames(true)); err != nil {
return fmt.Errorf("invalid page data: %w", err)
}
Tests
Each relaxed rule has a test that fails without it. One test decodes a captured feed body that has untagged fields. Another sends a body with an invalid byte and checks for U+FFFD in the output. A third compares the CLI output with the v1 bytes. When I remove an option from a set, its test says which peer breaks.