simdjson

JSON for Go that locates a document's entire structure in a few vector passes and navigates that index instead of the bytes. Built on simd.go.

It provides a broad encoding/json-compatible surface — Marshal, Unmarshal, Decoder, Encoder, Valid, Compact, Indent — and direct access to the index, so a field can be read out of a document without decoding the rest of it.

No cgo. Runtime builds use committed assembly supplied by simd v1.20.0; kernel generation and updates belong to that dependency. The package builds for amd64, arm64, riscv64, s390x, ppc64le and loong64. The benchmark tables are amd64-only.

The root module requires Go 1.26.5 or later and github.com/sebishogun/simd v1.20.0.

go get github.com/sebishogun/simdjson

doc, err := simdjson.Parse(data)
if err != nil {
	return err
}

name := doc.Get("user", "name").String()
age := doc.Get("user", "age").Int()

doc.Get("items").ForEach(func(v simdjson.Value) bool {
	total += v.Key("score").Float()
	return true
})

Documentation

This page is the shipped technical front page. The deeper record:

• docs/architecture.md — module boundaries, the two-stage pipeline, ownership, document map.
• docs/lld/index-and-validation.md — Parse/Scan/Valid/GetPath contracts, the index representation, the 2 GiB limit, error semantics.
• docs/lld/value-and-ownership.md — the Value surface, borrowed vs owned results, lifetimes, concurrency.
• docs/lld/streaming-and-parallelism.md — Decoder/Encoder/ForEachLine contracts, batching, prefetch, the serial Decode state rule.
• docs/lld/marshal-and-codegen.md — Marshal family, Options, compatibility, RegisterEncoder, structgen.
• docs/roadmap.md — only still-open, source-backed opportunities; measurement-first, nothing promised.
• docs/verification.md — the exact gates, the 8.3% noise floor, and the one known make verify blocker.
• docs/plans/2026-08-13-simdjson-production-design.md and docs/plans/2026-08-13-simdjson-production.md — the design record and implementation plan for the production work.
• AGENTS.md, CLAUDE.md — the standing briefs: reading order, gates, and the rules that hold every change.
• docs/competition.md, docs/cpp-baseline.md, docs/parallelism.md, docs/lazy-paths.md — canonical analysis of the field.
• docs/wrong.md — append-only record of measured failures.
• docs/bench/ and docs/figures/ — the snapshot and the charts drawn from it; testdata/bench/README.md is the gate's methodology.

API

Parsing and navigation

Parse validates the document against JSON's grammar and returns a Doc. Scan builds the same index and identifies the root without the grammar descent. MustParse panics instead of returning an error. Parser reuses its index across documents.

From a Doc: Root, Get, Path, Unmarshal.

A Value is a cursor into the index. Get, Key, Index and Path move; Kind, Len and Exists describe; String, StringNoCopy, Int, Float, Bool, IsNull, Time and Raw read; Decode unmarshals a subtree into a Go value. Iteration is available as callbacks — ForEach, ForEachKey — or as range-over-func: All, Values, Keys, Members.

GetPath reads one dotted path straight from a byte slice. GetMany reads several paths, each supplied as a slice of object-key components; it does not accept dotted strings, array indices, wildcards or queries.

On gjson's own published fixture and rotated paths (bench/getpath_rows_test.go — the 506-byte widget document), gjson and jsonparser answer a one-shot path in about 145 ns where GetPath costs 575 ns. They scan forward and stop; this indexes first, and GetPath's documented promise that malformed input yields Invalid is what that index buys. From the second query on the same document, Parse plus Doc.Path runs each path at 139 ns, level with gjson's every-query price.

On the 1.17 MB generated document (the access-shape benchmark in bench/fair_test.go), the index wins from the first query once the field is past the front:

one field on 1.17 MB	gjson	jsonparser	this
early, at the front	0.05 µs	0.05 µs	168 µs
late, at the back	412 µs	778 µs	231 µs
every item's score	1,539 µs	1,435 µs	593 µs

encoding/json drop-in

Marshal, MarshalIndent, Unmarshal, Valid, Compact, Indent, HTMLEscape, NewDecoder, NewEncoder, RawMessage, Marshaler, Unmarshaler, and encoding.TextMarshaler / encoding.TextUnmarshaler. Decoder supports UseNumber, DisallowUnknownFields and Token. The omitempty, omitzero and ,string struct tags are honoured. Several compatibility types are aliases of the standard library's, so existing type assertions continue to work. SyntaxError is package-defined with the same SyntaxError.Offset meaning and compatible error behavior. This package must construct it, which is impossible through encoding/json.SyntaxError's unexported message field.

MarshalTo appends to a caller-supplied buffer and MarshalWrite writes to an io.Writer. Options selects encoder behaviour explicitly — Options.EscapeHTML, Options.SortMapKeys, Options.ValidateStrings, Options.OmitZeroStructFields — and has the same three methods.

Generated encoders

The default struct encoder builds and caches a field-table walker at run time, and that walk costs about 11% on a real document against straight-line code. For types you own, tools/structgen emits the straight-line code at build time:

//go:generate go run github.com/sebishogun/simdjson/tools/structgen -types User,Status

The generated file registers itself via RegisterEncoder; Marshal then uses it for that type everywhere it appears — top level, struct fields, slice elements, map values. Output is byte-identical to the reflect path, which the generator's own differential asserts. It declines any type it cannot encode exactly — maps, pointer-to-pointer fields, interfaces, []byte, embedded fields, tag options, types with their own MarshalJSON — and a declined type keeps the reflect encoder. Nothing is compiled at run time.

Generated encoders can also be written by hand against the same seam: RegisterEncoder, with AppendString, AppendInt, AppendUint, AppendFloat and AppendBool as the primitives, and the same byte-for-byte contract.

Editing, streaming and files

SetPath, SetRawPath and DeletePath return a new slice with a byte-range edit, without decoding the document or writing through the input. Skip validates and locates the first complete JSON value in a slice, returning its [start:end] offsets and excluding trailing bytes. An edit validates the document and replacement before it splices. sjson validates neither. On the 631 KB twitter corpus (the SetPath benchmark in bench/editing_rows_test.go):

one field edit	this	sjson
shallow path	2,013 MB/s	4,038 MB/s
deep path	1,805 MB/s	2,035 MB/s

sjson runs 2.0x faster on the shallow row and 1.13x on the deep one. The validation contract costs a pass, and the table prices it.

ForEachLine, ForEachLineReader and ForEachLineReaderParallel read newline-delimited JSON. Callback Value instances borrow reused batch storage and must not be retained after the callback returns.

OpenFile(path, validate) maps and indexes a file. Call Close when finished. Bytes, Doc, Value.Raw and Value.StringNoCopy results are valid only until Close; the mapped Bytes slice is read-only. Value.String returns a copy which can outlive the mapping.

Parse or Scan

Parse checks every value against the grammar and rejects what encoding/json rejects. Use it for input from outside.

Scan builds the index and identifies the root, skipping the descent that proves the parts never read are well-formed. Malformed input then yields wrong answers rather than errors — nothing reads out of bounds and nothing panics, but the result carries no guarantee. Use it for bytes you produced.

GetPath also uses the non-validating scan path for one-shot lookup. Use Parse, or Scan followed by Doc.Path, when you want an explicit reusable document.

Validation is the larger part of the cost, which is what makes the distinction worth having.

Parser reuses its index between documents. The current benchmark snapshot has no parser-reuse row, so no allocation or latency claim is made here.

Performance

The snapshot-backed tables below use docs/bench/compare-2026-08-08.json: eight samples per row, shuffled, one process per row, and the minimum of each on an idle amd64/AVX-512 machine. The snapshot records Go 1.26.2, while the root module now requires Go 1.26.5. The encoding/json columns use the v1 engine. make bench-v2 measures the experimental jsonv2 engine separately; no jsonv2 snapshot is committed, so its numbers are not summarized here. Competitors run in the same process on the same bytes; bench/ is the harness.

Parsing — a document in, a navigable and validated structure out:

	this	fastjson	minio
twitter, 0.63 MB	198 µs	239 µs	305 µs	1.21×
citm, 1.73 MB	548 µs	695 µs	655 µs	1.20×
canada, 2.25 MB	1,136 µs	1,822 µs	5,504 µs	1.60×

Scan on the same three documents is 50 / 194 / 331 µs. It is a different operation: it does not validate.

Validating, against sonic, the other library doing it with vector instructions:

	this	sonic	encoding/json
twitter	82 µs	173 µs	1,242 µs	2.10×
citm	285 µs	440 µs	3,166 µs	1.54×
canada	885 µs	986 µs	4,175 µs	1.11×

Into Go values, each corpus into its natural struct (bench/decode_rows_test.go, minimum of three):

Unmarshal → struct	this	goccy	sonic	encoding/json
twitter	311 µs	361 µs	414 µs	2,562 µs
canada	2.58 ms	6.1 ms	2.64 ms	14.5 ms
citm	1.12 ms	0.97 ms	1.47 ms	7.5 ms
2 MB []float64	2.00 ms	5.1 ms	2.22 ms	11.2 ms

The memory column, same rows (-benchmem, minimum of two):

bytes / allocations per op	this	goccy	sonic	encoding/json
twitter	177 KB / 14	701 KB / 103	753 KB / 182	194 KB / 1,410
canada	1.0 MB / 966	4.2 MB / 56,538	4.9 MB / 2,588	3.1 MB / 3,095
citm	276 KB / 4,871	2.0 MB / 12,565	2.2 MB / 15,344	373 KB / 6,430
2 MB []float64	2.1 MB / 31	4.2 MB / 107,555	3.6 MB / 62	4.1 MB / 31

Interned strings, pooled scratch and one-walk numbers add up: fourteen allocations decode twitter into structs. goccy's citm speed lead costs 7.4× the memory and 2.6× the allocations; on the any tables the same holds — ours runs every shape at 40–70% of sonic's bytes and a quarter to a half of its allocations.

canada is level with sonic — 2.5% apart, inside the noise floor — after the compiled-array, extent-float and one-pass work.

The field's own fixtures — the Small/Medium/Large payloads every Go JSON README descends from (ported verbatim from buger/jsonparser; outputs byte-agreed with encoding/json before any timing; stdlib-compatible configs only, which most published tables for these fixtures do not use):

ns/op, minimum of eight	ours	goccy	segmentio	sonic	jsoniter	stdlib
Unmarshal small (190 B)	416	201	359	424	390	1,330
Unmarshal medium (2.2 KB)	2,055	1,484	1,974	2,635	3,185	9,771
Unmarshal large (28 KB)	20,522	16,184	29,840	33,186	54,117	117,785
Marshal small	91	98	112	144	194	212
Marshal medium	168	110	126	176	229	245
Marshal large	1,263	1,305	1,575	1,391	2,732	3,438

goccy's scanner core owns this size class, and wrong.md holds the instruction-level decomposition of why (537 decode instructions per field here against its 680 for everything). Getting the small decode row from 1,042 ns and 5.2 KB of garbage per call to 416 ns and 323 B — past sonic — is what this table's first measurement bought; Marshal small is the generated-encoder row (tools/structgen), now the row's best, and Marshal large is level with goccy at 3.3%, ours in front. Everything else in the column beats every library except goccy at every size. citm is goccy's row, cut from 41% to 22% by the one-walk integer parse (segmentio's 1,388 MB/s now trails our 1,463): tiny objects of small integers, where a hand-tuned scanner pays less per token than this design's index amortizes. An index-free prototype measured 4–5× slower on every corpus, and the entry in wrong.md has the numbers. jsoniter and segmentio trail everywhere else measured — 191 and 404 MB/s on canada, 213 and 484 on the []float64 — and both are in the harness under stdlib-compatible configurations.

Out of Go values:

	this	sonic	goccy	encoding/json
Marshal, a struct	60 µs	35 µs	97 µs	112 µs
Marshal, map[string]struct, 256 entries	24 µs	23 µs	41 µs	62 µs

A decoded document — map[string]any with everything under it — encodes across cores when the output is a quarter megabyte or more: element ranges of a large []any shard to workers and the results stitch in order, byte-identical to the serial encode. Re-measured after that change, two passes of five, worse of the minima:

Marshal, decoded, sorted keys	this	sonic	goccy	encoding/json
twitter	237 µs	829 µs	1,824 µs	2,217 µs
citm	340 µs	1,401 µs	2,651 µs	3,294 µs
canada	2,284 µs	4,437 µs	6,976 µs	7,758 µs

sonic leads the two struct rows. Both are string escaping: its quote.c reserves worst-case output space and writes escapes inline in one vector pass, where this package's kernel stops at each byte needing an escape and returns to Go to emit it. Escaping costs 15.0 µs here on top of a 35.0 µs base; sonic's 27 µs covers escaping and UTF-8 validation together.

sonic's two passes differed by 19% on the struct row, where every other number here agreed within 1.6%, so that cell is a range.

Configuration: sonic.ConfigStd throughout, which sorts map keys, escapes HTML and validates strings. sonic.Marshal does none of the three; thirty calls on the same map produce five different outputs. Both are in the harness, the second marked not comparable.

Decoded into any — map[string]any and []any out of every corpus shape, MB/s, minimum of eight samples per row. This family was sonic's on all twelve shapes until the any path was cured of per-string and per-key unquote and numbers stopped paying an allocation per box (the float payloads live in a document slab, like decoded strings):

into any, MB/s	ours	sonic	goccy	stdlib
twitter	555	538	337	194
citm	792	610	375	206
canada	499	344	174	145
numbers	530	550	203	155
github_events	594	605	389	204
apache_builds	514	521	415	190
gsoc-2018	1,367	1,491	781	284
instruments	484	465	292	179
update-center	375	351	270	162
mesh	373	382	149	131
mesh.pretty	802	662	294	184
marine_ik	391	329	153	126

Bold marks a lead past the 8.3% noise floor; unmarked cells are statistically level, measured against sonic v1.15.2. The split follows the data's shape: this package leads on canada by 1.45×, citm by 1.30×, mesh.pretty by 1.21× and marine_ik by 1.19×, where its []any values carve exact-size from a document-scoped slab. sonic leads gsoc-2018 by 1.09×. The other seven shapes are level. If decoding into any is your hot path, the shape of your documents decides; measure both. goccy and stdlib trail throughout.

Text in, text out, against encoding/json, MB/s:

	twitter	citm	canada	vs stdlib
Valid	7,648	6,054	2,489	4.6–15.2×
Compact	1,588	2,154	2,164	4.2–5.5×
Indent	1,210	1,227	599	2.0–3.3×

Valid is at or ahead of sonic on all twelve corpus shapes — past the noise floor on eleven (2.1× on twitter, 2.5× on apache_builds, 1.9–2.2× across the small-document shapes) and statistically level on the twelfth. Three kernels carry it: stage one's quote parity by carry-less multiply (simd v1.11.0), the grammar walk fused into one scalar routine over the stage-one masks (simd v1.12.0), and — since simd v1.13.0 — the whole of Valid as a single fused pass, per-block masks that never leave registers feeding parity, escape validation and the grammar machine with no mask buffers written or read. That fusion is what closed gsoc-2018, 3.3 MB of escape-heavy strings and the last shape sonic held (1.43×, the measured price of the staged design): one pass now answers it 32% faster than the staged pipeline it replaces. A density probe routes number-dominated documents (canada is 94% number bytes) to the descent walk instead, which pays nothing per block between one number and the next; docs/wrong.md holds that measurement, alongside every rejected step on the way here.

Under concurrency — aggregate throughput, every goroutine decoding its own twitter into its own struct (the many-requests server shape in bench/parallel_curve_test.go):

MB/s aggregate, full machine	ours	goccy	sonic	encoding/json
full-machine parallel run	27,824	21,783	13,795	2,453

This is 1.28× goccy, 2.02× sonic and 11.3× stdlib. The per-thread curve is not in the committed record, so no claim about its shape is made here. This is the many documents axis; the one document axis — a single payload sharded across cores past 8 MB — is the at-scale family above.

Cold start — the first operation on a never-seen type, measured by building a fresh type per iteration (bench/coldstart_test.go), which is what a deploy's first request meets and what the Pretouch warm-up in published tables hides:

first contact, ns	ours	encoding/json	goccy	sonic
Unmarshal, 5-field struct	3,205	3,546	4,629	857,495

sonic's 268× is its JIT compiling the fresh type — the cost Pretouch warm-ups hide; ours is a table build, and structgen'd types pay nothing at all.

Streaming, 50,000 newline-delimited records, 6.5 MB:

	this	goccy	sonic	encoding/json
Decoder	9.7 ms	11.9 ms	13.7 ms	37.8 ms
Encoder	5.9 ms	6.9 ms	9.1 ms	9.6 ms

Allocation for the same input is 9.5 MB in 150,183 allocations, against goccy's 12.9 MB in 306,525.

Record size is the axis that decides it. Streams of two-kilobyte records -- real tweets, newline-delimited, decoded into any -- run 515 MB/s here against sonic's 505; at fifty-kilobyte records the per-value work is almost entirely the any-decode itself and sonic's assembled walker takes it, 505 against 445. The crossover is the same residual the any-decode table above prices, reached through a different door.

Small documents are the size where an index does not pay. It costs the same few passes whether the document is 64 bytes or a megabyte:

	this (Parse)	fastjson	encoding/json.Valid
64 B	145 ns	39 ns	73 ns
200 B	236 ns	101 ns	214 ns
2 KB	859 ns	1,062 ns	3,191 ns
20 KB	7,963 ns	10,660 ns	22,646 ns

The crossover is between 200 bytes and 2 KB. Below it, encoding/json is the better choice. The third column is validation rather than decoding, matching the small-input benchmark in bench/small_test.go.

Pulling one field out of a document. 10,000 items, 1.17 MB, one field read. Everything here validates the whole document:

	10,000 items
this — Parse+Get	0.805 ms
valyala/fastjson	1.345 ms	1.67×
minio/simdjson-go	2.025 ms	2.52×
bytedance/sonic	5.471 ms	6.80×
goccy/go-json	9.202 ms	11.4×

These are the validating equivalent-work row and matching field-get rows in the 2026-08-08 snapshot. No stdlib row for this comparison is committed, so none is printed. The non-validating Scan+Get row on the same document takes 0.169 ms. fastjson builds a value tree into a reusable arena rather than an index, so navigation afterwards is a pointer walk where this is a lookup into a position array.

Against lazy scanners. gjson and jsonparser scan for a path and stop at the first match rather than parsing the document. gjson.Get is not the same operation as Parse: it does not validate, and answers from input that is not JSON.

input	gjson.Get returns	valid JSON
{"a" 1} — no colon	"1"	no
{"a":1 — unterminated	"1"	no
{"a":01} — invalid number	"01"	no

With validation on both sides, on a 10,000-item document:

	gjson	this
both validating — gjson.Valid+Get against Parse+Get	741 µs	805 µs	gjson 1.09× — at the noise floor
neither validating — gjson.Get against Scan+Get	0.05 µs	170 µs	gjson ~3,200×

For one field, stop-at-first-match wins by construction when nothing is validated, and validation brings gjson's lead down to the noise-floor bound. Two comparisons where the operations do match:

reading the whole document once	time	result
gjson.Valid+Get	741 µs	a bool and the field
Scan+Get	170 µs	a reusable index and the field — 4.3×
Parse+Get	805 µs	that index, the field, and the grammar proved

gjson retains nothing, so each Get rescans from byte zero, while this indexes once. Reading items.N.score for N across a 10,000-item document:

queries	gjson	this
1	0.09 µs	168 µs	gjson ~2,000×
10	1,934 µs	448 µs	4.3×
100	20,359 µs	3,072 µs	6.6×
1,000	205 ms	30.0 ms	6.8×

The crossover is between one and ten queries. Both are quadratic — gjson rescans to reach element N, Index(j) walks j elements — but each step here is a lookup rather than a byte scan.

gjson offers a path language this does not: wildcards, #(age>45) filters, about fifteen modifiers, JSON Lines and custom modifiers. This has Get, Key, Index, Path and ForEach. Its own documentation states that the Get* functions "expect that the json is well-formed" and that bad JSON "may return back unexpected results", which is what the validating row above measures.

Choosing. gjson or jsonparser for one field from a document you produced and will not query again. This package when the document comes from outside and must be checked, when the same document will be queried more than a handful of times, or when the target is not amd64.

At a gigabyte and beyond

Real documents repeated to size, with a deliberate best and worst case:

1 GB, one piece	Scan	Valid	Parse	encoding/json.Valid
best case: minified ASCII, long values	7,302 MB/s	5,859	5,143	583
worst case: nothing but brackets and escapes	1,166	1,497	707	471

Six times between the two for Scan. A single throughput number for a JSON parser is an average over shapes that differ by that much.

Those rows are single-threaded. From 8 MB up, Scan and Parse build the structural index across cores — segments are indexed in parallel and the bracket pairs that cross a segment are merged serially, with output bit-identical to the single-threaded path, errors included. 64 MB of numbers-heavy JSON scans at 31.1 GB/s on 32 cores against 5.1 single-threaded; BenchmarkParallelScan reproduces it. When the root is an array of containers — the shape huge documents have — the bracket index gives every element's exact extent, and the grammar walks themselves shard across workers, ranges balanced by bytes rather than element count so twenty-eight two-megabyte documents split as well as three million records. At 64 MB: Valid 2.2 → 12.2 GB/s, Parse 1.8 → 11.5 GB/s, and Unmarshal into a struct slice 1.95 → 15.3 GB/s — each held identical to its serial path by a differential, errors included; other shapes walk serially over the same index. Compact and Indent join them — validation through the same parallel walk, and each transform sharded two-phase off the masks with its two-value writer state (depth and the pending-newline flag) carried across segment folds — at 0.5 → 2.5 GB/s and 0.27 → 1.23 GB/s respectively on 60 MB documents.

Past 2 GiB, Parse and Scan return an error naming the alternative — see Limits. That alternative is Decoder, which has no size limit. Ten gigabytes of tweets, 2.93 M records of about 3.4 KB each, from cd bench && go test -run TestHuge -huge -huge-bytes 10000000000 . — better of two passes, worse of the two heap peaks:

10 GB	decoded into	time	throughput	peak heap
line-delimited	Value	3.46 s	2,892 MB/s	8.0 MB
line-delimited	struct, 4 fields	5.03 s	1,988	9.4 MB
line-delimited	map[string]any	30.03 s	333	9.3 MB
one array	Value	3.66 s	2,731	8.5 MB
one array	struct, 4 fields	5.04 s	1,985	8.9 MB
one array	map[string]any	29.82 s	335	9.2 MB
one array, 300 M small elements	struct, 3 fields	22.65 s	441	9.3 MB

Under ten megabytes of heap for ten gigabytes of input, because nothing is held whole.

The decode target sets the throughput more than the parser does. Value builds no Go value and is the parser's own rate; map[string]any allocates a map and an interface per field and is 8× slower on identical bytes. Each row names its target for that reason.

Whole-document Unmarshal of a root array eight megabytes and up decodes across cores: element extents come from the parallel index, workers decode straight into the result slice, and any anomaly falls back to the serial decode, which owns the error. 32 MB of tweets into structs runs at 15.3 GB/s against 1.95 single-threaded — 7.9×, minimum of three — with values and errors identical to the serial path by differential.

A single enormous array works as well as line-delimited records. Read the opening bracket with Token, then More and Decode, as with the standard library:

dec := simdjson.NewDecoder(r)
if _, err := dec.Token(); err != nil { // the opening [
	return err
}
for dec.More() {
	var rec Record
	if err := dec.Decode(&rec); err != nil {
		return err
	}
	process(rec)
}

An object works the same way, with Token for each key and Decode for the value after it.

Nine shapes

twitter, citm and canada cover strings-and-objects, objects-and-whitespace and numbers. shapes_test.go adds deep nesting, wide objects, long strings, escape-heavy strings, non-ASCII, bare numbers, bare literals, pretty-printed and empty containers — each about a megabyte, each checked against encoding/json through every entry point before it is timed. Scan holds 12.3–12.5 GB/s on all of them except the two that are nothing but brackets.

The shape of it

Every row in the tables above, drawn. Ratio is time ÷ this library's time on the same bytes; the dashed line is 1.0, so bars below it are rows another library wins — and they are all here, because a chart that only shows the winning side is a sales pitch. The throughput charts carry the same rows in MB/s. All figures are reproducible from the committed snapshot. The snapshot they are drawn from (docs/bench/) names the machine, the instruction-set tier, the Go version and the date, and make bench-all re-measures and re-renders.

Raw throughput, for the record:

How these were measured. Every benchmark runs in its own fresh process — no benchmark's warm cache, branch history or allocator state carries into the next — and the order is shuffled per run. Each number is the minimum of eight samples, the estimator this repository's gate uses (layout noise is one-sided; the minimum converges to the true code speed). The machine is quiet, the tier is the one named in the snapshot (simd.Tier()), and the rivals run in the same process family on the same bytes. Slow rows — a benchmark whose single iteration exceeds the discovery threshold — are skipped and listed in the snapshot rather than run for hours; -include-slow restores them. The comparison record is make bench-all and its snapshots live in docs/bench/. The separate root-module regression-gate baseline is in testdata/bench/; make bench-check exercises that gate rather than the multi-library suite.

Limits

Document size. Parse and Scan index a document in one piece and cap at 2 GiB, because a bracket position is an int32 and the index is already 0.93× the size of the document; int64 positions would take it past 1.4× and charge every ordinary parse for a size that has a better answer. Above the cap they return an error naming Decoder, which has no package-defined total-input limit and processes a stream incrementally. Individual values still require working memory, so practical limits depend on value size and available resources.

Whole-document decoding. An index pays for reaching into a document, and whole-document decoding has a different cost profile. See the measured struct, any, and large-array tables above rather than assuming one implementation wins every shape.

String copies, always. Value.String returns a new Go string in every case: a clean string costs one []byte→string copy, an escaped string is unescaped into fresh storage. The zero-copy path is StringNoCopy, and its lifetime rule is below.

StringNoCopy aliases input. Keep the original byte slice alive and unmodified while using the returned string. Use String when the value escapes the input's lifetime.

Small inputs. Below roughly a kilobyte the fixed cost of the index is the whole cost. See the table above.

Correctness

Correctness is defined as agreeing with encoding/json and tested that way: hand-written cases, 2,000 randomised documents built from atoms chosen to collide (structure inside strings, escaped quotes, escaped backslashes, surrogate pairs), and differential fuzzing.

Eight fuzz targets compare against the standard library — parse, unmarshal, marshal, text operations, Decoder, Token, streamed array elements and UTF-8 validation — and demand the same bytes and the same error-or-not, not merely the same meaning.

go test ./...
go test -run '^$' -fuzz FuzzAgainstStdlib -fuzztime 60s
make verify        # fmt, vet, tests, race, every instruction tier, purego
make fuzz          # every differential target
make test-cross    # manual Docker/QEMU lane: arm64, s390x, ppc64le

The ordinary verification suite runs the host against scalar, sse2, avx2 and avx512 dispatch selections and against -tags purego. make test-cross is a separate manual lane, not part of make verify or CI.

Findings that shaped the implementation, including measurements that argued against changes that were then reverted, are recorded in docs/wrong.md.

How it works

Two stages, the design C++ simdjson introduced.

Stage one classifies the document with vector compares and answers everything that follows with bit arithmetic. Three passes produce a bitmask each — one bit per input byte — for the quotes, the backslashes and the six structural characters. A conventional parser reads a byte and branches on what it is, which is a dependent, unpredictable branch per byte. This has no per-byte branch at all.

Stage two walks the surviving positions. A megabyte document might hold fifty thousand structural characters, so the second stage sees fifty thousand items rather than a million bytes.

The difficulty is in stage one. A { inside a string is text, and a " preceded by an odd number of backslashes closes nothing — in "a\\" the quote follows two backslashes and does close the string, while in "a\" it follows one and does not. Both are resolved before any position is interpreted, as arithmetic over sixty-four bytes at a time:

• which quotes are escaped — adding the odd-length backslash-run starts back into the backslash mask propagates a carry through each run and lands it one past the run's end, turning "the parity of this run" into a single add;
• which bytes are inside a string — an inclusive prefix XOR of the surviving quote mask, six shift-and-xor steps per word, with the parity carried into the next word by sign-extending its top bit;
• which structural characters survive — an and-not.

None of it costs anything per match.

Streaming indexes per buffer rather than per value, in partial mode, which treats a value cut in half by the end of the buffer as a fact to report rather than an error: it indexes what is there and records how far that reaches. Array elements are batched by bytes rather than by count — an element of a megabyte fills a batch alone, a hundred-byte record shares one with six hundred others — and the batch boundary is read off the index rather than found by a separate scan. A sustained Value loop over records goes further: batches are capped so the buffer holds the next one, a background task prepares it — index, scan, and validation fanned across cores — while the current one drains, and delivery hands each record out from its staged extent alone, with no whitespace skip, bracket match or validate left on the mainline. 1,450 → 1,974 MB/s on 64 MB of newline-delimited records. Decode keeps its serial per-value walk deliberately: decoding element k starts from the caller's variable as element k−1 left it, so the results chain by contract.

When to use it, and when not to

Every claim below cites a table in this README or a file in this repository; none of it is asserted from goodwill.

Use this library when:

• Documents are a megabyte or more. Fastest parse and validate among the libraries and configurations measured on the listed corpora; past 8 MB the parallel family has no counterpart among the libraries examined here (docs/cpp-baseline.md).
• You serve many requests. Fastest aggregate decode of the four measured libraries: 27.8 GB/s on the snapshot machine against goccy's 21.8 and sonic's 13.8 (concurrency table).
• Streams: NDJSON, logs, exports. The streaming tables, the 10 GB rows under ten megabytes of heap, and a Value loop that pipelines its batches across cores.
• Numbers dominate. canada-class struct decode level with sonic, plain []float64 ahead of it, the numbers corpus ahead — the one-walk integer/float parsers and slab boxing did this.
• You query one document more than once. The second dotted-path query on the 506-byte fixture costs 139 ns against gjson's 145 ns for every query; on the 1.17 MB access rows the index wins from the first query once the field is past the front (GetPath notes above).
• You decode into any. Five leads and seven level rows across twelve shapes (the any table).
• Deploy posture matters. No cgo, no JIT, no runtime executable memory, builds on six architectures, the fastest cold start among the measured set (first-contact row — sonic pays 268× compiling), and the conformance suites (JSONTestSuite, jsonchecker, UTF-8 stress) run in the ordinary test pass with zero disagreements against encoding/json.
• You own your types and want the last drop. tools/structgen emits compile-time encoders: level with goccy on the field's small Marshal fixture, byte-identical output enforced.

Prefer something else when:

• Tiny one-shot parses dominate. Sub-2 KB single documents: goccy's scanner core leads decode (201 ns vs our 416 on the field's small fixture), and the standard library is often sufficient.
• Small-struct Marshal is the hot path. sonic's fused JIT writer holds ~2× (the decomposition in wrong.md says exactly why, and what it would cost to chase) — if its posture fits your deploy.
• Dense tiny-object decode. citm-class shapes: goccy leads by ~1.2× (same scanner-core wall, measured three ways).
• One-shot field-gets on small documents. gjson answers in 145 ns where we pay 575 for the index and the validity promise; the trade flips on size or repetition (GetPath notes).
• Indented output is the product. MarshalIndent trails goccy's fused indent encoder by 1.11×.

Known costs, stated: the index is real memory (roughly document-sized; Parser reuse amortizes it); performance numbers are measured on amd64, while the other five architectures are build targets rather than latency claims; and published comparisons elsewhere often use lossy configurations (unsorted keys, skipped validation, warmed JITs) that this harness deliberately refuses, so our numbers for competitors run lower than their READMEs and are the defensible ones.

Status

The latest local tag is v0.6.0, while the latest published GitHub release is v0.2.0. The API remains pre-1.0. The root module requires Go 1.26.5 and simd v1.20.0; the separate comparison module in bench/ still declares Go 1.25 and is not covered by the root CI lane.

The broad encoding/json-compatible surface passes the vendored standard library decode, encode, stream and tag tests, and its entry points are covered by differential fuzz targets. Package-specific APIs such as Scan, indexed Value cursors, mapping, edits and extended options retain their documented semantics. Wall-clock numbers are measured on amd64 only.

The rest of the family

The maintained inventory of libraries built on simd is in the simd README. Platform, performance and release status stay with each repository instead of being copied here.

License

MIT — see LICENSE. Depends on simd.go (MIT).