Skip to content

docs: qualify the operator>> stream positioning guarantee - #5343

Open
nlohmann wants to merge 1 commit into
developfrom
claude/issue-5340-short-term-e64cc3
Open

docs: qualify the operator>> stream positioning guarantee#5343
nlohmann wants to merge 1 commit into
developfrom
claude/issue-5340-short-term-e64cc3

Conversation

@nlohmann

@nlohmann nlohmann commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Short-term part of #5340.

What

operator>>'s notes claim it "leaves the stream positioned right after" the parsed value, so concatenated JSON values can be read back to back. That does not hold when the value is a number.

A number is only terminated by the character that follows it. lexer::scan_number() reads that character and calls lexer::unget(), but unget() is deliberately simulated — it rewinds only the lexer's own bookkeeping (chars_read_total, chars_read_current_line, token_string), not the input. input_stream_adapter::get_character() consumes via sbumpc() with no matching sungetc(), so the terminating character stays consumed from the stream.

This PR documents the actual behaviour:

  • docs/mkdocs/docs/api/operator_gtgt.md — drops the unqualified "positioned right after it" wording and adds a warning admonition showing the failing case (1true), the whitespace-separated fix (1 true), and which inputs are unaffected.
  • docs/mkdocs/docs/features/parsing/json_lines.md — qualifies the cross-reference, which repeated the unqualified claim.

Verification

Every claim in the new admonition was compiled and run against this branch's include/ (i.e. unmodified parser):

input first value second extraction
1true 1 throws parse_error.101 (last read: 'r')
1 true 1 true
truefalse true false
[1][2] [1] [2]
{"a":1}{"b":2} {"a":1} {"b":2}
"a""b" "a" "b"

Note the issue's original table lists 1 true as working. It does produce the right values, but the stream position is still off by one — the byte that gets eaten just happens to be the separator. The wording here reflects that.

Public API

No breaking changes. Documentation only — no header, no source, and no behaviour is touched. No amalgamation needed.

Follow-up

The actual fix (giving input_stream_adapter a real unget and restoring a still-pending character at the end of a non-strict parse) is in #PLACEHOLDER, opened as a draft: it changes observable behaviour of operator>> and sax_parse(strict=false), so it is a candidate for the next major release rather than a patch release. When that lands, the admonition added here should be replaced with a version note.

#5326 adds a "Strictness and trailing data" section to features/parsing/index.md that restates the positioning guarantee. It is still open, so this PR does not touch that file — whichever merges second should pick up the qualification.


  • The changes are described in detail, both the what and why.
  • An existing issue is referenced.
  • Code coverage — N/A (documentation only; no code changed).
  • The documentation is updated.
  • make amalgamate — N/A (no source code change).

This pull request was written by Claude Code.

operator>>'s notes state that it leaves the stream positioned right
after the parsed value, so that concatenated JSON values can be read
back to back. That does not hold when the value is a number: a number
is only terminated by the character that follows it, and the lexer's
unget() is simulated (it rewinds only the lexer's own bookkeeping),
so that character stays consumed from the stream.

Document the actual behaviour: the guarantee holds for all value types
except numbers, which must be followed by whitespace. Also qualify the
cross-reference on the JSON Lines page, which repeated the unqualified
claim.

Documentation only; the behaviour itself is tracked in #5340.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant