docs: qualify the operator>> stream positioning guarantee - #5343
Open
nlohmann wants to merge 1 commit into
Open
Conversation
operator>>'s notes state that it leaves the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That does not hold when the value is a number: a number is only terminated by the character that follows it, and the lexer's unget() is simulated (it rewinds only the lexer's own bookkeeping), so that character stays consumed from the stream. Document the actual behaviour: the guarantee holds for all value types except numbers, which must be followed by whitespace. Also qualify the cross-reference on the JSON Lines page, which repeated the unqualified claim. Documentation only; the behaviour itself is tracked in #5340. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short-term part of #5340.
What
operator>>'s notes claim it "leaves the stream positioned right after" the parsed value, so concatenated JSON values can be read back to back. That does not hold when the value is a number.A number is only terminated by the character that follows it.
lexer::scan_number()reads that character and callslexer::unget(), butunget()is deliberately simulated — it rewinds only the lexer's own bookkeeping (chars_read_total,chars_read_current_line,token_string), not the input.input_stream_adapter::get_character()consumes viasbumpc()with no matchingsungetc(), so the terminating character stays consumed from the stream.This PR documents the actual behaviour:
docs/mkdocs/docs/api/operator_gtgt.md— drops the unqualified "positioned right after it" wording and adds a warning admonition showing the failing case (1true), the whitespace-separated fix (1 true), and which inputs are unaffected.docs/mkdocs/docs/features/parsing/json_lines.md— qualifies the cross-reference, which repeated the unqualified claim.Verification
Every claim in the new admonition was compiled and run against this branch's
include/(i.e. unmodified parser):1true1parse_error.101(last read: 'r')1 true1truetruefalsetruefalse[1][2][1][2]{"a":1}{"b":2}{"a":1}{"b":2}"a""b""a""b"Note the issue's original table lists
1 trueas working. It does produce the right values, but the stream position is still off by one — the byte that gets eaten just happens to be the separator. The wording here reflects that.Public API
No breaking changes. Documentation only — no header, no source, and no behaviour is touched. No amalgamation needed.
Follow-up
The actual fix (giving
input_stream_adaptera real unget and restoring a still-pending character at the end of a non-strict parse) is in #PLACEHOLDER, opened as a draft: it changes observable behaviour ofoperator>>andsax_parse(strict=false), so it is a candidate for the next major release rather than a patch release. When that lands, the admonition added here should be replaced with a version note.#5326 adds a "Strictness and trailing data" section to
features/parsing/index.mdthat restates the positioning guarantee. It is still open, so this PR does not touch that file — whichever merges second should pick up the qualification.make amalgamate— N/A (no source code change).This pull request was written by Claude Code.