From 172b519016097ea16031198123eef987c2058b22 Mon Sep 17 00:00:00 2001 From: Kyue Date: Wed, 22 Jul 2026 17:45:10 +0100 Subject: [PATCH 01/14] Fix stack overflow in binary format parsers (CBOR/MessagePack/UBJSON/BSON) The binary format parsers in binary_reader.hpp parse nested arrays/objects using unbounded recursive descent, unlike the text JSON parser which is iterative. A deeply nested (but otherwise tiny) CBOR, MessagePack, UBJSON, or BSON document therefore exhausts the native call stack and crashes the process, rather than failing gracefully. Add a depth_guard RAII helper that tracks container-nesting depth on the binary_reader and rejects input nested deeper than a fixed max_depth (1024) with a clean parse_error (new id 116) instead of recursing further. The guard is placed at each recursive entry point: parse_cbor_internal, parse_msgpack_internal, parse_bson_internal (which also covers nested BSON sub-documents via parse_bson_element_internal), and get_ubjson_value (the actual recursive entry point for UBJSON/BJData, including the "optimized homogeneous array" code path that recurses without going through parse_ubjson_internal). Closes #5104. Testing: - Reproduced the crash for real against unpatched code on Windows: a 100k-deep CBOR/MessagePack/UBJSON/BSON payload each terminate the process with STATUS_STACK_OVERFLOW (exit code 0xC00000FD), matching the reported Segmentation fault. - Rebuilt against the patched header: all four formats now throw a clean [json.exception.parse_error.116] instead of crashing; process exits 0. - Added regression tests (unit-cbor.cpp, unit-msgpack.cpp, unit-ubjson.cpp, unit-bson.cpp) reproducing deep nesting for each format and asserting the clean parse_error; ran them against the fix (pass) and confirmed they return the crash on unpatched code first. - Compiled and ran the full unit-cbor/unit-msgpack/unit-ubjson/unit-bson test binaries (g++ 16, C++17) against both the patched and pristine baseline: identical pre-existing failure set (8 test cases that require the external json_test_data corpus, not present in this checkout) in both builds, and the patched build additionally passes the new tests with zero regressions elsewhere (3,549,340 assertions passed on the patched build vs 3,549,332 on baseline - exactly the added assertions). - Confirmed moderate/legitimate nesting (well under the new limit) and a normal round-trip still parse correctly, so the fix does not introduce false positives for realistic input. - Regenerated single_include/nlohmann/json.hpp via tools/amalgamate, and documented the new parse_error.116 in docs/mkdocs/docs/home/exceptions.md. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Kyue --- docs/mkdocs/docs/home/exceptions.md | 15 ++ .../nlohmann/detail/input/binary_reader.hpp | 73 +++++++ single_include/nlohmann/json.hpp | 199 ++++++++++++------ tests/src/unit-bson.cpp | 37 ++++ tests/src/unit-cbor.cpp | 14 ++ tests/src/unit-msgpack.cpp | 14 ++ tests/src/unit-ubjson.cpp | 19 ++ 7 files changed, 308 insertions(+), 63 deletions(-) diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index 859ebeb487..ad8f0ddc2b 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -366,6 +366,21 @@ A UBJSON high-precision number could not be parsed. [json.exception.parse_error.115] parse error at byte 5: syntax error while parsing UBJSON high-precision number: invalid number text: 1A ``` +### json.exception.parse_error.116 + +A binary input format (CBOR, MessagePack, UBJSON/BJData, or BSON) contained +arrays/objects nested deeper than the library's internal recursion limit. +The binary format parsers use recursive descent, so unbounded input nesting +would otherwise translate into unbounded native call-stack recursion and +could crash the process; inputs nested beyond the limit are rejected with +this parse error instead. + +!!! failure "Example message" + + ``` + [json.exception.parse_error.116] parse error at byte 1024: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded + ``` + ## Iterator errors This exception is thrown if iterators passed to a library function do not match diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index 4f8ab79f21..e45dbb4926 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -159,6 +159,39 @@ class binary_reader } private: + /*! + @brief RAII helper that tracks the nesting depth of containers while + parsing a binary input format. + + The binary format parsers (CBOR, MessagePack, UBJSON/BJData, BSON) parse + nested arrays/objects using recursive descent, so unbounded input nesting + directly translates into unbounded native call-stack recursion. This + guard increments the reader's depth counter for the lifetime of a single + recursive parse step so callers can reject inputs that nest deeper than + @ref max_depth before the recursion has a chance to exhaust the stack. + */ + class depth_guard + { + public: + explicit depth_guard(binary_reader& reader) noexcept : reader_(reader) + { + ++reader_.depth; + } + + ~depth_guard() + { + --reader_.depth; + } + + depth_guard(const depth_guard&) = delete; + depth_guard& operator=(const depth_guard&) = delete; + depth_guard(depth_guard&&) = delete; + depth_guard& operator=(depth_guard&&) = delete; + + private: + binary_reader& reader_; + }; + ////////// // BSON // ////////// @@ -169,6 +202,14 @@ class binary_reader */ bool parse_bson_internal() { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + } + std::int32_t document_size{}; get_number(input_format_t::bson, document_size); @@ -447,6 +488,14 @@ class binary_reader bool parse_cbor_internal(const bool get_char, const cbor_tag_handler_t tag_handler) { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + switch (get_char ? get() : current) { // EOF @@ -1209,6 +1258,14 @@ class binary_reader */ bool parse_msgpack_internal() { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + switch (get()) { // EOF @@ -2343,6 +2400,14 @@ class binary_reader */ bool get_ubjson_value(const char_int_type prefix) { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + switch (prefix) { case char_traits::eof(): // EOF @@ -3086,6 +3151,11 @@ class binary_reader private: static JSON_INLINE_VARIABLE constexpr std::size_t npos = detail::unknown_size(); + /// maximum allowed nesting depth of arrays/objects for the binary input + /// formats; guards the recursive-descent parsers against stack overflow + /// on maliciously/accidentally deeply nested input + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 1024; + /// input adapter InputAdapterType ia; @@ -3095,6 +3165,9 @@ class binary_reader /// the number of characters read std::size_t chars_read = 0; + /// current container nesting depth, see @ref max_depth and @ref depth_guard + std::size_t depth = 0; + /// whether we can assume little endianness const bool is_little_endian = little_endianness(); diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 580b9bd021..892aaa4eb4 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -3722,71 +3722,71 @@ NLOHMANN_JSON_NAMESPACE_END // SPDX-License-Identifier: MIT #ifndef INCLUDE_NLOHMANN_JSON_FWD_HPP_ - #define INCLUDE_NLOHMANN_JSON_FWD_HPP_ +#define INCLUDE_NLOHMANN_JSON_FWD_HPP_ - #include // int64_t, uint64_t - #include // map - #include // allocator - #include // string - #include // vector +#include // int64_t, uint64_t +#include // map +#include // allocator +#include // string +#include // vector - // #include +// #include - /*! - @brief namespace for Niels Lohmann - @see https://github.com/nlohmann - @since version 1.0.0 - */ - NLOHMANN_JSON_NAMESPACE_BEGIN +/*! +@brief namespace for Niels Lohmann +@see https://github.com/nlohmann +@since version 1.0.0 +*/ +NLOHMANN_JSON_NAMESPACE_BEGIN - /*! - @brief default JSONSerializer template argument +/*! +@brief default JSONSerializer template argument - This serializer ignores the template arguments and uses ADL - ([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) - for serialization. - */ - template - struct adl_serializer; - - /// a class to store JSON values - /// @sa https://json.nlohmann.me/api/basic_json/ - template class ObjectType = - std::map, - template class ArrayType = std::vector, - class StringType = std::string, class BooleanType = bool, - class NumberIntegerType = std::int64_t, - class NumberUnsignedType = std::uint64_t, - class NumberFloatType = double, - template class AllocatorType = std::allocator, - template class JSONSerializer = - adl_serializer, - class BinaryType = std::vector, // cppcheck-suppress syntaxError - class CustomBaseClass = void> - class basic_json; - - /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document - /// @sa https://json.nlohmann.me/api/json_pointer/ - template - class json_pointer; +This serializer ignores the template arguments and uses ADL +([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) +for serialization. +*/ +template +struct adl_serializer; + +/// a class to store JSON values +/// @sa https://json.nlohmann.me/api/basic_json/ +template class ObjectType = + std::map, + template class ArrayType = std::vector, + class StringType = std::string, class BooleanType = bool, + class NumberIntegerType = std::int64_t, + class NumberUnsignedType = std::uint64_t, + class NumberFloatType = double, + template class AllocatorType = std::allocator, + template class JSONSerializer = + adl_serializer, + class BinaryType = std::vector, // cppcheck-suppress syntaxError + class CustomBaseClass = void> +class basic_json; - /*! - @brief default specialization - @sa https://json.nlohmann.me/api/json/ - */ - using json = basic_json<>; +/// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document +/// @sa https://json.nlohmann.me/api/json_pointer/ +template +class json_pointer; - /// @brief a minimal map-like container that preserves insertion order - /// @sa https://json.nlohmann.me/api/ordered_map/ - template - struct ordered_map; +/*! +@brief default specialization +@sa https://json.nlohmann.me/api/json/ +*/ +using json = basic_json<>; - /// @brief specialization that maintains the insertion order of object keys - /// @sa https://json.nlohmann.me/api/ordered_json/ - using ordered_json = basic_json; +/// @brief a minimal map-like container that preserves insertion order +/// @sa https://json.nlohmann.me/api/ordered_map/ +template +struct ordered_map; - NLOHMANN_JSON_NAMESPACE_END +/// @brief specialization that maintains the insertion order of object keys +/// @sa https://json.nlohmann.me/api/ordered_json/ +using ordered_json = basic_json; + +NLOHMANN_JSON_NAMESPACE_END #endif // INCLUDE_NLOHMANN_JSON_FWD_HPP_ @@ -5857,7 +5857,7 @@ NLOHMANN_JSON_NAMESPACE_END // #include -// JSON_HAS_CPP_17 + // JSON_HAS_CPP_17 #ifdef JSON_HAS_CPP_17 #include // optional #endif @@ -10708,6 +10708,39 @@ class binary_reader } private: + /*! + @brief RAII helper that tracks the nesting depth of containers while + parsing a binary input format. + + The binary format parsers (CBOR, MessagePack, UBJSON/BJData, BSON) parse + nested arrays/objects using recursive descent, so unbounded input nesting + directly translates into unbounded native call-stack recursion. This + guard increments the reader's depth counter for the lifetime of a single + recursive parse step so callers can reject inputs that nest deeper than + @ref max_depth before the recursion has a chance to exhaust the stack. + */ + class depth_guard + { + public: + explicit depth_guard(binary_reader& reader) noexcept : reader_(reader) + { + ++reader_.depth; + } + + ~depth_guard() + { + --reader_.depth; + } + + depth_guard(const depth_guard&) = delete; + depth_guard& operator=(const depth_guard&) = delete; + depth_guard(depth_guard&&) = delete; + depth_guard& operator=(depth_guard&&) = delete; + + private: + binary_reader& reader_; + }; + ////////// // BSON // ////////// @@ -10718,6 +10751,14 @@ class binary_reader */ bool parse_bson_internal() { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + } + std::int32_t document_size{}; get_number(input_format_t::bson, document_size); @@ -10996,6 +11037,14 @@ class binary_reader bool parse_cbor_internal(const bool get_char, const cbor_tag_handler_t tag_handler) { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + switch (get_char ? get() : current) { // EOF @@ -11758,6 +11807,14 @@ class binary_reader */ bool parse_msgpack_internal() { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + switch (get()) { // EOF @@ -12892,6 +12949,14 @@ class binary_reader */ bool get_ubjson_value(const char_int_type prefix) { + const depth_guard dg(*this); + if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + switch (prefix) { case char_traits::eof(): // EOF @@ -13635,6 +13700,11 @@ class binary_reader private: static JSON_INLINE_VARIABLE constexpr std::size_t npos = detail::unknown_size(); + /// maximum allowed nesting depth of arrays/objects for the binary input + /// formats; guards the recursive-descent parsers against stack overflow + /// on maliciously/accidentally deeply nested input + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 1024; + /// input adapter InputAdapterType ia; @@ -13644,6 +13714,9 @@ class binary_reader /// the number of characters read std::size_t chars_read = 0; + /// current container nesting depth, see @ref max_depth and @ref depth_guard + std::size_t depth = 0; + /// whether we can assume little endianness const bool is_little_endian = little_endianness(); @@ -21257,10 +21330,10 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bool allow_exceptions = true, const bool ignore_comments = false, const bool ignore_trailing_commas = false - ) + ) { return ::nlohmann::detail::parser(std::move(adapter), - std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); + std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); } private: @@ -21958,8 +22031,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::enable_if_t < !detail::is_basic_json::value && detail::is_compatible_type::value, int > = 0 > basic_json(CompatibleType && val) noexcept(noexcept( // NOLINT(bugprone-forwarding-reference-overload,bugprone-exception-escape) - JSONSerializer::to_json(std::declval(), - std::forward(val)))) + JSONSerializer::to_json(std::declval(), + std::forward(val)))) { JSONSerializer::to_json(*this, std::forward(val)); set_parents(); @@ -22762,7 +22835,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<0> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), std::declval()))) + JSONSerializer::from_json(std::declval(), std::declval()))) { auto ret = ValueType(); JSONSerializer::from_json(*this, ret); @@ -22804,7 +22877,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_non_default_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<1> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval()))) + JSONSerializer::from_json(std::declval()))) { return JSONSerializer::from_json(*this); } @@ -22954,7 +23027,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType & get_to(ValueType& v) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), v))) + JSONSerializer::from_json(std::declval(), v))) { JSONSerializer::from_json(*this, v); return v; diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index 28837d75e5..b89c0fc143 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -854,6 +854,43 @@ TEST_CASE("Unsupported BSON input") CHECK(!json::sax_parse(bson, &scp, json::input_format_t::bson)); } +TEST_CASE("BSON regressions") +{ + SECTION("stack overflow via deeply nested input (issue #5104)") + { + // A chain of 1100 embedded BSON documents (record type 0x03), each + // wrapping the previous one under key "a". The recursive-descent + // parser must reject this with a clean parse_error once the nesting + // exceeds the depth limit, rather than exhausting the native call + // stack. + std::vector inner = {5, 0, 0, 0, 0}; // innermost empty document + for (int i = 0; i < 1100; ++i) + { + std::vector elem = {0x03, 'a', 0x00}; // embedded-document element, key "a" + elem.insert(elem.end(), inner.begin(), inner.end()); + + const auto doc_size = static_cast(4 + elem.size() + 1); + std::vector doc = + { + static_cast(doc_size & 0xFF), + static_cast((doc_size >> 8) & 0xFF), + static_cast((doc_size >> 16) & 0xFF), + static_cast((doc_size >> 24) & 0xFF), + }; + doc.insert(doc.end(), elem.begin(), elem.end()); + doc.push_back(0x00); // terminator + + inner = doc; + } + + json _; + CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), + "[json.exception.parse_error.116] parse error at byte 7168: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + json::parse_error&); + CHECK(json::from_bson(inner, true, false).is_discarded()); + } +} + TEST_CASE("BSON numerical data") { SECTION("number") diff --git a/tests/src/unit-cbor.cpp b/tests/src/unit-cbor.cpp index af73642997..b628c0f02b 100644 --- a/tests/src/unit-cbor.cpp +++ b/tests/src/unit-cbor.cpp @@ -1996,6 +1996,20 @@ TEST_CASE("CBOR regressions") } } } + + SECTION("stack overflow via deeply nested input (issue #5104)") + { + // 2000 nested CBOR arrays of length 1 (0x81 each); the recursive-descent + // parser must reject this with a clean parse_error once the nesting + // exceeds the depth limit, rather than exhausting the native call stack. + std::vector vec(2000, 0x81); + vec.push_back(0x00); + json _; // NOLINT(readability-identifier-naming) + CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), + "[json.exception.parse_error.116] parse error at byte 1024: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", + json::parse_error&); + CHECK(json::from_cbor(vec, true, false).is_discarded()); + } } #endif diff --git a/tests/src/unit-msgpack.cpp b/tests/src/unit-msgpack.cpp index 4358cd4646..a81b4c3daa 100644 --- a/tests/src/unit-msgpack.cpp +++ b/tests/src/unit-msgpack.cpp @@ -1570,6 +1570,20 @@ TEST_CASE("MessagePack") CHECK(json::from_msgpack(vec, true, false).is_discarded()); } } + + SECTION("stack overflow via deeply nested input (issue #5104)") + { + // 2000 nested fixarrays of length 1 (0x91 each); the recursive-descent + // parser must reject this with a clean parse_error once the nesting + // exceeds the depth limit, rather than exhausting the native call stack. + std::vector vec(2000, 0x91); + vec.push_back(0x00); + json _; + CHECK_THROWS_WITH_AS(_ = json::from_msgpack(vec), + "[json.exception.parse_error.116] parse error at byte 1024: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", + json::parse_error&); + CHECK(json::from_msgpack(vec, true, false).is_discarded()); + } } SECTION("SAX aborts") diff --git a/tests/src/unit-ubjson.cpp b/tests/src/unit-ubjson.cpp index 6df9acfac6..c79775553c 100644 --- a/tests/src/unit-ubjson.cpp +++ b/tests/src/unit-ubjson.cpp @@ -1638,6 +1638,25 @@ TEST_CASE("UBJSON") CHECK_THROWS_AS(_ = json::sax_parse(v_ubjson, &scp, json::input_format_t::ubjson), json::out_of_range&); } } + + SECTION("stack overflow via deeply nested input (issue #5104)") + { + // 2000 nested UBJSON arrays; the recursive-descent parser must reject + // this with a clean parse_error once the nesting exceeds the depth + // limit, rather than exhausting the native call stack. + std::string payload(2000, '['); + payload += "Z"; + for (int i = 0; i < 2000; ++i) + { + payload += ']'; + } + std::vector const v(payload.begin(), payload.end()); + json _; + CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), + "[json.exception.parse_error.116] parse error at byte 1025: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", + json::parse_error&); + CHECK(json::from_ubjson(v, true, false).is_discarded()); + } } SECTION("SAX aborts") From 730879327f2ca2f697adcaab4f44a70809d7758e Mon Sep 17 00:00:00 2001 From: Kyue Date: Wed, 22 Jul 2026 20:42:10 +0100 Subject: [PATCH 02/14] Apply CI-generated amalgamation formatting patch single_include/nlohmann/json.hpp had drifted out of astyle-formatting sync in a few pre-existing regions unrelated to this PR's actual change (json_fwd.hpp block indentation, a couple of wrapped call sites); this is the exact patch the "Check amalgamation" GitHub Action generated and offered as a downloadable artifact. Applying it as-is, no manual edits. Signed-off-by: Kyue --- single_include/nlohmann/json.hpp | 126 +++++++++++++++---------------- 1 file changed, 63 insertions(+), 63 deletions(-) diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 892aaa4eb4..9cad71262e 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -3722,71 +3722,71 @@ NLOHMANN_JSON_NAMESPACE_END // SPDX-License-Identifier: MIT #ifndef INCLUDE_NLOHMANN_JSON_FWD_HPP_ -#define INCLUDE_NLOHMANN_JSON_FWD_HPP_ + #define INCLUDE_NLOHMANN_JSON_FWD_HPP_ -#include // int64_t, uint64_t -#include // map -#include // allocator -#include // string -#include // vector - -// #include + #include // int64_t, uint64_t + #include // map + #include // allocator + #include // string + #include // vector + // #include -/*! -@brief namespace for Niels Lohmann -@see https://github.com/nlohmann -@since version 1.0.0 -*/ -NLOHMANN_JSON_NAMESPACE_BEGIN -/*! -@brief default JSONSerializer template argument + /*! + @brief namespace for Niels Lohmann + @see https://github.com/nlohmann + @since version 1.0.0 + */ + NLOHMANN_JSON_NAMESPACE_BEGIN -This serializer ignores the template arguments and uses ADL -([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) -for serialization. -*/ -template -struct adl_serializer; - -/// a class to store JSON values -/// @sa https://json.nlohmann.me/api/basic_json/ -template class ObjectType = - std::map, - template class ArrayType = std::vector, - class StringType = std::string, class BooleanType = bool, - class NumberIntegerType = std::int64_t, - class NumberUnsignedType = std::uint64_t, - class NumberFloatType = double, - template class AllocatorType = std::allocator, - template class JSONSerializer = - adl_serializer, - class BinaryType = std::vector, // cppcheck-suppress syntaxError - class CustomBaseClass = void> -class basic_json; + /*! + @brief default JSONSerializer template argument -/// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document -/// @sa https://json.nlohmann.me/api/json_pointer/ -template -class json_pointer; + This serializer ignores the template arguments and uses ADL + ([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) + for serialization. + */ + template + struct adl_serializer; + + /// a class to store JSON values + /// @sa https://json.nlohmann.me/api/basic_json/ + template class ObjectType = + std::map, + template class ArrayType = std::vector, + class StringType = std::string, class BooleanType = bool, + class NumberIntegerType = std::int64_t, + class NumberUnsignedType = std::uint64_t, + class NumberFloatType = double, + template class AllocatorType = std::allocator, + template class JSONSerializer = + adl_serializer, + class BinaryType = std::vector, // cppcheck-suppress syntaxError + class CustomBaseClass = void> + class basic_json; + + /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document + /// @sa https://json.nlohmann.me/api/json_pointer/ + template + class json_pointer; -/*! -@brief default specialization -@sa https://json.nlohmann.me/api/json/ -*/ -using json = basic_json<>; + /*! + @brief default specialization + @sa https://json.nlohmann.me/api/json/ + */ + using json = basic_json<>; -/// @brief a minimal map-like container that preserves insertion order -/// @sa https://json.nlohmann.me/api/ordered_map/ -template -struct ordered_map; + /// @brief a minimal map-like container that preserves insertion order + /// @sa https://json.nlohmann.me/api/ordered_map/ + template + struct ordered_map; -/// @brief specialization that maintains the insertion order of object keys -/// @sa https://json.nlohmann.me/api/ordered_json/ -using ordered_json = basic_json; + /// @brief specialization that maintains the insertion order of object keys + /// @sa https://json.nlohmann.me/api/ordered_json/ + using ordered_json = basic_json; -NLOHMANN_JSON_NAMESPACE_END + NLOHMANN_JSON_NAMESPACE_END #endif // INCLUDE_NLOHMANN_JSON_FWD_HPP_ @@ -5857,7 +5857,7 @@ NLOHMANN_JSON_NAMESPACE_END // #include - // JSON_HAS_CPP_17 +// JSON_HAS_CPP_17 #ifdef JSON_HAS_CPP_17 #include // optional #endif @@ -21330,10 +21330,10 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bool allow_exceptions = true, const bool ignore_comments = false, const bool ignore_trailing_commas = false - ) + ) { return ::nlohmann::detail::parser(std::move(adapter), - std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); + std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); } private: @@ -22031,8 +22031,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::enable_if_t < !detail::is_basic_json::value && detail::is_compatible_type::value, int > = 0 > basic_json(CompatibleType && val) noexcept(noexcept( // NOLINT(bugprone-forwarding-reference-overload,bugprone-exception-escape) - JSONSerializer::to_json(std::declval(), - std::forward(val)))) + JSONSerializer::to_json(std::declval(), + std::forward(val)))) { JSONSerializer::to_json(*this, std::forward(val)); set_parents(); @@ -22835,7 +22835,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<0> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), std::declval()))) + JSONSerializer::from_json(std::declval(), std::declval()))) { auto ret = ValueType(); JSONSerializer::from_json(*this, ret); @@ -22877,7 +22877,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_non_default_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<1> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval()))) + JSONSerializer::from_json(std::declval()))) { return JSONSerializer::from_json(*this); } @@ -23027,7 +23027,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType & get_to(ValueType& v) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), v))) + JSONSerializer::from_json(std::declval(), v))) { JSONSerializer::from_json(*this, v); return v; From a3a2519ad6df0aa58914544a20c1401c9dc54eb3 Mon Sep 17 00:00:00 2001 From: Kyue Date: Wed, 22 Jul 2026 21:31:09 +0100 Subject: [PATCH 03/14] Lower binary-format max_depth to 128 to fix MSVC Debug stack overflow CI on this branch crashed with a real SIGSEGV (stack overflow) in test-bson_cpp11 under msvc (Debug, x64), msvc-arm64 (Debug), and msvc-vs2026 (Debug, x64) - the depth_guard added to fix #5104 correctly caps recursion at max_depth=1024, but reaching depth 1024 itself already overflows a 1 MiB default thread stack in unoptimized/Debug builds, where each nesting level costs several real (non-inlined) call frames instead of one. No MSVC toolchain is available in this environment, so instead of guessing a safe constant, I only lowered it (1024 -> 128, a 4x margin under the depth that crashed) and am relying on CI's real Windows/MSVC runners - which reproduced the original bug - to confirm this closes it. Verified locally with g++ 16/C++17 (not MSVC): recompiled and ran the affected unit-bson/cbor/msgpack/ubjson test binaries after updating the four depth-limit regression tests' expected byte offsets (which shift with max_depth: CBOR/MessagePack 1024->128, UBJSON 1025->129, BSON 7168->896, cross-checked against actual thrown exception messages, not guessed). All four pass; the only failures are the same 9 pre-existing "external json_test_data corpus not present" cases already documented as a known baseline gap in this checkout - no new regressions. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Kyue --- docs/mkdocs/docs/home/exceptions.md | 2 +- include/nlohmann/detail/input/binary_reader.hpp | 10 ++++++++-- single_include/nlohmann/json.hpp | 10 ++++++++-- tests/src/unit-bson.cpp | 2 +- tests/src/unit-cbor.cpp | 2 +- tests/src/unit-msgpack.cpp | 2 +- tests/src/unit-ubjson.cpp | 2 +- 7 files changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index ad8f0ddc2b..a23d31aa1c 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -378,7 +378,7 @@ this parse error instead. !!! failure "Example message" ``` - [json.exception.parse_error.116] parse error at byte 1024: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded + [json.exception.parse_error.116] parse error at byte 128: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded ``` ## Iterator errors diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index e45dbb4926..3eafe17ccb 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -3153,8 +3153,14 @@ class binary_reader /// maximum allowed nesting depth of arrays/objects for the binary input /// formats; guards the recursive-descent parsers against stack overflow - /// on maliciously/accidentally deeply nested input - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 1024; + /// on maliciously/accidentally deeply nested input. + /// + /// Kept well below the theoretical limit of a 1 MiB default thread stack: + /// each nesting level costs multiple real call frames (the recursive + /// entry point plus helpers it calls into), and unoptimized/Debug builds + /// (notably MSVC Debug, which crashed CI at max_depth=1024 - see #5104) + /// use substantially more stack per frame than an optimized build. + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 128; /// input adapter InputAdapterType ia; diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 9cad71262e..f3ec759c3e 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -13702,8 +13702,14 @@ class binary_reader /// maximum allowed nesting depth of arrays/objects for the binary input /// formats; guards the recursive-descent parsers against stack overflow - /// on maliciously/accidentally deeply nested input - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 1024; + /// on maliciously/accidentally deeply nested input. + /// + /// Kept well below the theoretical limit of a 1 MiB default thread stack: + /// each nesting level costs multiple real call frames (the recursive + /// entry point plus helpers it calls into), and unoptimized/Debug builds + /// (notably MSVC Debug, which crashed CI at max_depth=1024 - see #5104) + /// use substantially more stack per frame than an optimized build. + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 128; /// input adapter InputAdapterType ia; diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index b89c0fc143..21be0b5448 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -885,7 +885,7 @@ TEST_CASE("BSON regressions") json _; CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), - "[json.exception.parse_error.116] parse error at byte 7168: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + "[json.exception.parse_error.116] parse error at byte 896: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", json::parse_error&); CHECK(json::from_bson(inner, true, false).is_discarded()); } diff --git a/tests/src/unit-cbor.cpp b/tests/src/unit-cbor.cpp index b628c0f02b..e94daec22c 100644 --- a/tests/src/unit-cbor.cpp +++ b/tests/src/unit-cbor.cpp @@ -2006,7 +2006,7 @@ TEST_CASE("CBOR regressions") vec.push_back(0x00); json _; // NOLINT(readability-identifier-naming) CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), - "[json.exception.parse_error.116] parse error at byte 1024: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 128: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_cbor(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-msgpack.cpp b/tests/src/unit-msgpack.cpp index a81b4c3daa..2226f1e534 100644 --- a/tests/src/unit-msgpack.cpp +++ b/tests/src/unit-msgpack.cpp @@ -1580,7 +1580,7 @@ TEST_CASE("MessagePack") vec.push_back(0x00); json _; CHECK_THROWS_WITH_AS(_ = json::from_msgpack(vec), - "[json.exception.parse_error.116] parse error at byte 1024: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 128: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_msgpack(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-ubjson.cpp b/tests/src/unit-ubjson.cpp index c79775553c..1912bc3a90 100644 --- a/tests/src/unit-ubjson.cpp +++ b/tests/src/unit-ubjson.cpp @@ -1653,7 +1653,7 @@ TEST_CASE("UBJSON") std::vector const v(payload.begin(), payload.end()); json _; CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), - "[json.exception.parse_error.116] parse error at byte 1025: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 129: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_ubjson(v, true, false).is_discarded()); } From 0df0aa9623c3f05fe158760d034518ec4e2c8da6 Mon Sep 17 00:00:00 2001 From: Kyue Date: Wed, 22 Jul 2026 22:34:10 +0100 Subject: [PATCH 04/14] Raise max_depth from 128 to 600 - 128 broke legitimate deep input My previous commit dropped max_depth to 128 to fix the MSVC Debug stack overflow from #5104, but that number was picked without checking whether it was actually high enough for real input. It wasn't: the library's own roundtrip fixture (tests/data/json_testsuite/sample.json, used by all four binary format roundtrip tests) legitimately nests 458 levels deep, so 128 started rejecting valid, previously-working input - visible in CI as widespread new failures across nearly every job, not just the Debug builds this was meant to fix. 600 covers that fixture with real margin (about 30% above the 458 it actually needs) while staying well clear of the 1024 that overflowed the stack in the first place (about 40% below it). Verified locally (g++, no MSVC available): downloaded the actual sample.json fixture and measured its real max nesting depth directly (458) rather than guessing, recomputed the four depth-limit regression tests' expected byte offsets for the new limit the same way as before (cross-checked against actual thrown exception text), and reran the full binary-format test files - all four depth-guard tests pass, and the only other failures are the same pre-existing "external test corpus not present in this checkout" gaps as before, nothing newly broken. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Kyue --- docs/mkdocs/docs/home/exceptions.md | 2 +- include/nlohmann/detail/input/binary_reader.hpp | 16 ++++++++++------ single_include/nlohmann/json.hpp | 16 ++++++++++------ tests/src/unit-bson.cpp | 2 +- tests/src/unit-cbor.cpp | 2 +- tests/src/unit-msgpack.cpp | 2 +- tests/src/unit-ubjson.cpp | 2 +- 7 files changed, 25 insertions(+), 17 deletions(-) diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index a23d31aa1c..ffccf034ba 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -378,7 +378,7 @@ this parse error instead. !!! failure "Example message" ``` - [json.exception.parse_error.116] parse error at byte 128: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded + [json.exception.parse_error.116] parse error at byte 600: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded ``` ## Iterator errors diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index 3eafe17ccb..98b26367dd 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -3155,12 +3155,16 @@ class binary_reader /// formats; guards the recursive-descent parsers against stack overflow /// on maliciously/accidentally deeply nested input. /// - /// Kept well below the theoretical limit of a 1 MiB default thread stack: - /// each nesting level costs multiple real call frames (the recursive - /// entry point plus helpers it calls into), and unoptimized/Debug builds - /// (notably MSVC Debug, which crashed CI at max_depth=1024 - see #5104) - /// use substantially more stack per frame than an optimized build. - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 128; + /// This has to satisfy two constraints in tension: high enough that it + /// doesn't reject legitimate input (the library's own roundtrip test + /// fixture, tests/data/json_testsuite/sample.json, nests 458 levels + /// deep), but low enough that reaching it doesn't itself exhaust the + /// native call stack - each nesting level costs multiple real call + /// frames, and unoptimized/Debug builds (notably MSVC Debug, which + /// crashed CI at max_depth=1024 - see #5104) use substantially more + /// stack per frame than an optimized build. 600 leaves comfortable + /// margin on both sides of that range. + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 600; /// input adapter InputAdapterType ia; diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index f3ec759c3e..2979ea148e 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -13704,12 +13704,16 @@ class binary_reader /// formats; guards the recursive-descent parsers against stack overflow /// on maliciously/accidentally deeply nested input. /// - /// Kept well below the theoretical limit of a 1 MiB default thread stack: - /// each nesting level costs multiple real call frames (the recursive - /// entry point plus helpers it calls into), and unoptimized/Debug builds - /// (notably MSVC Debug, which crashed CI at max_depth=1024 - see #5104) - /// use substantially more stack per frame than an optimized build. - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 128; + /// This has to satisfy two constraints in tension: high enough that it + /// doesn't reject legitimate input (the library's own roundtrip test + /// fixture, tests/data/json_testsuite/sample.json, nests 458 levels + /// deep), but low enough that reaching it doesn't itself exhaust the + /// native call stack - each nesting level costs multiple real call + /// frames, and unoptimized/Debug builds (notably MSVC Debug, which + /// crashed CI at max_depth=1024 - see #5104) use substantially more + /// stack per frame than an optimized build. 600 leaves comfortable + /// margin on both sides of that range. + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 600; /// input adapter InputAdapterType ia; diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index 21be0b5448..55952caffa 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -885,7 +885,7 @@ TEST_CASE("BSON regressions") json _; CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), - "[json.exception.parse_error.116] parse error at byte 896: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + "[json.exception.parse_error.116] parse error at byte 4200: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", json::parse_error&); CHECK(json::from_bson(inner, true, false).is_discarded()); } diff --git a/tests/src/unit-cbor.cpp b/tests/src/unit-cbor.cpp index e94daec22c..b1e96c8cf4 100644 --- a/tests/src/unit-cbor.cpp +++ b/tests/src/unit-cbor.cpp @@ -2006,7 +2006,7 @@ TEST_CASE("CBOR regressions") vec.push_back(0x00); json _; // NOLINT(readability-identifier-naming) CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), - "[json.exception.parse_error.116] parse error at byte 128: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 600: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_cbor(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-msgpack.cpp b/tests/src/unit-msgpack.cpp index 2226f1e534..56485ee493 100644 --- a/tests/src/unit-msgpack.cpp +++ b/tests/src/unit-msgpack.cpp @@ -1580,7 +1580,7 @@ TEST_CASE("MessagePack") vec.push_back(0x00); json _; CHECK_THROWS_WITH_AS(_ = json::from_msgpack(vec), - "[json.exception.parse_error.116] parse error at byte 128: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 600: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_msgpack(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-ubjson.cpp b/tests/src/unit-ubjson.cpp index 1912bc3a90..69da1e571b 100644 --- a/tests/src/unit-ubjson.cpp +++ b/tests/src/unit-ubjson.cpp @@ -1653,7 +1653,7 @@ TEST_CASE("UBJSON") std::vector const v(payload.begin(), payload.end()); json _; CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), - "[json.exception.parse_error.116] parse error at byte 129: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 601: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_ubjson(v, true, false).is_discarded()); } From 2372e68b4e619e2d3bbd28a6f307efd66716aa57 Mon Sep 17 00:00:00 2001 From: Kyue Date: Thu, 23 Jul 2026 12:13:37 +0100 Subject: [PATCH 05/14] Force-inline parse_ubjson_internal to fix remaining UBJSON stack overflow CI (clang-cl-12 x64, Windows Debug) still crashed with a real SIGSEGV in test-ubjson_cpp11 on the #5104 regression test even with max_depth=600 in place, while the equivalent CBOR/MessagePack/BSON depth-limit tests passed fine at the same max_depth. The depth_guard itself was applied correctly on every UBJSON recursive call site (verified by reading every recursive path in get_ubjson_array/get_ubjson_object and the optimized $type#count path) - the bug wasn't a missing or bypassed check, it was that reaching the *same* depth of 600 costs UBJSON more real stack than it costs the other formats. Root cause: CBOR/MessagePack fold "read the next value's dispatch" and "the depth_guard" into one function (parse_cbor_internal / parse_msgpack_internal), so their recursive chain is two stack frames per nesting level (that function + get_cbor_array/get_msgpack_array). UBJSON splits this into two functions instead of one: parse_ubjson_internal() (fetch the next byte) calling get_ubjson_value() (the depth_guard + the switch), because get_ubjson_value() also has to be callable directly with an already-known type marker from the optimized `$type#count` container path - a feature the other formats don't have. That leaves parse_ubjson_internal() as a genuine extra frame on top of get_ubjson_value() and get_ubjson_array()/get_ubjson_object(): three frames per nesting level instead of two. Measured with `g++ -O0 -fstack-usage` (proxy for MSVC Debug's similarly unoptimized frames): parse_ubjson_internal/get_ubjson_value/get_ubjson_array together cost ~1488 bytes per nesting level pre-fix vs CBOR's parse_cbor_internal/get_cbor_array at ~1088 bytes/level - about 37% more stack per level for the same max_depth, comfortably enough to explain overflowing the 4 MB stack (tests/CMakeLists.txt already sets /STACK:4000000 for these test binaries, see #2955) at depth 600 in MSVC Debug specifically. Fix: mark parse_ubjson_internal() JSON_HEDLEY_ALWAYS_INLINE so it collapses into its caller in every build configuration, including unoptimized Debug builds where the compiler wouldn't otherwise bother inlining a function this trivial. Confirmed with objdump that this eliminates every `call` to parse_ubjson_internal in the compiled object (previously 2 real calls, now 0) - i.e. it's not just a hint, the frame is actually gone - without touching get_ubjson_value's depth_guard/max_depth logic at all, so the 128->600 max_depth tuning from the previous two commits (needed for the 458-level-deep legitimate roundtrip fixture) is untouched. Regenerated single_include/nlohmann/json.hpp for this one function only (re-running the full amalgamate script produced unrelated whitespace churn in this environment, so I hand-applied the equivalent change to keep the diff minimal and matching). Verified locally with g++ 16 (MinGW, Debug, no MSVC available in this environment): rebuilt test-ubjson_cpp11/test-cbor_cpp11/test-msgpack_cpp11/ test-bson_cpp11 against both the multi-header and the amalgamated single_include build, and all non-data-file-dependent test cases pass, including the #5104 stack-overflow regression test (691k+ assertions) and the pre-existing legitimate-deep-input roundtrip test. The only failures are the already-documented external-test-corpus-not-downloaded gap (network access unavailable here) and one pre-existing, unrelated vector::reserve failure in a CBOR RFC 7049 test that reproduces identically before and after this change. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Kyue --- include/nlohmann/detail/input/binary_reader.hpp | 15 +++++++++++++++ single_include/nlohmann/json.hpp | 15 +++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index 98b26367dd..c73317a44a 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -1897,7 +1897,22 @@ class binary_reader character should be considered instead @return whether a valid UBJSON value was passed to the SAX parser + + @note Unlike CBOR/MessagePack, UBJSON needs get_ubjson_value() (which + carries the @ref depth_guard) to be reachable both from here *and* + directly with an already-known type marker (the optimized `$type#count` + container path), so the dispatch cannot simply be folded into this + function the way parse_cbor_internal()/parse_msgpack_internal() do. That + leaves this thin wrapper as a genuine extra stack frame on every ordinary + array/object element, on top of get_ubjson_value() and + get_ubjson_array()/get_ubjson_object() - one more frame per nesting level + than CBOR/MessagePack/BSON need for the same @ref max_depth. Force-inlining + it (rather than trusting Debug/-O0 builds to do so) keeps UBJSON's + recursion cost per level in line with the other formats' and was + necessary to stop deeply nested UBJSON input from overflowing the stack + well before max_depth is reached in MSVC Debug builds (#5104). */ + JSON_HEDLEY_ALWAYS_INLINE bool parse_ubjson_internal(const bool get_char = true) { return get_ubjson_value(get_char ? get_ignore_noop() : current); diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 2979ea148e..e0c4473dcd 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -12446,7 +12446,22 @@ class binary_reader character should be considered instead @return whether a valid UBJSON value was passed to the SAX parser + + @note Unlike CBOR/MessagePack, UBJSON needs get_ubjson_value() (which + carries the @ref depth_guard) to be reachable both from here *and* + directly with an already-known type marker (the optimized `$type#count` + container path), so the dispatch cannot simply be folded into this + function the way parse_cbor_internal()/parse_msgpack_internal() do. That + leaves this thin wrapper as a genuine extra stack frame on every ordinary + array/object element, on top of get_ubjson_value() and + get_ubjson_array()/get_ubjson_object() - one more frame per nesting level + than CBOR/MessagePack/BSON need for the same @ref max_depth. Force-inlining + it (rather than trusting Debug/-O0 builds to do so) keeps UBJSON's + recursion cost per level in line with the other formats' and was + necessary to stop deeply nested UBJSON input from overflowing the stack + well before max_depth is reached in MSVC Debug builds (#5104). */ + JSON_HEDLEY_ALWAYS_INLINE bool parse_ubjson_internal(const bool get_char = true) { return get_ubjson_value(get_char ? get_ignore_noop() : current); From 256d7b24ef866d39921a4eb899f3238575738de2 Mon Sep 17 00:00:00 2001 From: Kyue Date: Thu, 23 Jul 2026 14:43:32 +0100 Subject: [PATCH 06/14] Rewrite binary-format container parsing as iterative (non-recursive) nlohmann suggested a non-recursive parser would be a better fix than continuing to tune max_depth against compiler/platform-specific stack frame sizes (the UBJSON force-inline fix still wasn't enough on clang-cl Debug). This does that: the container-parsing loops for CBOR, MessagePack, UBJSON/BJData, and BSON no longer recurse on the native call stack for nested arrays/objects. Each format's parse_*_internal() now runs a single iterative loop with an explicit, heap-allocated stack of small per-format "resume this container" frames (cbor_container_frame, msgpack_container_frame, ubjson_container_frame, bson_container_frame); descending into a nested array/object pushes a frame and loops back to parse its first child instead of recursing, and finishing a container pops its frame and resumes the parent. Scalar values (numbers, strings, binaries) are unaffected - they never recursed into container parsing in the first place. This makes native stack depth O(1) regardless of input nesting depth, so it eliminates the whole bug class (#5104) rather than tuning a magic number against a specific compiler's frame sizes: CBOR's separate tag-unwrapping recursion is now a plain loop too (it was also uncapped in all but name, since it recursed through the same depth-guarded function). depth_guard and the per-call recursion-depth counter are gone entirely - they'd have nothing left to guard - and max_depth (now 100000, up from 600) is kept purely as a sanity/DoS cap against absurd/malicious inputs, not a stack-safety mechanism; it's checked against the explicit stack's size at container-entry time instead. While rewriting BSON's container handling I noticed nested *arrays* (record type 0x04) never went through the old depth_guard at all - only nested objects (type 0x03) did, via parse_bson_internal. The unified iterative loop applies the depth cap to both uniformly now, closing that gap; a regression test for it is included alongside the updated #5104 tests. Testing: - Updated the four existing #5104 regression tests (CBOR/MessagePack/ UBJSON/BSON) for the new max_depth=100000 and the resulting byte offsets in the parse_error messages; added a BSON array-nesting variant of the same test for the gap above. - Ran the full existing test suite (g++ 16, Debug, MinGW) - CBOR, MessagePack (cpp11+cpp17), UBJSON, BSON, BJData, and every other suite (94 binaries total) - and diffed assertion-by-assertion against a pristine baseline build of the pre-rewrite tree: every binary matches the baseline's pass/fail counts exactly except the two new BSON assertions, which pass. The one pre-existing failure per binary format suite (missing external json_test_data fuzz corpus - no network access in this environment - plus one unrelated vector::reserve failure in a CBOR RFC 7049 test) reproduces identically in both trees. - Added a throwaway smoke test parsing 500,000 levels of nesting (~830x the old max_depth of 600, far beyond anything the old recursive parser could have survived at any max_depth) for all four formats via a minimal non-DOM-building SAX consumer: all four reject it cleanly via parse_error.116 with no crash, confirming this isn't luck at one specific depth. Separately confirmed 50,000 levels of *legitimate* nesting (under the new cap, well past the old one) parses to completion correctly, and that the BJData "optimized homogeneous container" and ND-array-annotation paths - the trickiest part of the UBJSON rewrite - are exercised by the existing test suite and match baseline exactly. - Regenerated single_include/nlohmann/json.hpp via tools/amalgamate and reformatted with the pinned astyle (3.4.13) matching the CI "Check amalgamation" workflow's exact sequence (amalgamate.py for both json.hpp/json_fwd.hpp configs, then astyle on the amalgamated output and on include/tests/docs sources); diffed clean against a fresh regeneration. No real clang-cl/MSVC available in this environment, so the exact CI toolchain isn't locally reproducible - but the whole point of this rewrite is that stack safety no longer depends on compiler/platform frame sizes at all, so a clean run under a different compiler (optimized or not) is meaningful structural evidence rather than a coincidence of this one toolchain's frame sizes, the way the previous max_depth tuning attempts were. Co-Authored-By: Claude Sonnet 5 Signed-off-by: Kyue --- docs/mkdocs/docs/home/exceptions.md | 14 +- .../nlohmann/detail/input/binary_reader.hpp | 3510 ++++++++------- single_include/nlohmann/json.hpp | 3746 +++++++++-------- tests/src/unit-bson.cpp | 112 +- tests/src/unit-cbor.cpp | 13 +- tests/src/unit-msgpack.cpp | 13 +- tests/src/unit-ubjson.cpp | 15 +- 7 files changed, 3986 insertions(+), 3437 deletions(-) diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index ffccf034ba..c66a3f0d91 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -369,16 +369,18 @@ A UBJSON high-precision number could not be parsed. ### json.exception.parse_error.116 A binary input format (CBOR, MessagePack, UBJSON/BJData, or BSON) contained -arrays/objects nested deeper than the library's internal recursion limit. -The binary format parsers use recursive descent, so unbounded input nesting -would otherwise translate into unbounded native call-stack recursion and -could crash the process; inputs nested beyond the limit are rejected with -this parse error instead. +arrays/objects nested deeper than the library's internal depth limit +(100000). The binary format parsers track container nesting on an explicit, +heap-allocated stack rather than the native call stack, so nesting depth can +no longer cause a stack overflow regardless of this limit; the limit exists +purely as a sanity check against absurd/malicious inputs (for example, one +designed to exhaust memory or time rather than the call stack), and inputs +nested beyond it are rejected with this parse error instead. !!! failure "Example message" ``` - [json.exception.parse_error.116] parse error at byte 600: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded + [json.exception.parse_error.116] parse error at byte 100001: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded ``` ## Iterator errors diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index c73317a44a..b3a1dcfc97 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -159,71 +159,209 @@ class binary_reader } private: + ////////// + // BSON // + ////////// + /*! - @brief RAII helper that tracks the nesting depth of containers while - parsing a binary input format. - - The binary format parsers (CBOR, MessagePack, UBJSON/BJData, BSON) parse - nested arrays/objects using recursive descent, so unbounded input nesting - directly translates into unbounded native call-stack recursion. This - guard increments the reader's depth counter for the lifetime of a single - recursive parse step so callers can reject inputs that nest deeper than - @ref max_depth before the recursion has a chance to exhaust the stack. + @brief a single pending "resume this document/array" entry used by the + iterative container traversal in @ref parse_bson_internal; see + @ref cbor_container_frame for the general idea. + + BSON documents and arrays share one on-the-wire layout (a size-prefixed, + 0x00-terminated list of typed elements); @ref is_object only controls + whether each element's key is relayed to the SAX consumer via + @c sax->key() (an object) or not (an array, whose "0", "1", ... keys the + SAX array interface has no use for). Every BSON container is + terminator-based (there is no known-element-count form), so unlike CBOR/ + MessagePack/UBJSON this frame needs no length/indefinite bookkeeping at + all. */ - class depth_guard + struct bson_container_frame { - public: - explicit depth_guard(binary_reader& reader) noexcept : reader_(reader) - { - ++reader_.depth; - } - - ~depth_guard() - { - --reader_.depth; - } - - depth_guard(const depth_guard&) = delete; - depth_guard& operator=(const depth_guard&) = delete; - depth_guard(depth_guard&&) = delete; - depth_guard& operator=(depth_guard&&) = delete; - - private: - binary_reader& reader_; + bool is_object; ///< false: array, true: object/document }; - ////////// - // BSON // - ////////// - /*! @brief Reads in a BSON-object and passes it to the SAX-parser. @return whether a valid BSON-value was passed to the SAX parser */ bool parse_bson_internal() { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); - } + std::vector stack; - std::int32_t document_size{}; - get_number(input_format_t::bson, document_size); + // reads the 4-byte document-size prefix (its value is unused beyond + // consuming those bytes, matching the pre-existing behavior) and + // emits the corresponding SAX start_object()/start_array() event + auto begin_document = [&](const bool is_object) -> bool + { + std::int32_t document_size{}; + get_number(input_format_t::bson, document_size); + return is_object ? sax->start_object(detail::unknown_size()) : sax->start_array(detail::unknown_size()); + }; - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) + if (JSON_HEDLEY_UNLIKELY(!begin_document(true))) { return false; } + stack.push_back(bson_container_frame{true}); - if (JSON_HEDLEY_UNLIKELY(!parse_bson_element_list(/*is_array*/false))) + while (true) { - return false; - } + bson_container_frame& top = stack.back(); + + const auto element_type = get(); + if (element_type == 0) + { + // 0x00 terminates the current document/array + if (JSON_HEDLEY_UNLIKELY(!(top.is_object ? sax->end_object() : sax->end_array()))) + { + return false; + } + stack.pop_back(); + if (stack.empty()) + { + return true; + } + continue; + } + + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::bson, "element list"))) + { + return false; + } + + const std::size_t element_type_parse_position = chars_read; + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_bson_cstr(key))) + { + return false; + } + if (top.is_object && JSON_HEDLEY_UNLIKELY(!sax->key(key))) + { + return false; + } + + switch (element_type) + { + case 0x01: // double + { + double number{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, number) && sax->number_float(static_cast(number), "")))) + { + return false; + } + continue; + } + + case 0x02: // string + { + std::int32_t len{}; + string_t value; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, len) && get_bson_string(len, value) && sax->string(value)))) + { + return false; + } + continue; + } + + case 0x03: // object + { + if (JSON_HEDLEY_UNLIKELY(!begin_document(true))) + { + return false; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + } + stack.push_back(bson_container_frame{true}); + continue; + } + + case 0x04: // array + { + if (JSON_HEDLEY_UNLIKELY(!begin_document(false))) + { + return false; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + } + stack.push_back(bson_container_frame{false}); + continue; + } + + case 0x05: // binary + { + std::int32_t len{}; + binary_t value; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, len) && get_bson_binary(len, value) && sax->binary(value)))) + { + return false; + } + continue; + } - return sax->end_object(); + case 0x08: // boolean + if (JSON_HEDLEY_UNLIKELY(!sax->boolean(get() != 0))) + { + return false; + } + continue; + + case 0x0A: // null + if (JSON_HEDLEY_UNLIKELY(!sax->null())) + { + return false; + } + continue; + + case 0x10: // int32 + { + std::int32_t value{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, value) && sax->number_integer(value)))) + { + return false; + } + continue; + } + + case 0x12: // int64 + { + std::int64_t value{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, value) && sax->number_integer(value)))) + { + return false; + } + continue; + } + + case 0x11: // uint64 + { + std::uint64_t value{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, value) && sax->number_unsigned(value)))) + { + return false; + } + continue; + } + + default: // anything else is not supported (yet) + { + std::array cr{{}}; + static_cast((std::snprintf)(cr.data(), cr.size(), "%.2hhX", static_cast(element_type))); // NOLINT(cppcoreguidelines-pro-type-vararg,hicpp-vararg) + const std::string cr_str{cr.data()}; + return sax->parse_error(element_type_parse_position, cr_str, + parse_error::create(114, element_type_parse_position, concat("Unsupported BSON record type 0x", cr_str), nullptr)); + } + } + } } /*! @@ -302,158 +440,6 @@ class binary_reader return get_binary(input_format_t::bson, len, result); } - /*! - @brief Read a BSON document element of the given @a element_type. - @param[in] element_type The BSON element type, c.f. http://bsonspec.org/spec.html - @param[in] element_type_parse_position The position in the input stream, - where the `element_type` was read. - @warning Not all BSON element types are supported yet. An unsupported - @a element_type will give rise to a parse_error.114: - Unsupported BSON record type 0x... - @return whether a valid BSON-object/array was passed to the SAX parser - */ - bool parse_bson_element_internal(const char_int_type element_type, - const std::size_t element_type_parse_position) - { - switch (element_type) - { - case 0x01: // double - { - double number{}; - return get_number(input_format_t::bson, number) && sax->number_float(static_cast(number), ""); - } - - case 0x02: // string - { - std::int32_t len{}; - string_t value; - return get_number(input_format_t::bson, len) && get_bson_string(len, value) && sax->string(value); - } - - case 0x03: // object - { - return parse_bson_internal(); - } - - case 0x04: // array - { - return parse_bson_array(); - } - - case 0x05: // binary - { - std::int32_t len{}; - binary_t value; - return get_number(input_format_t::bson, len) && get_bson_binary(len, value) && sax->binary(value); - } - - case 0x08: // boolean - { - return sax->boolean(get() != 0); - } - - case 0x0A: // null - { - return sax->null(); - } - - case 0x10: // int32 - { - std::int32_t value{}; - return get_number(input_format_t::bson, value) && sax->number_integer(value); - } - - case 0x12: // int64 - { - std::int64_t value{}; - return get_number(input_format_t::bson, value) && sax->number_integer(value); - } - - case 0x11: // uint64 - { - std::uint64_t value{}; - return get_number(input_format_t::bson, value) && sax->number_unsigned(value); - } - - default: // anything else is not supported (yet) - { - std::array cr{{}}; - static_cast((std::snprintf)(cr.data(), cr.size(), "%.2hhX", static_cast(element_type))); // NOLINT(cppcoreguidelines-pro-type-vararg,hicpp-vararg) - const std::string cr_str{cr.data()}; - return sax->parse_error(element_type_parse_position, cr_str, - parse_error::create(114, element_type_parse_position, concat("Unsupported BSON record type 0x", cr_str), nullptr)); - } - } - } - - /*! - @brief Read a BSON element list (as specified in the BSON-spec) - - The same binary layout is used for objects and arrays, hence it must be - indicated with the argument @a is_array which one is expected - (true --> array, false --> object). - - @param[in] is_array Determines if the element list being read is to be - treated as an object (@a is_array == false), or as an - array (@a is_array == true). - @return whether a valid BSON-object/array was passed to the SAX parser - */ - bool parse_bson_element_list(const bool is_array) - { - string_t key; - - while (auto element_type = get()) - { - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::bson, "element list"))) - { - return false; - } - - const std::size_t element_type_parse_position = chars_read; - if (JSON_HEDLEY_UNLIKELY(!get_bson_cstr(key))) - { - return false; - } - - if (!is_array && !sax->key(key)) - { - return false; - } - - if (JSON_HEDLEY_UNLIKELY(!parse_bson_element_internal(element_type, element_type_parse_position))) - { - return false; - } - - // get_bson_cstr only appends - key.clear(); - } - - return true; - } - - /*! - @brief Reads an array from the BSON input and passes it to the SAX-parser. - @return whether a valid BSON-array was passed to the SAX parser - */ - bool parse_bson_array() - { - std::int32_t document_size{}; - get_number(input_format_t::bson, document_size); - - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) - { - return false; - } - - if (JSON_HEDLEY_UNLIKELY(!parse_bson_element_list(/*is_array*/true))) - { - return false; - } - - return sax->end_array(); - } - ////////// // CBOR // ////////// @@ -485,479 +471,658 @@ class binary_reader return sax->number_integer(static_cast(-1) - static_cast(number)); } + /*! + @brief a single pending "resume this array/object" entry used by the + iterative container traversal in @ref parse_cbor_internal + + Nested CBOR containers are parsed by pushing one of these onto a + heap-allocated std::vector instead of recursing on the native call stack + for every nesting level (see #5104): parsing a scalar never recurses, + and parsing a container just pushes a frame here and loops back around + to parse its first child; when a container finishes, its frame is + popped and parsing resumes where the parent left off. Native call-stack + depth used while parsing is therefore O(1) regardless of how deeply the + input nests. + */ + struct cbor_container_frame + { + bool is_object; ///< false: array, true: object/map + bool indefinite; ///< true: length is unknown; read until a 0xFF break marker + std::size_t remaining; ///< remaining element count; meaningful only if !indefinite + bool awaiting_key; ///< object only: true if the next thing to read is a key, not a value + }; + bool parse_cbor_internal(const bool get_char, const cbor_tag_handler_t tag_handler) { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - } + std::vector stack; - switch (get_char ? get() : current) + // whether the next value-read below should fetch a fresh byte via + // get() (true) or reuse the byte already sitting in `current` (false) + bool fetch_char = get_char; + + // called whenever a value (scalar, or a container that was just + // completed/immediately-empty) was produced at the position the + // current top-of-stack frame is waiting on; returns true if that + // means the whole parse is finished (the stack is empty, i.e. this + // was the outermost value), false if parsing should continue. + auto produce = [&]() -> bool { - // EOF - case char_traits::eof(): - return unexpect_eof(input_format_t::cbor, "value"); - - // Integer 0x00..0x17 (0..23) - case 0x00: - case 0x01: - case 0x02: - case 0x03: - case 0x04: - case 0x05: - case 0x06: - case 0x07: - case 0x08: - case 0x09: - case 0x0A: - case 0x0B: - case 0x0C: - case 0x0D: - case 0x0E: - case 0x0F: - case 0x10: - case 0x11: - case 0x12: - case 0x13: - case 0x14: - case 0x15: - case 0x16: - case 0x17: - return sax->number_unsigned(static_cast(current)); - - case 0x18: // Unsigned integer (one-byte uint8_t follows) + if (stack.empty()) { - std::uint8_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); + return true; } - - case 0x19: // Unsigned integer (two-byte uint16_t follows) + cbor_container_frame& top = stack.back(); + if (top.is_object) { - std::uint16_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); + top.awaiting_key = true; } - - case 0x1A: // Unsigned integer (four-byte uint32_t follows) + if (!top.indefinite) { - std::uint32_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); + --top.remaining; } + return false; + }; - case 0x1B: // Unsigned integer (eight-byte uint64_t follows) - { - std::uint64_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); - } - - // Negative integer -1-0x00..-1-0x17 (-1..-24) - case 0x20: - case 0x21: - case 0x22: - case 0x23: - case 0x24: - case 0x25: - case 0x26: - case 0x27: - case 0x28: - case 0x29: - case 0x2A: - case 0x2B: - case 0x2C: - case 0x2D: - case 0x2E: - case 0x2F: - case 0x30: - case 0x31: - case 0x32: - case 0x33: - case 0x34: - case 0x35: - case 0x36: - case 0x37: - return sax->number_integer(static_cast(0x20 - 1 - current)); - - case 0x38: // Negative integer (one-byte uint8_t follows) - return get_cbor_negative_integer(); - - case 0x39: // Negative integer -1-n (two-byte uint16_t follows) - return get_cbor_negative_integer(); - - case 0x3A: // Negative integer -1-n (four-byte uint32_t follows) - return get_cbor_negative_integer(); - - case 0x3B: // Negative integer -1-n (eight-byte uint64_t follows) - return get_cbor_negative_integer(); - - // Binary data (0x00..0x17 bytes follow) - case 0x40: - case 0x41: - case 0x42: - case 0x43: - case 0x44: - case 0x45: - case 0x46: - case 0x47: - case 0x48: - case 0x49: - case 0x4A: - case 0x4B: - case 0x4C: - case 0x4D: - case 0x4E: - case 0x4F: - case 0x50: - case 0x51: - case 0x52: - case 0x53: - case 0x54: - case 0x55: - case 0x56: - case 0x57: - case 0x58: // Binary data (one-byte uint8_t for n follows) - case 0x59: // Binary data (two-byte uint16_t for n follow) - case 0x5A: // Binary data (four-byte uint32_t for n follow) - case 0x5B: // Binary data (eight-byte uint64_t for n follow) - case 0x5F: // Binary data (indefinite length) + // begin parsing a nested array/object of the given length + // (detail::unknown_size() for indefinite length); returns 0 on SAX + // failure or exceeding max_depth (caller should `return false`), 1 + // if a frame was pushed (caller should `continue` the outer loop to + // parse the container's first element), 2 if the container was + // immediately completed because it is empty (caller should treat + // this like any other produced value via `produce()`). + auto enter_container = [&](const bool is_object, const std::size_t len) -> int + { + const bool started = is_object ? sax->start_object(len) : sax->start_array(len); + if (JSON_HEDLEY_UNLIKELY(!started)) { - binary_t b; - return get_cbor_binary(b) && sax->binary(b); + return 0; } - - // UTF-8 string (0x00..0x17 bytes follow) - case 0x60: - case 0x61: - case 0x62: - case 0x63: - case 0x64: - case 0x65: - case 0x66: - case 0x67: - case 0x68: - case 0x69: - case 0x6A: - case 0x6B: - case 0x6C: - case 0x6D: - case 0x6E: - case 0x6F: - case 0x70: - case 0x71: - case 0x72: - case 0x73: - case 0x74: - case 0x75: - case 0x76: - case 0x77: - case 0x78: // UTF-8 string (one-byte uint8_t for n follows) - case 0x79: // UTF-8 string (two-byte uint16_t for n follow) - case 0x7A: // UTF-8 string (four-byte uint32_t for n follow) - case 0x7B: // UTF-8 string (eight-byte uint64_t for n follow) - case 0x7F: // UTF-8 string (indefinite length) + if (len == 0) { - string_t s; - return get_cbor_string(s) && sax->string(s); - } - - // array (0x00..0x17 data items follow) - case 0x80: - case 0x81: - case 0x82: - case 0x83: - case 0x84: - case 0x85: - case 0x86: - case 0x87: - case 0x88: - case 0x89: - case 0x8A: - case 0x8B: - case 0x8C: - case 0x8D: - case 0x8E: - case 0x8F: - case 0x90: - case 0x91: - case 0x92: - case 0x93: - case 0x94: - case 0x95: - case 0x96: - case 0x97: - return get_cbor_array( - conditional_static_cast(static_cast(current) & 0x1Fu), tag_handler); - - case 0x98: // array (one-byte uint8_t for n follows) - { - std::uint8_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(static_cast(len), tag_handler); + return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; } - - case 0x99: // array (two-byte uint16_t for n follow) + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - std::uint16_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(static_cast(len), tag_handler); + sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; } + stack.push_back(cbor_container_frame{is_object, len == detail::unknown_size(), len, is_object}); + return 1; + }; - case 0x9A: // array (four-byte uint32_t for n follow) - { - std::uint32_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(conditional_static_cast(len), tag_handler); - } +#define JSON_CBOR_VALUE(expr) \ + if (JSON_HEDLEY_UNLIKELY(!(expr))) { return false; } \ + if (produce()) { return true; } \ + continue + +#define JSON_CBOR_ENTER(is_obj, length) \ + switch (enter_container(is_obj, length)) \ + { \ + case 0: return false; \ + case 2: if (produce()) { return true; } continue; \ + default: continue; \ + } - case 0x9B: // array (eight-byte uint64_t for n follow) + while (true) + { + if (!stack.empty()) { - std::uint64_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(conditional_static_cast(len), tag_handler); - } + cbor_container_frame& top = stack.back(); - case 0x9F: // array (indefinite length) - return get_cbor_array(detail::unknown_size(), tag_handler); + if (top.is_object && top.awaiting_key) + { + const bool finished = top.indefinite ? (get() == 0xFF) : (top.remaining == 0); + if (finished) + { + if (JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; + } - // map (0x00..0x17 pairs of data items follow) - case 0xA0: - case 0xA1: - case 0xA2: - case 0xA3: - case 0xA4: - case 0xA5: - case 0xA6: - case 0xA7: - case 0xA8: - case 0xA9: - case 0xAA: - case 0xAB: - case 0xAC: - case 0xAD: - case 0xAE: - case 0xAF: - case 0xB0: - case 0xB1: - case 0xB2: - case 0xB3: - case 0xB4: - case 0xB5: - case 0xB6: - case 0xB7: - return get_cbor_object(conditional_static_cast(static_cast(current) & 0x1Fu), tag_handler); + if (!top.indefinite) + { + get(); + } + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) + { + return false; + } + top.awaiting_key = false; + fetch_char = true; + } + else if (!top.is_object) + { + const bool finished = top.indefinite ? (get() == 0xFF) : (top.remaining == 0); + if (finished) + { + if (JSON_HEDLEY_UNLIKELY(!sax->end_array())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; + } + fetch_char = !top.indefinite; + } + // else: object frame with a key already read; fetch_char is + // already set to true from when the key was read above. + } + +read_value: + switch (fetch_char ? get() : current) + { + // EOF + case char_traits::eof(): + return unexpect_eof(input_format_t::cbor, "value"); + + // Integer 0x00..0x17 (0..23) + case 0x00: + case 0x01: + case 0x02: + case 0x03: + case 0x04: + case 0x05: + case 0x06: + case 0x07: + case 0x08: + case 0x09: + case 0x0A: + case 0x0B: + case 0x0C: + case 0x0D: + case 0x0E: + case 0x0F: + case 0x10: + case 0x11: + case 0x12: + case 0x13: + case 0x14: + case 0x15: + case 0x16: + case 0x17: + JSON_CBOR_VALUE(sax->number_unsigned(static_cast(current))); + + case 0x18: // Unsigned integer (one-byte uint8_t follows) + { + std::uint8_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); + } - case 0xB8: // map (one-byte uint8_t for n follows) - { - std::uint8_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(static_cast(len), tag_handler); - } + case 0x19: // Unsigned integer (two-byte uint16_t follows) + { + std::uint16_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); + } - case 0xB9: // map (two-byte uint16_t for n follow) - { - std::uint16_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(static_cast(len), tag_handler); - } + case 0x1A: // Unsigned integer (four-byte uint32_t follows) + { + std::uint32_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); + } - case 0xBA: // map (four-byte uint32_t for n follow) - { - std::uint32_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(conditional_static_cast(len), tag_handler); - } + case 0x1B: // Unsigned integer (eight-byte uint64_t follows) + { + std::uint64_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); + } - case 0xBB: // map (eight-byte uint64_t for n follow) - { - std::uint64_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(conditional_static_cast(len), tag_handler); - } - - case 0xBF: // map (indefinite length) - return get_cbor_object(detail::unknown_size(), tag_handler); - - case 0xC6: // tagged item - case 0xC7: - case 0xC8: - case 0xC9: - case 0xCA: - case 0xCB: - case 0xCC: - case 0xCD: - case 0xCE: - case 0xCF: - case 0xD0: - case 0xD1: - case 0xD2: - case 0xD3: - case 0xD4: - case 0xD8: // tagged item (1 byte follows) - case 0xD9: // tagged item (2 bytes follow) - case 0xDA: // tagged item (4 bytes follow) - case 0xDB: // tagged item (8 bytes follow) - { - switch (tag_handler) - { - case cbor_tag_handler_t::error: + // Negative integer -1-0x00..-1-0x17 (-1..-24) + case 0x20: + case 0x21: + case 0x22: + case 0x23: + case 0x24: + case 0x25: + case 0x26: + case 0x27: + case 0x28: + case 0x29: + case 0x2A: + case 0x2B: + case 0x2C: + case 0x2D: + case 0x2E: + case 0x2F: + case 0x30: + case 0x31: + case 0x32: + case 0x33: + case 0x34: + case 0x35: + case 0x36: + case 0x37: + JSON_CBOR_VALUE(sax->number_integer(static_cast(0x20 - 1 - current))); + + case 0x38: // Negative integer (one-byte uint8_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); + + case 0x39: // Negative integer -1-n (two-byte uint16_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); + + case 0x3A: // Negative integer -1-n (four-byte uint32_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); + + case 0x3B: // Negative integer -1-n (eight-byte uint64_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); + + // Binary data (0x00..0x17 bytes follow) + case 0x40: + case 0x41: + case 0x42: + case 0x43: + case 0x44: + case 0x45: + case 0x46: + case 0x47: + case 0x48: + case 0x49: + case 0x4A: + case 0x4B: + case 0x4C: + case 0x4D: + case 0x4E: + case 0x4F: + case 0x50: + case 0x51: + case 0x52: + case 0x53: + case 0x54: + case 0x55: + case 0x56: + case 0x57: + case 0x58: // Binary data (one-byte uint8_t for n follows) + case 0x59: // Binary data (two-byte uint16_t for n follow) + case 0x5A: // Binary data (four-byte uint32_t for n follow) + case 0x5B: // Binary data (eight-byte uint64_t for n follow) + case 0x5F: // Binary data (indefinite length) + { + binary_t b; + JSON_CBOR_VALUE(get_cbor_binary(b) && sax->binary(b)); + } + + // UTF-8 string (0x00..0x17 bytes follow) + case 0x60: + case 0x61: + case 0x62: + case 0x63: + case 0x64: + case 0x65: + case 0x66: + case 0x67: + case 0x68: + case 0x69: + case 0x6A: + case 0x6B: + case 0x6C: + case 0x6D: + case 0x6E: + case 0x6F: + case 0x70: + case 0x71: + case 0x72: + case 0x73: + case 0x74: + case 0x75: + case 0x76: + case 0x77: + case 0x78: // UTF-8 string (one-byte uint8_t for n follows) + case 0x79: // UTF-8 string (two-byte uint16_t for n follow) + case 0x7A: // UTF-8 string (four-byte uint32_t for n follow) + case 0x7B: // UTF-8 string (eight-byte uint64_t for n follow) + case 0x7F: // UTF-8 string (indefinite length) + { + string_t s; + JSON_CBOR_VALUE(get_cbor_string(s) && sax->string(s)); + } + + // array (0x00..0x17 data items follow) + case 0x80: + case 0x81: + case 0x82: + case 0x83: + case 0x84: + case 0x85: + case 0x86: + case 0x87: + case 0x88: + case 0x89: + case 0x8A: + case 0x8B: + case 0x8C: + case 0x8D: + case 0x8E: + case 0x8F: + case 0x90: + case 0x91: + case 0x92: + case 0x93: + case 0x94: + case 0x95: + case 0x96: + case 0x97: + JSON_CBOR_ENTER(false, conditional_static_cast(static_cast(current) & 0x1Fu)); + + case 0x98: // array (one-byte uint8_t for n follows) + { + std::uint8_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); + return false; } + JSON_CBOR_ENTER(false, static_cast(len)); + } - case cbor_tag_handler_t::ignore: + case 0x99: // array (two-byte uint16_t for n follow) + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) { - // ignore binary subtype - switch (current) - { - case 0xD8: - { - std::uint8_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - case 0xD9: - { - std::uint16_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - case 0xDA: - { - std::uint32_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - case 0xDB: - { - std::uint64_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - default: - break; - } - return parse_cbor_internal(true, tag_handler); + return false; } + JSON_CBOR_ENTER(false, static_cast(len)); + } - case cbor_tag_handler_t::store: + case 0x9A: // array (four-byte uint32_t for n follow) + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) { - binary_t b; - // use binary subtype and store in a binary container - switch (current) + return false; + } + JSON_CBOR_ENTER(false, conditional_static_cast(len)); + } + + case 0x9B: // array (eight-byte uint64_t for n follow) + { + std::uint64_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(false, conditional_static_cast(len)); + } + + case 0x9F: // array (indefinite length) + JSON_CBOR_ENTER(false, detail::unknown_size()); + + // map (0x00..0x17 pairs of data items follow) + case 0xA0: + case 0xA1: + case 0xA2: + case 0xA3: + case 0xA4: + case 0xA5: + case 0xA6: + case 0xA7: + case 0xA8: + case 0xA9: + case 0xAA: + case 0xAB: + case 0xAC: + case 0xAD: + case 0xAE: + case 0xAF: + case 0xB0: + case 0xB1: + case 0xB2: + case 0xB3: + case 0xB4: + case 0xB5: + case 0xB6: + case 0xB7: + JSON_CBOR_ENTER(true, conditional_static_cast(static_cast(current) & 0x1Fu)); + + case 0xB8: // map (one-byte uint8_t for n follows) + { + std::uint8_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, static_cast(len)); + } + + case 0xB9: // map (two-byte uint16_t for n follow) + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, static_cast(len)); + } + + case 0xBA: // map (four-byte uint32_t for n follow) + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, conditional_static_cast(len)); + } + + case 0xBB: // map (eight-byte uint64_t for n follow) + { + std::uint64_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, conditional_static_cast(len)); + } + + case 0xBF: // map (indefinite length) + JSON_CBOR_ENTER(true, detail::unknown_size()); + + case 0xC6: // tagged item + case 0xC7: + case 0xC8: + case 0xC9: + case 0xCA: + case 0xCB: + case 0xCC: + case 0xCD: + case 0xCE: + case 0xCF: + case 0xD0: + case 0xD1: + case 0xD2: + case 0xD3: + case 0xD4: + case 0xD8: // tagged item (1 byte follows) + case 0xD9: // tagged item (2 bytes follow) + case 0xDA: // tagged item (4 bytes follow) + case 0xDB: // tagged item (8 bytes follow) + { + switch (tag_handler) + { + case cbor_tag_handler_t::error: { - case 0xD8: - { - std::uint8_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; - } - case 0xD9: - { - std::uint16_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; - } - case 0xDA: + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); + } + + case cbor_tag_handler_t::ignore: + { + // ignore binary subtype + switch (current) { - std::uint32_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; + case 0xD8: + { + std::uint8_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + case 0xD9: + { + std::uint16_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + case 0xDA: + { + std::uint32_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + case 0xDB: + { + std::uint64_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + default: + break; } - case 0xDB: + fetch_char = true; + goto read_value; + } + + case cbor_tag_handler_t::store: + { + binary_t b; + // use binary subtype and store in a binary container + switch (current) { - std::uint64_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; + case 0xD8: + { + std::uint8_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + case 0xD9: + { + std::uint16_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + case 0xDA: + { + std::uint32_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + case 0xDB: + { + std::uint64_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + default: + fetch_char = true; + goto read_value; } - default: - return parse_cbor_internal(true, tag_handler); + get(); + JSON_CBOR_VALUE(get_cbor_binary(b) && sax->binary(b)); } - get(); - return get_cbor_binary(b) && sax->binary(b); - } - default: // LCOV_EXCL_LINE - JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE - return false; // LCOV_EXCL_LINE + default: // LCOV_EXCL_LINE + JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE + return false; // LCOV_EXCL_LINE + } } - } - case 0xF4: // false - return sax->boolean(false); + case 0xF4: // false + JSON_CBOR_VALUE(sax->boolean(false)); - case 0xF5: // true - return sax->boolean(true); + case 0xF5: // true + JSON_CBOR_VALUE(sax->boolean(true)); - case 0xF6: // null - return sax->null(); + case 0xF6: // null + JSON_CBOR_VALUE(sax->null()); - case 0xF9: // Half-Precision Float (two-byte IEEE 754) - { - const auto byte1_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) + case 0xF9: // Half-Precision Float (two-byte IEEE 754) { - return false; - } - const auto byte2_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) - { - return false; - } - - const auto byte1 = static_cast(byte1_raw); - const auto byte2 = static_cast(byte2_raw); - - // Code from RFC 7049, Appendix D, Figure 3: - // As half-precision floating-point numbers were only added - // to IEEE 754 in 2008, today's programming platforms often - // still only have limited support for them. It is very - // easy to include at least decoding support for them even - // without such support. An example of a small decoder for - // half-precision floating-point numbers in the C language - // is shown in Fig. 3. - const auto half = static_cast((byte1 << 8u) + byte2); - const double val = [&half] - { - const int exp = (half >> 10u) & 0x1Fu; - const unsigned int mant = half & 0x3FFu; - JSON_ASSERT(0 <= exp&& exp <= 32); - JSON_ASSERT(mant <= 1024); - switch (exp) + const auto byte1_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) { - case 0: - return std::ldexp(mant, -24); - case 31: - return (mant == 0) - ? std::numeric_limits::infinity() - : std::numeric_limits::quiet_NaN(); - default: - return std::ldexp(mant + 1024, exp - 25); + return false; + } + const auto byte2_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) + { + return false; } - }(); - return sax->number_float((half & 0x8000u) != 0 - ? static_cast(-val) - : static_cast(val), ""); - } - case 0xFA: // Single-Precision Float (four-byte IEEE 754) - { - float number{}; - return get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), ""); - } + const auto byte1 = static_cast(byte1_raw); + const auto byte2 = static_cast(byte2_raw); + + // Code from RFC 7049, Appendix D, Figure 3: + // As half-precision floating-point numbers were only added + // to IEEE 754 in 2008, today's programming platforms often + // still only have limited support for them. It is very + // easy to include at least decoding support for them even + // without such support. An example of a small decoder for + // half-precision floating-point numbers in the C language + // is shown in Fig. 3. + const auto half = static_cast((byte1 << 8u) + byte2); + const double val = [&half] + { + const int exp = (half >> 10u) & 0x1Fu; + const unsigned int mant = half & 0x3FFu; + JSON_ASSERT(0 <= exp&& exp <= 32); + JSON_ASSERT(mant <= 1024); + switch (exp) + { + case 0: + return std::ldexp(mant, -24); + case 31: + return (mant == 0) + ? std::numeric_limits::infinity() + : std::numeric_limits::quiet_NaN(); + default: + return std::ldexp(mant + 1024, exp - 25); + } + }(); + JSON_CBOR_VALUE(sax->number_float((half & 0x8000u) != 0 + ? static_cast(-val) + : static_cast(val), "")); + } - case 0xFB: // Double-Precision Float (eight-byte IEEE 754) - { - double number{}; - return get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), ""); - } + case 0xFA: // Single-Precision Float (four-byte IEEE 754) + { + float number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), "")); + } - default: // anything else (0xFF is handled inside the other types) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); + case 0xFB: // Double-Precision Float (eight-byte IEEE 754) + { + double number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), "")); + } + + default: // anything else (0xFF is handled inside the other types) + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); + } } } + +#undef JSON_CBOR_VALUE +#undef JSON_CBOR_ENTER } /*! @@ -1156,483 +1321,517 @@ class binary_reader } } + ///////////// + // MsgPack // + ///////////// + /*! - @param[in] len the length of the array or detail::unknown_size() for an - array of indefinite size - @param[in] tag_handler how CBOR tags should be treated - @return whether array creation completed + @brief a single pending "resume this array/object" entry used by the + iterative container traversal in @ref parse_msgpack_internal; + see @ref cbor_container_frame for the general idea. MessagePack + has no indefinite-length containers, so unlike CBOR this frame + does not need an "indefinite" flag. */ - bool get_cbor_array(const std::size_t len, - const cbor_tag_handler_t tag_handler) + struct msgpack_container_frame { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(len))) + bool is_object; ///< false: array, true: map + std::size_t remaining; ///< remaining element count + bool awaiting_key; ///< map only: true if the next thing to read is a key, not a value + }; + + /*! + @return whether a valid MessagePack value was passed to the SAX parser + */ + bool parse_msgpack_internal() + { + std::vector stack; + + auto produce = [&]() -> bool { + if (stack.empty()) + { + return true; + } + msgpack_container_frame& top = stack.back(); + if (top.is_object) + { + top.awaiting_key = true; + } + --top.remaining; return false; - } + }; - if (len != detail::unknown_size()) + auto enter_container = [&](const bool is_object, const std::size_t len) -> int { - for (std::size_t i = 0; i < len; ++i) + const bool started = is_object ? sax->start_object(len) : sax->start_array(len); + if (JSON_HEDLEY_UNLIKELY(!started)) { - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(true, tag_handler))) - { - return false; - } + return 0; } - } - else - { - while (get() != 0xFF) + if (len == 0) { - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(false, tag_handler))) - { - return false; - } + return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; } - } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(msgpack_container_frame{is_object, len, is_object}); + return 1; + }; - return sax->end_array(); +#define JSON_MSGPACK_VALUE(expr) \ + if (JSON_HEDLEY_UNLIKELY(!(expr))) { return false; } \ + if (produce()) { return true; } \ + continue + +#define JSON_MSGPACK_ENTER(is_obj, length) \ + switch (enter_container(is_obj, length)) \ + { \ + case 0: return false; \ + case 2: if (produce()) { return true; } continue; \ + default: continue; \ } - /*! - @param[in] len the length of the object or detail::unknown_size() for an - object of indefinite size - @param[in] tag_handler how CBOR tags should be treated - @return whether object creation completed - */ - bool get_cbor_object(const std::size_t len, - const cbor_tag_handler_t tag_handler) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(len))) - { - return false; - } - - if (len != 0) + while (true) { - string_t key; - if (len != detail::unknown_size()) + if (!stack.empty()) { - for (std::size_t i = 0; i < len; ++i) + msgpack_container_frame& top = stack.back(); + + if (top.is_object && top.awaiting_key) { - get(); - if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) + if (top.remaining == 0) { - return false; + if (JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; } - - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(true, tag_handler))) + get(); + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key))) { return false; } - key.clear(); + top.awaiting_key = false; } - } - else - { - while (get() != 0xFF) + else if (!top.is_object) { - if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) - { - return false; - } - - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(true, tag_handler))) + if (top.remaining == 0) { - return false; + if (JSON_HEDLEY_UNLIKELY(!sax->end_array())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; } - key.clear(); } - } - } - - return sax->end_object(); - } - - ///////////// - // MsgPack // - ///////////// - - /*! - @return whether a valid MessagePack value was passed to the SAX parser - */ - bool parse_msgpack_internal() - { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - } - - switch (get()) - { - // EOF - case char_traits::eof(): - return unexpect_eof(input_format_t::msgpack, "value"); - - // positive fixint - case 0x00: - case 0x01: - case 0x02: - case 0x03: - case 0x04: - case 0x05: - case 0x06: - case 0x07: - case 0x08: - case 0x09: - case 0x0A: - case 0x0B: - case 0x0C: - case 0x0D: - case 0x0E: - case 0x0F: - case 0x10: - case 0x11: - case 0x12: - case 0x13: - case 0x14: - case 0x15: - case 0x16: - case 0x17: - case 0x18: - case 0x19: - case 0x1A: - case 0x1B: - case 0x1C: - case 0x1D: - case 0x1E: - case 0x1F: - case 0x20: - case 0x21: - case 0x22: - case 0x23: - case 0x24: - case 0x25: - case 0x26: - case 0x27: - case 0x28: - case 0x29: - case 0x2A: - case 0x2B: - case 0x2C: - case 0x2D: - case 0x2E: - case 0x2F: - case 0x30: - case 0x31: - case 0x32: - case 0x33: - case 0x34: - case 0x35: - case 0x36: - case 0x37: - case 0x38: - case 0x39: - case 0x3A: - case 0x3B: - case 0x3C: - case 0x3D: - case 0x3E: - case 0x3F: - case 0x40: - case 0x41: - case 0x42: - case 0x43: - case 0x44: - case 0x45: - case 0x46: - case 0x47: - case 0x48: - case 0x49: - case 0x4A: - case 0x4B: - case 0x4C: - case 0x4D: - case 0x4E: - case 0x4F: - case 0x50: - case 0x51: - case 0x52: - case 0x53: - case 0x54: - case 0x55: - case 0x56: - case 0x57: - case 0x58: - case 0x59: - case 0x5A: - case 0x5B: - case 0x5C: - case 0x5D: - case 0x5E: - case 0x5F: - case 0x60: - case 0x61: - case 0x62: - case 0x63: - case 0x64: - case 0x65: - case 0x66: - case 0x67: - case 0x68: - case 0x69: - case 0x6A: - case 0x6B: - case 0x6C: - case 0x6D: - case 0x6E: - case 0x6F: - case 0x70: - case 0x71: - case 0x72: - case 0x73: - case 0x74: - case 0x75: - case 0x76: - case 0x77: - case 0x78: - case 0x79: - case 0x7A: - case 0x7B: - case 0x7C: - case 0x7D: - case 0x7E: - case 0x7F: - return sax->number_unsigned(static_cast(current)); - - // fixmap - case 0x80: - case 0x81: - case 0x82: - case 0x83: - case 0x84: - case 0x85: - case 0x86: - case 0x87: - case 0x88: - case 0x89: - case 0x8A: - case 0x8B: - case 0x8C: - case 0x8D: - case 0x8E: - case 0x8F: - return get_msgpack_object(conditional_static_cast(static_cast(current) & 0x0Fu)); - - // fixarray - case 0x90: - case 0x91: - case 0x92: - case 0x93: - case 0x94: - case 0x95: - case 0x96: - case 0x97: - case 0x98: - case 0x99: - case 0x9A: - case 0x9B: - case 0x9C: - case 0x9D: - case 0x9E: - case 0x9F: - return get_msgpack_array(conditional_static_cast(static_cast(current) & 0x0Fu)); - - // fixstr - case 0xA0: - case 0xA1: - case 0xA2: - case 0xA3: - case 0xA4: - case 0xA5: - case 0xA6: - case 0xA7: - case 0xA8: - case 0xA9: - case 0xAA: - case 0xAB: - case 0xAC: - case 0xAD: - case 0xAE: - case 0xAF: - case 0xB0: - case 0xB1: - case 0xB2: - case 0xB3: - case 0xB4: - case 0xB5: - case 0xB6: - case 0xB7: - case 0xB8: - case 0xB9: - case 0xBA: - case 0xBB: - case 0xBC: - case 0xBD: - case 0xBE: - case 0xBF: - case 0xD9: // str 8 - case 0xDA: // str 16 - case 0xDB: // str 32 - { - string_t s; - return get_msgpack_string(s) && sax->string(s); - } - - case 0xC0: // nil - return sax->null(); - - case 0xC2: // false - return sax->boolean(false); + // else: map frame with a key already read; fall through to + // read that key's value. + } + + switch (get()) + { + // EOF + case char_traits::eof(): + return unexpect_eof(input_format_t::msgpack, "value"); + + // positive fixint + case 0x00: + case 0x01: + case 0x02: + case 0x03: + case 0x04: + case 0x05: + case 0x06: + case 0x07: + case 0x08: + case 0x09: + case 0x0A: + case 0x0B: + case 0x0C: + case 0x0D: + case 0x0E: + case 0x0F: + case 0x10: + case 0x11: + case 0x12: + case 0x13: + case 0x14: + case 0x15: + case 0x16: + case 0x17: + case 0x18: + case 0x19: + case 0x1A: + case 0x1B: + case 0x1C: + case 0x1D: + case 0x1E: + case 0x1F: + case 0x20: + case 0x21: + case 0x22: + case 0x23: + case 0x24: + case 0x25: + case 0x26: + case 0x27: + case 0x28: + case 0x29: + case 0x2A: + case 0x2B: + case 0x2C: + case 0x2D: + case 0x2E: + case 0x2F: + case 0x30: + case 0x31: + case 0x32: + case 0x33: + case 0x34: + case 0x35: + case 0x36: + case 0x37: + case 0x38: + case 0x39: + case 0x3A: + case 0x3B: + case 0x3C: + case 0x3D: + case 0x3E: + case 0x3F: + case 0x40: + case 0x41: + case 0x42: + case 0x43: + case 0x44: + case 0x45: + case 0x46: + case 0x47: + case 0x48: + case 0x49: + case 0x4A: + case 0x4B: + case 0x4C: + case 0x4D: + case 0x4E: + case 0x4F: + case 0x50: + case 0x51: + case 0x52: + case 0x53: + case 0x54: + case 0x55: + case 0x56: + case 0x57: + case 0x58: + case 0x59: + case 0x5A: + case 0x5B: + case 0x5C: + case 0x5D: + case 0x5E: + case 0x5F: + case 0x60: + case 0x61: + case 0x62: + case 0x63: + case 0x64: + case 0x65: + case 0x66: + case 0x67: + case 0x68: + case 0x69: + case 0x6A: + case 0x6B: + case 0x6C: + case 0x6D: + case 0x6E: + case 0x6F: + case 0x70: + case 0x71: + case 0x72: + case 0x73: + case 0x74: + case 0x75: + case 0x76: + case 0x77: + case 0x78: + case 0x79: + case 0x7A: + case 0x7B: + case 0x7C: + case 0x7D: + case 0x7E: + case 0x7F: + JSON_MSGPACK_VALUE(sax->number_unsigned(static_cast(current))); + + // fixmap + case 0x80: + case 0x81: + case 0x82: + case 0x83: + case 0x84: + case 0x85: + case 0x86: + case 0x87: + case 0x88: + case 0x89: + case 0x8A: + case 0x8B: + case 0x8C: + case 0x8D: + case 0x8E: + case 0x8F: + JSON_MSGPACK_ENTER(true, conditional_static_cast(static_cast(current) & 0x0Fu)); + + // fixarray + case 0x90: + case 0x91: + case 0x92: + case 0x93: + case 0x94: + case 0x95: + case 0x96: + case 0x97: + case 0x98: + case 0x99: + case 0x9A: + case 0x9B: + case 0x9C: + case 0x9D: + case 0x9E: + case 0x9F: + JSON_MSGPACK_ENTER(false, conditional_static_cast(static_cast(current) & 0x0Fu)); + + // fixstr + case 0xA0: + case 0xA1: + case 0xA2: + case 0xA3: + case 0xA4: + case 0xA5: + case 0xA6: + case 0xA7: + case 0xA8: + case 0xA9: + case 0xAA: + case 0xAB: + case 0xAC: + case 0xAD: + case 0xAE: + case 0xAF: + case 0xB0: + case 0xB1: + case 0xB2: + case 0xB3: + case 0xB4: + case 0xB5: + case 0xB6: + case 0xB7: + case 0xB8: + case 0xB9: + case 0xBA: + case 0xBB: + case 0xBC: + case 0xBD: + case 0xBE: + case 0xBF: + case 0xD9: // str 8 + case 0xDA: // str 16 + case 0xDB: // str 32 + { + string_t s; + JSON_MSGPACK_VALUE(get_msgpack_string(s) && sax->string(s)); + } - case 0xC3: // true - return sax->boolean(true); + case 0xC0: // nil + JSON_MSGPACK_VALUE(sax->null()); + + case 0xC2: // false + JSON_MSGPACK_VALUE(sax->boolean(false)); + + case 0xC3: // true + JSON_MSGPACK_VALUE(sax->boolean(true)); + + case 0xC4: // bin 8 + case 0xC5: // bin 16 + case 0xC6: // bin 32 + case 0xC7: // ext 8 + case 0xC8: // ext 16 + case 0xC9: // ext 32 + case 0xD4: // fixext 1 + case 0xD5: // fixext 2 + case 0xD6: // fixext 4 + case 0xD7: // fixext 8 + case 0xD8: // fixext 16 + { + binary_t b; + JSON_MSGPACK_VALUE(get_msgpack_binary(b) && sax->binary(b)); + } - case 0xC4: // bin 8 - case 0xC5: // bin 16 - case 0xC6: // bin 32 - case 0xC7: // ext 8 - case 0xC8: // ext 16 - case 0xC9: // ext 32 - case 0xD4: // fixext 1 - case 0xD5: // fixext 2 - case 0xD6: // fixext 4 - case 0xD7: // fixext 8 - case 0xD8: // fixext 16 - { - binary_t b; - return get_msgpack_binary(b) && sax->binary(b); - } + case 0xCA: // float 32 + { + float number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), "")); + } - case 0xCA: // float 32 - { - float number{}; - return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), ""); - } + case 0xCB: // float 64 + { + double number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), "")); + } - case 0xCB: // float 64 - { - double number{}; - return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), ""); - } + case 0xCC: // uint 8 + { + std::uint8_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCC: // uint 8 - { - std::uint8_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCD: // uint 16 + { + std::uint16_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCD: // uint 16 - { - std::uint16_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCE: // uint 32 + { + std::uint32_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCE: // uint 32 - { - std::uint32_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCF: // uint 64 + { + std::uint64_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCF: // uint 64 - { - std::uint64_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xD0: // int 8 + { + std::int8_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD0: // int 8 - { - std::int8_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD1: // int 16 + { + std::int16_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD1: // int 16 - { - std::int16_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD2: // int 32 + { + std::int32_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD2: // int 32 - { - std::int32_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD3: // int 64 + { + std::int64_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD3: // int 64 - { - std::int64_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xDC: // array 16 + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(false, static_cast(len)); + } - case 0xDC: // array 16 - { - std::uint16_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_array(static_cast(len)); - } + case 0xDD: // array 32 + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(false, conditional_static_cast(len)); + } - case 0xDD: // array 32 - { - std::uint32_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_array(conditional_static_cast(len)); - } + case 0xDE: // map 16 + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(true, static_cast(len)); + } - case 0xDE: // map 16 - { - std::uint16_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_object(static_cast(len)); - } + case 0xDF: // map 32 + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(true, conditional_static_cast(len)); + } - case 0xDF: // map 32 - { - std::uint32_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_object(conditional_static_cast(len)); - } - - // negative fixint - case 0xE0: - case 0xE1: - case 0xE2: - case 0xE3: - case 0xE4: - case 0xE5: - case 0xE6: - case 0xE7: - case 0xE8: - case 0xE9: - case 0xEA: - case 0xEB: - case 0xEC: - case 0xED: - case 0xEE: - case 0xEF: - case 0xF0: - case 0xF1: - case 0xF2: - case 0xF3: - case 0xF4: - case 0xF5: - case 0xF6: - case 0xF7: - case 0xF8: - case 0xF9: - case 0xFA: - case 0xFB: - case 0xFC: - case 0xFD: - case 0xFE: - case 0xFF: - return sax->number_integer(static_cast(current)); - - default: // anything else - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format_t::msgpack, concat("invalid byte: 0x", last_token), "value"), nullptr)); + // negative fixint + case 0xE0: + case 0xE1: + case 0xE2: + case 0xE3: + case 0xE4: + case 0xE5: + case 0xE6: + case 0xE7: + case 0xE8: + case 0xE9: + case 0xEA: + case 0xEB: + case 0xEC: + case 0xED: + case 0xEE: + case 0xEF: + case 0xF0: + case 0xF1: + case 0xF2: + case 0xF3: + case 0xF4: + case 0xF5: + case 0xF6: + case 0xF7: + case 0xF8: + case 0xF9: + case 0xFA: + case 0xFB: + case 0xFC: + case 0xFD: + case 0xFE: + case 0xFF: + JSON_MSGPACK_VALUE(sax->number_integer(static_cast(current))); + + default: // anything else + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format_t::msgpack, concat("invalid byte: 0x", last_token), "value"), nullptr)); + } } } + +#undef JSON_MSGPACK_VALUE +#undef JSON_MSGPACK_ENTER } /*! @@ -1835,87 +2034,525 @@ class binary_reader } } + //////////// + // UBJSON // + //////////// + + /*! + @brief a single pending "resume this array/object" entry used by the + iterative container traversal in @ref parse_ubjson_internal; see + @ref cbor_container_frame for the general idea. + + UBJSON/BJData needs two extra bits of state beyond CBOR/MessagePack: + @ref fixed_type captures the optimized `$type#count` container's + homogeneous element type (0 means "heterogeneous": read a fresh type + marker per element, like an ordinary UBJSON array/object); @ref + extra_close_object is set for the synthetic JData-annotated-array object + ("_ArrayType_"/"_ArraySize_"/"_ArrayData_") BJData wraps an ND-array in, + where finishing the `_ArrayData_` array must also close that wrapping + object. + */ + struct ubjson_container_frame + { + bool is_object; ///< false: array, true: object + bool indefinite; ///< true: no known count; read until a ']'/'}' terminator + std::size_t remaining; ///< remaining element count; meaningful only if !indefinite + bool awaiting_key; ///< object only: true if the next thing to read is a key, not a value + char_int_type fixed_type; ///< 0: heterogeneous; else every element has this known type marker + bool extra_close_object; ///< true: end_array() for this frame must be followed by an end_object() + }; + /*! - @param[in] len the length of the array - @return whether array creation completed + @param[in] get_char whether a new character should be retrieved from the + input (true, default) or whether the last read + character should be considered instead + + @return whether a valid UBJSON value was passed to the SAX parser */ - bool get_msgpack_array(const std::size_t len) + bool parse_ubjson_internal(const bool get_char = true) { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(len))) + std::vector stack; + bool fetch_char = get_char; + + auto produce = [&]() -> bool { + if (stack.empty()) + { + return true; + } + ubjson_container_frame& top = stack.back(); + if (top.is_object) + { + top.awaiting_key = true; + } + if (!top.indefinite) + { + --top.remaining; + } + else + { + // indefinite-length containers are terminated by inspecting + // `current`, so advance past the just-produced element/pair + // to whatever byte should be inspected next (mirrors the + // `get_ignore_noop()` at the end of each iteration of the + // original recursive implementation's while-loops) + get_ignore_noop(); + } return false; - } + }; - for (std::size_t i = 0; i < len; ++i) + // returns 0 = failure (caller returns false), 1 = a frame was + // pushed (caller continues the outer loop to parse the first + // element), 2 = a value was produced immediately - either because + // the container turned out to be empty, or (BJData only) because + // the "optimized binary array" special case produced a single + // binary value directly, without ever being a SAX array at all + // (caller should invoke produce() same as for any other value) + auto enter_array = [&]() -> int { - if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_internal())) + std::pair size_and_type; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) { - return false; + return 0; } - } - return sax->end_array(); - } + // ND-array: encode as an object in JData annotated array format + // (https://github.com/NeuroJSON/jdata). The wrapping object was + // already start_object()'d (with the "_ArraySize_" key/array + // already emitted) inside get_ubjson_size_type(), so only + // "_ArrayType_" and "_ArrayData_" remain to be added here. + if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) + { + size_and_type.second &= ~(static_cast(1) << 8); + auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), size_and_type.second, [](const bjd_type & p, char_int_type t) + { + return p.first < t; + }); + string_t key = "_ArrayType_"; + if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != size_and_type.second)) + { + auto last_token = get_token_string(); + sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr)); + return 0; + } - /*! - @param[in] len the length of the object - @return whether object creation completed - */ - bool get_msgpack_object(const std::size_t len) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(len))) - { - return false; - } + string_t type = it->second; // sax->string() takes a reference + if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type))) + { + return 0; + } + + if (size_and_type.second == 'C' || size_and_type.second == 'B') + { + size_and_type.second = 'U'; + } + + key = "_ArrayData_"; + if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first))) + { + return 0; + } + + if (size_and_type.first == 0) + { + return (sax->end_array() && sax->end_object()) ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, true}); + return 1; + } + + // BJData type marker 'B': decode as binary (not a SAX array at all) + if (input_format == input_format_t::bjdata && size_and_type.first != npos && size_and_type.second == 'B') + { + binary_t result; + return (get_binary(input_format, size_and_type.first, result) && sax->binary(result)) ? 2 : 0; + } + + if (size_and_type.first != npos) + { + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) + { + return 0; + } + + // a homogeneous type of 'N' (no-op) means no elements are + // actually read, no matter what count was declared + if (size_and_type.second == 'N' || size_and_type.first == 0) + { + return sax->end_array() ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, false}); + return 1; + } + + // indefinite length + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) + { + return 0; + } + if (current == ']') + { + return sax->end_array() ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{false, true, 0, false, 0, false}); + return 1; + }; - string_t key; - for (std::size_t i = 0; i < len; ++i) + auto enter_object = [&]() -> int { - get(); - if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key))) + std::pair size_and_type; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) { - return false; + return 0; } - if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_internal())) + // do not accept ND-array size in objects in BJData + if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) { - return false; + auto last_token = get_token_string(); + sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, "BJData object does not support ND-array size in optimized format", "object"), nullptr)); + return 0; } - key.clear(); - } - return sax->end_object(); + if (size_and_type.first != npos) + { + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(size_and_type.first))) + { + return 0; + } + if (size_and_type.first == 0) + { + return sax->end_object() ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{true, false, size_and_type.first, true, size_and_type.second, false}); + return 1; + } + + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) + { + return 0; + } + if (current == '}') + { + return sax->end_object() ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{true, true, 0, true, 0, false}); + return 1; + }; + +#define JSON_UBJSON_VALUE(expr) \ + if (JSON_HEDLEY_UNLIKELY(!(expr))) { return false; } \ + if (produce()) { return true; } \ + continue + +#define JSON_UBJSON_ENTER(fn) \ + switch (fn()) \ + { \ + case 0: return false; \ + case 2: if (produce()) { return true; } continue; \ + default: continue; \ } - //////////// - // UBJSON // - //////////// + while (true) + { + if (!stack.empty()) + { + ubjson_container_frame& top = stack.back(); + + if (top.is_object && top.awaiting_key) + { + const bool finished = top.indefinite ? (current == '}') : (top.remaining == 0); + if (finished) + { + if (JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; + } + + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key, !top.indefinite) || !sax->key(key))) + { + return false; + } + top.awaiting_key = false; + } + else if (!top.is_object) + { + const bool finished = top.indefinite ? (current == ']') : (top.remaining == 0); + if (finished) + { + if (JSON_HEDLEY_UNLIKELY(!sax->end_array())) + { + return false; + } + const bool extra_close = top.extra_close_object; + stack.pop_back(); + if (extra_close && JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + if (produce()) + { + return true; + } + continue; + } + } + // else: object frame with a key already read; fall through + // to read that key's value below. + } + + char_int_type prefix; + if (!stack.empty()) + { + ubjson_container_frame& top = stack.back(); + if (top.fixed_type != 0) + { + prefix = top.fixed_type; + } + else if (!top.is_object && top.indefinite) + { + prefix = current; + } + else + { + prefix = get_ignore_noop(); + } + } + else + { + prefix = fetch_char ? get_ignore_noop() : current; + } + + switch (prefix) + { + case char_traits::eof(): // EOF + return unexpect_eof(input_format, "value"); + + case 'T': // true + JSON_UBJSON_VALUE(sax->boolean(true)); + case 'F': // false + JSON_UBJSON_VALUE(sax->boolean(false)); + + case 'Z': // null + JSON_UBJSON_VALUE(sax->null()); + + case 'B': // byte + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint8_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'U': + { + std::uint8_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'i': + { + std::int8_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'I': + { + std::int16_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'l': + { + std::int32_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'L': + { + std::int64_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'u': + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint16_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'm': + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint32_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'M': + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint64_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'h': + { + if (input_format != input_format_t::bjdata) + { + break; + } + const auto byte1_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + { + return false; + } + const auto byte2_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + { + return false; + } + + const auto byte1 = static_cast(byte1_raw); + const auto byte2 = static_cast(byte2_raw); + + // Code from RFC 7049, Appendix D, Figure 3: + // As half-precision floating-point numbers were only added + // to IEEE 754 in 2008, today's programming platforms often + // still only have limited support for them. It is very + // easy to include at least decoding support for them even + // without such support. An example of a small decoder for + // half-precision floating-point numbers in the C language + // is shown in Fig. 3. + const auto half = static_cast((byte2 << 8u) + byte1); + const double val = [&half] + { + const int exp = (half >> 10u) & 0x1Fu; + const unsigned int mant = half & 0x3FFu; + JSON_ASSERT(0 <= exp&& exp <= 32); + JSON_ASSERT(mant <= 1024); + switch (exp) + { + case 0: + return std::ldexp(mant, -24); + case 31: + return (mant == 0) + ? std::numeric_limits::infinity() + : std::numeric_limits::quiet_NaN(); + default: + return std::ldexp(mant + 1024, exp - 25); + } + }(); + JSON_UBJSON_VALUE(sax->number_float((half & 0x8000u) != 0 + ? static_cast(-val) + : static_cast(val), "")); + } + + case 'd': + { + float number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_float(static_cast(number), "")); + } + + case 'D': + { + double number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_float(static_cast(number), "")); + } + + case 'H': + { + JSON_UBJSON_VALUE(get_ubjson_high_precision_number()); + } + + case 'C': // char + { + get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "char"))) + { + return false; + } + if (JSON_HEDLEY_UNLIKELY(current > 127)) + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, + exception_message(input_format, concat("byte after 'C' must be in range 0x00..0x7F; last byte: 0x", last_token), "char"), nullptr)); + } + string_t s(1, static_cast(current)); + JSON_UBJSON_VALUE(sax->string(s)); + } + + case 'S': // string + { + string_t s; + JSON_UBJSON_VALUE(get_ubjson_string(s) && sax->string(s)); + } + + case '[': // array + JSON_UBJSON_ENTER(enter_array); - /*! - @param[in] get_char whether a new character should be retrieved from the - input (true, default) or whether the last read - character should be considered instead + case '{': // object + JSON_UBJSON_ENTER(enter_object); - @return whether a valid UBJSON value was passed to the SAX parser + default: // anything else + break; + } + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, exception_message(input_format, "invalid byte: 0x" + last_token, "value"), nullptr)); + } - @note Unlike CBOR/MessagePack, UBJSON needs get_ubjson_value() (which - carries the @ref depth_guard) to be reachable both from here *and* - directly with an already-known type marker (the optimized `$type#count` - container path), so the dispatch cannot simply be folded into this - function the way parse_cbor_internal()/parse_msgpack_internal() do. That - leaves this thin wrapper as a genuine extra stack frame on every ordinary - array/object element, on top of get_ubjson_value() and - get_ubjson_array()/get_ubjson_object() - one more frame per nesting level - than CBOR/MessagePack/BSON need for the same @ref max_depth. Force-inlining - it (rather than trusting Debug/-O0 builds to do so) keeps UBJSON's - recursion cost per level in line with the other formats' and was - necessary to stop deeply nested UBJSON input from overflowing the stack - well before max_depth is reached in MSVC Debug builds (#5104). - */ - JSON_HEDLEY_ALWAYS_INLINE - bool parse_ubjson_internal(const bool get_char = true) - { - return get_ubjson_value(get_char ? get_ignore_noop() : current); +#undef JSON_UBJSON_VALUE +#undef JSON_UBJSON_ENTER } /*! @@ -2226,587 +2863,187 @@ class binary_reader break; } std::uint32_t number{}; - if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) - { - return false; - } - result = conditional_static_cast(number); - return true; - } - - case 'M': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint64_t number{}; - if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) - { - return false; - } - if (!value_in_range_of(number)) - { - return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, - exception_message(input_format, "integer value overflow", "size"), nullptr)); - } - result = detail::conditional_static_cast(number); - return true; - } - - case '[': - { - if (input_format != input_format_t::bjdata) - { - break; - } - if (is_ndarray) // ndarray dimensional vector can only contain integers and cannot embed another array - { - return sax->parse_error(chars_read, get_token_string(), parse_error::create(113, chars_read, exception_message(input_format, "ndarray dimensional vector is not allowed", "size"), nullptr)); - } - std::vector dim; - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_ndarray_size(dim))) - { - return false; - } - if (dim.size() == 1 || (dim.size() == 2 && dim.at(0) == 1)) // return normal array size if 1D row vector - { - result = dim.at(dim.size() - 1); - return true; - } - if (!dim.empty()) // if ndarray, convert to an object in JData annotated array format - { - for (auto i : dim) // test if any dimension in an ndarray is 0, if so, return a 1D empty container - { - if ( i == 0 ) - { - result = 0; - return true; - } - } - - string_t key = "_ArraySize_"; - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size()))) - { - return false; - } - result = 1; - for (auto i : dim) - { - // Pre-multiplication overflow check: if i > 0 and result > SIZE_MAX/i, then result*i would overflow. - // This check must happen before multiplication since overflow detection after the fact is unreliable - // as modular arithmetic can produce any value, not just 0 or SIZE_MAX. - if (JSON_HEDLEY_UNLIKELY(i > 0 && result > (std::numeric_limits::max)() / i)) - { - return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); - } - result *= i; - // Additional post-multiplication check to catch any edge cases the pre-check might miss - if (result == 0 || result == npos) - { - return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); - } - if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast(i)))) - { - return false; - } - } - is_ndarray = true; - return sax->end_array(); - } - result = 0; - return true; - } - - default: - break; - } - auto last_token = get_token_string(); - std::string message; - - if (input_format != input_format_t::bjdata) - { - message = "expected length type specification (U, i, I, l, L) after '#'; last byte: 0x" + last_token; - } - else - { - message = "expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x" + last_token; - } - return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, exception_message(input_format, message, "size"), nullptr)); - } - - /*! - @brief determine the type and size for a container - - In the optimized UBJSON format, a type and a size can be provided to allow - for a more compact representation. - - @param[out] result pair of the size and the type - @param[in] inside_ndarray whether the parser is parsing an ND array dimensional vector - - @return whether pair creation completed - */ - bool get_ubjson_size_type(std::pair& result, bool inside_ndarray = false) - { - result.first = npos; // size - result.second = 0; // type - bool is_ndarray = false; - - get_ignore_noop(); - - if (current == '$') - { - result.second = get(); // must not ignore 'N', because 'N' maybe the type - if (input_format == input_format_t::bjdata - && JSON_HEDLEY_UNLIKELY(std::binary_search(bjd_optimized_type_markers.begin(), bjd_optimized_type_markers.end(), result.second))) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, concat("marker 0x", last_token, " is not a permitted optimized array type"), "type"), nullptr)); - } - - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "type"))) - { - return false; - } - - get_ignore_noop(); - if (JSON_HEDLEY_UNLIKELY(current != '#')) - { - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "value"))) - { - return false; - } - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr)); - } - - const bool is_error = get_ubjson_size_value(result.first, is_ndarray); - if (input_format == input_format_t::bjdata && is_ndarray) - { - if (inside_ndarray) - { - return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, - exception_message(input_format, "ndarray can not be recursive", "size"), nullptr)); - } - result.second |= (1 << 8); // use bit 8 to indicate ndarray, all UBJSON and BJData markers should be ASCII letters - } - return is_error; - } - - if (current == '#') - { - const bool is_error = get_ubjson_size_value(result.first, is_ndarray); - if (input_format == input_format_t::bjdata && is_ndarray) - { - return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, - exception_message(input_format, "ndarray requires both type and size", "size"), nullptr)); - } - return is_error; - } - - return true; - } - - /*! - @param prefix the previously read or set type prefix - @return whether value creation completed - */ - bool get_ubjson_value(const char_int_type prefix) - { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - } - - switch (prefix) - { - case char_traits::eof(): // EOF - return unexpect_eof(input_format, "value"); - - case 'T': // true - return sax->boolean(true); - case 'F': // false - return sax->boolean(false); - - case 'Z': // null - return sax->null(); - - case 'B': // byte - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint8_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'U': - { - std::uint8_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'i': - { - std::int8_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'I': - { - std::int16_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'l': - { - std::int32_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'L': - { - std::int64_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'u': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint16_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'm': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint32_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'M': - { - if (input_format != input_format_t::bjdata) + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) { - break; + return false; } - std::uint64_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); + result = conditional_static_cast(number); + return true; } - case 'h': + case 'M': { if (input_format != input_format_t::bjdata) { break; } - const auto byte1_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + std::uint64_t number{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) { return false; } - const auto byte2_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + if (!value_in_range_of(number)) { - return false; + return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, + exception_message(input_format, "integer value overflow", "size"), nullptr)); } - - const auto byte1 = static_cast(byte1_raw); - const auto byte2 = static_cast(byte2_raw); - - // Code from RFC 7049, Appendix D, Figure 3: - // As half-precision floating-point numbers were only added - // to IEEE 754 in 2008, today's programming platforms often - // still only have limited support for them. It is very - // easy to include at least decoding support for them even - // without such support. An example of a small decoder for - // half-precision floating-point numbers in the C language - // is shown in Fig. 3. - const auto half = static_cast((byte2 << 8u) + byte1); - const double val = [&half] - { - const int exp = (half >> 10u) & 0x1Fu; - const unsigned int mant = half & 0x3FFu; - JSON_ASSERT(0 <= exp&& exp <= 32); - JSON_ASSERT(mant <= 1024); - switch (exp) - { - case 0: - return std::ldexp(mant, -24); - case 31: - return (mant == 0) - ? std::numeric_limits::infinity() - : std::numeric_limits::quiet_NaN(); - default: - return std::ldexp(mant + 1024, exp - 25); - } - }(); - return sax->number_float((half & 0x8000u) != 0 - ? static_cast(-val) - : static_cast(val), ""); - } - - case 'd': - { - float number{}; - return get_number(input_format, number) && sax->number_float(static_cast(number), ""); - } - - case 'D': - { - double number{}; - return get_number(input_format, number) && sax->number_float(static_cast(number), ""); - } - - case 'H': - { - return get_ubjson_high_precision_number(); + result = detail::conditional_static_cast(number); + return true; } - case 'C': // char + case '[': { - get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "char"))) + if (input_format != input_format_t::bjdata) { - return false; + break; } - if (JSON_HEDLEY_UNLIKELY(current > 127)) + if (is_ndarray) // ndarray dimensional vector can only contain integers and cannot embed another array { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, - exception_message(input_format, concat("byte after 'C' must be in range 0x00..0x7F; last byte: 0x", last_token), "char"), nullptr)); + return sax->parse_error(chars_read, get_token_string(), parse_error::create(113, chars_read, exception_message(input_format, "ndarray dimensional vector is not allowed", "size"), nullptr)); } - string_t s(1, static_cast(current)); - return sax->string(s); - } - - case 'S': // string - { - string_t s; - return get_ubjson_string(s) && sax->string(s); - } - - case '[': // array - return get_ubjson_array(); - - case '{': // object - return get_ubjson_object(); - - default: // anything else - break; - } - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, exception_message(input_format, "invalid byte: 0x" + last_token, "value"), nullptr)); - } - - /*! - @return whether array creation completed - */ - bool get_ubjson_array() - { - std::pair size_and_type; - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) - { - return false; - } - - // if bit-8 of size_and_type.second is set to 1, encode bjdata ndarray as an object in JData annotated array format (https://github.com/NeuroJSON/jdata): - // {"_ArrayType_" : "typeid", "_ArraySize_" : [n1, n2, ...], "_ArrayData_" : [v1, v2, ...]} - - if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) - { - size_and_type.second &= ~(static_cast(1) << 8); // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker - auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), size_and_type.second, [](const bjd_type & p, char_int_type t) - { - return p.first < t; - }); - string_t key = "_ArrayType_"; - if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != size_and_type.second)) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr)); - } - - string_t type = it->second; // sax->string() takes a reference - if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type))) - { - return false; - } - - if (size_and_type.second == 'C' || size_and_type.second == 'B') - { - size_and_type.second = 'U'; - } - - key = "_ArrayData_"; - if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) )) - { - return false; - } - - for (std::size_t i = 0; i < size_and_type.first; ++i) - { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_value(size_and_type.second))) + std::vector dim; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_ndarray_size(dim))) { return false; } - } - - return (sax->end_array() && sax->end_object()); - } - - // If BJData type marker is 'B' decode as binary - if (input_format == input_format_t::bjdata && size_and_type.first != npos && size_and_type.second == 'B') - { - binary_t result; - return get_binary(input_format, size_and_type.first, result) && sax->binary(result); - } - - if (size_and_type.first != npos) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) - { - return false; - } - - if (size_and_type.second != 0) - { - if (size_and_type.second != 'N') + if (dim.size() == 1 || (dim.size() == 2 && dim.at(0) == 1)) // return normal array size if 1D row vector { - for (std::size_t i = 0; i < size_and_type.first; ++i) + result = dim.at(dim.size() - 1); + return true; + } + if (!dim.empty()) // if ndarray, convert to an object in JData annotated array format + { + for (auto i : dim) // test if any dimension in an ndarray is 0, if so, return a 1D empty container { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_value(size_and_type.second))) + if ( i == 0 ) { - return false; + result = 0; + return true; } } - } - } - else - { - for (std::size_t i = 0; i < size_and_type.first; ++i) - { - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal())) + + string_t key = "_ArraySize_"; + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size()))) { return false; } + result = 1; + for (auto i : dim) + { + // Pre-multiplication overflow check: if i > 0 and result > SIZE_MAX/i, then result*i would overflow. + // This check must happen before multiplication since overflow detection after the fact is unreliable + // as modular arithmetic can produce any value, not just 0 or SIZE_MAX. + if (JSON_HEDLEY_UNLIKELY(i > 0 && result > (std::numeric_limits::max)() / i)) + { + return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); + } + result *= i; + // Additional post-multiplication check to catch any edge cases the pre-check might miss + if (result == 0 || result == npos) + { + return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); + } + if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast(i)))) + { + return false; + } + } + is_ndarray = true; + return sax->end_array(); } + result = 0; + return true; } + + default: + break; + } + auto last_token = get_token_string(); + std::string message; + + if (input_format != input_format_t::bjdata) + { + message = "expected length type specification (U, i, I, l, L) after '#'; last byte: 0x" + last_token; } else { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) - { - return false; - } - - while (current != ']') - { - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal(false))) - { - return false; - } - get_ignore_noop(); - } + message = "expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x" + last_token; } - - return sax->end_array(); + return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, exception_message(input_format, message, "size"), nullptr)); } /*! - @return whether object creation completed + @brief determine the type and size for a container + + In the optimized UBJSON format, a type and a size can be provided to allow + for a more compact representation. + + @param[out] result pair of the size and the type + @param[in] inside_ndarray whether the parser is parsing an ND array dimensional vector + + @return whether pair creation completed */ - bool get_ubjson_object() + bool get_ubjson_size_type(std::pair& result, bool inside_ndarray = false) { - std::pair size_and_type; - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) - { - return false; - } + result.first = npos; // size + result.second = 0; // type + bool is_ndarray = false; - // do not accept ND-array size in objects in BJData - if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, "BJData object does not support ND-array size in optimized format", "object"), nullptr)); - } + get_ignore_noop(); - string_t key; - if (size_and_type.first != npos) + if (current == '$') { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(size_and_type.first))) + result.second = get(); // must not ignore 'N', because 'N' maybe the type + if (input_format == input_format_t::bjdata + && JSON_HEDLEY_UNLIKELY(std::binary_search(bjd_optimized_type_markers.begin(), bjd_optimized_type_markers.end(), result.second))) + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, concat("marker 0x", last_token, " is not a permitted optimized array type"), "type"), nullptr)); + } + + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "type"))) { return false; } - if (size_and_type.second != 0) + get_ignore_noop(); + if (JSON_HEDLEY_UNLIKELY(current != '#')) { - for (std::size_t i = 0; i < size_and_type.first; ++i) + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "value"))) { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key) || !sax->key(key))) - { - return false; - } - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_value(size_and_type.second))) - { - return false; - } - key.clear(); + return false; } + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr)); } - else + + const bool is_error = get_ubjson_size_value(result.first, is_ndarray); + if (input_format == input_format_t::bjdata && is_ndarray) { - for (std::size_t i = 0; i < size_and_type.first; ++i) + if (inside_ndarray) { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key) || !sax->key(key))) - { - return false; - } - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal())) - { - return false; - } - key.clear(); + return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, + exception_message(input_format, "ndarray can not be recursive", "size"), nullptr)); } + result.second |= (1 << 8); // use bit 8 to indicate ndarray, all UBJSON and BJData markers should be ASCII letters } + return is_error; } - else - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) - { - return false; - } - while (current != '}') + if (current == '#') + { + const bool is_error = get_ubjson_size_value(result.first, is_ndarray); + if (input_format == input_format_t::bjdata && is_ndarray) { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key, false) || !sax->key(key))) - { - return false; - } - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal())) - { - return false; - } - get_ignore_noop(); - key.clear(); + return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, + exception_message(input_format, "ndarray requires both type and size", "size"), nullptr)); } + return is_error; } - return sax->end_object(); + return true; } // Note, no reader for UBJSON binary types is implemented because they do @@ -3167,19 +3404,19 @@ class binary_reader static JSON_INLINE_VARIABLE constexpr std::size_t npos = detail::unknown_size(); /// maximum allowed nesting depth of arrays/objects for the binary input - /// formats; guards the recursive-descent parsers against stack overflow - /// on maliciously/accidentally deeply nested input. + /// formats. /// - /// This has to satisfy two constraints in tension: high enough that it - /// doesn't reject legitimate input (the library's own roundtrip test - /// fixture, tests/data/json_testsuite/sample.json, nests 458 levels - /// deep), but low enough that reaching it doesn't itself exhaust the - /// native call stack - each nesting level costs multiple real call - /// frames, and unoptimized/Debug builds (notably MSVC Debug, which - /// crashed CI at max_depth=1024 - see #5104) use substantially more - /// stack per frame than an optimized build. 600 leaves comfortable - /// margin on both sides of that range. - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 600; + /// The container-parsing loops in this file are iterative (see e.g. + /// @ref cbor_container_frame): nesting depth is tracked as the size of a + /// heap-allocated std::vector, not native call-stack recursion, so this + /// limit is no longer a stack-safety mechanism - arbitrarily deep input + /// can no longer overflow the stack regardless of this value. It is kept + /// purely as a sanity/DoS cap on absurd inputs (e.g. a malicious payload + /// nesting billions of levels deep to exhaust memory/time), so it can + /// afford to be generous; it is far above the library's own deepest + /// legitimate test fixture (tests/data/json_testsuite/sample.json, 458 + /// levels). + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 100000; /// input adapter InputAdapterType ia; @@ -3190,9 +3427,6 @@ class binary_reader /// the number of characters read std::size_t chars_read = 0; - /// current container nesting depth, see @ref max_depth and @ref depth_guard - std::size_t depth = 0; - /// whether we can assume little endianness const bool is_little_endian = little_endianness(); diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index e0c4473dcd..11f88b60f6 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -10708,71 +10708,209 @@ class binary_reader } private: + ////////// + // BSON // + ////////// + /*! - @brief RAII helper that tracks the nesting depth of containers while - parsing a binary input format. - - The binary format parsers (CBOR, MessagePack, UBJSON/BJData, BSON) parse - nested arrays/objects using recursive descent, so unbounded input nesting - directly translates into unbounded native call-stack recursion. This - guard increments the reader's depth counter for the lifetime of a single - recursive parse step so callers can reject inputs that nest deeper than - @ref max_depth before the recursion has a chance to exhaust the stack. + @brief a single pending "resume this document/array" entry used by the + iterative container traversal in @ref parse_bson_internal; see + @ref cbor_container_frame for the general idea. + + BSON documents and arrays share one on-the-wire layout (a size-prefixed, + 0x00-terminated list of typed elements); @ref is_object only controls + whether each element's key is relayed to the SAX consumer via + @c sax->key() (an object) or not (an array, whose "0", "1", ... keys the + SAX array interface has no use for). Every BSON container is + terminator-based (there is no known-element-count form), so unlike CBOR/ + MessagePack/UBJSON this frame needs no length/indefinite bookkeeping at + all. */ - class depth_guard + struct bson_container_frame { - public: - explicit depth_guard(binary_reader& reader) noexcept : reader_(reader) - { - ++reader_.depth; - } - - ~depth_guard() - { - --reader_.depth; - } - - depth_guard(const depth_guard&) = delete; - depth_guard& operator=(const depth_guard&) = delete; - depth_guard(depth_guard&&) = delete; - depth_guard& operator=(depth_guard&&) = delete; - - private: - binary_reader& reader_; + bool is_object; ///< false: array, true: object/document }; - ////////// - // BSON // - ////////// - /*! @brief Reads in a BSON-object and passes it to the SAX-parser. @return whether a valid BSON-value was passed to the SAX parser */ bool parse_bson_internal() { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); - } + std::vector stack; - std::int32_t document_size{}; - get_number(input_format_t::bson, document_size); + // reads the 4-byte document-size prefix (its value is unused beyond + // consuming those bytes, matching the pre-existing behavior) and + // emits the corresponding SAX start_object()/start_array() event + auto begin_document = [&](const bool is_object) -> bool + { + std::int32_t document_size{}; + get_number(input_format_t::bson, document_size); + return is_object ? sax->start_object(detail::unknown_size()) : sax->start_array(detail::unknown_size()); + }; - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) + if (JSON_HEDLEY_UNLIKELY(!begin_document(true))) { return false; } + stack.push_back(bson_container_frame{true}); - if (JSON_HEDLEY_UNLIKELY(!parse_bson_element_list(/*is_array*/false))) + while (true) { - return false; - } + bson_container_frame& top = stack.back(); + + const auto element_type = get(); + if (element_type == 0) + { + // 0x00 terminates the current document/array + if (JSON_HEDLEY_UNLIKELY(!(top.is_object ? sax->end_object() : sax->end_array()))) + { + return false; + } + stack.pop_back(); + if (stack.empty()) + { + return true; + } + continue; + } + + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::bson, "element list"))) + { + return false; + } + + const std::size_t element_type_parse_position = chars_read; + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_bson_cstr(key))) + { + return false; + } + if (top.is_object && JSON_HEDLEY_UNLIKELY(!sax->key(key))) + { + return false; + } + + switch (element_type) + { + case 0x01: // double + { + double number{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, number) && sax->number_float(static_cast(number), "")))) + { + return false; + } + continue; + } + + case 0x02: // string + { + std::int32_t len{}; + string_t value; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, len) && get_bson_string(len, value) && sax->string(value)))) + { + return false; + } + continue; + } + + case 0x03: // object + { + if (JSON_HEDLEY_UNLIKELY(!begin_document(true))) + { + return false; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + } + stack.push_back(bson_container_frame{true}); + continue; + } + + case 0x04: // array + { + if (JSON_HEDLEY_UNLIKELY(!begin_document(false))) + { + return false; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + } + stack.push_back(bson_container_frame{false}); + continue; + } + + case 0x05: // binary + { + std::int32_t len{}; + binary_t value; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, len) && get_bson_binary(len, value) && sax->binary(value)))) + { + return false; + } + continue; + } + + case 0x08: // boolean + if (JSON_HEDLEY_UNLIKELY(!sax->boolean(get() != 0))) + { + return false; + } + continue; + + case 0x0A: // null + if (JSON_HEDLEY_UNLIKELY(!sax->null())) + { + return false; + } + continue; + + case 0x10: // int32 + { + std::int32_t value{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, value) && sax->number_integer(value)))) + { + return false; + } + continue; + } + + case 0x12: // int64 + { + std::int64_t value{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, value) && sax->number_integer(value)))) + { + return false; + } + continue; + } - return sax->end_object(); + case 0x11: // uint64 + { + std::uint64_t value{}; + if (JSON_HEDLEY_UNLIKELY(!(get_number(input_format_t::bson, value) && sax->number_unsigned(value)))) + { + return false; + } + continue; + } + + default: // anything else is not supported (yet) + { + std::array cr{{}}; + static_cast((std::snprintf)(cr.data(), cr.size(), "%.2hhX", static_cast(element_type))); // NOLINT(cppcoreguidelines-pro-type-vararg,hicpp-vararg) + const std::string cr_str{cr.data()}; + return sax->parse_error(element_type_parse_position, cr_str, + parse_error::create(114, element_type_parse_position, concat("Unsupported BSON record type 0x", cr_str), nullptr)); + } + } + } } /*! @@ -10851,158 +10989,6 @@ class binary_reader return get_binary(input_format_t::bson, len, result); } - /*! - @brief Read a BSON document element of the given @a element_type. - @param[in] element_type The BSON element type, c.f. http://bsonspec.org/spec.html - @param[in] element_type_parse_position The position in the input stream, - where the `element_type` was read. - @warning Not all BSON element types are supported yet. An unsupported - @a element_type will give rise to a parse_error.114: - Unsupported BSON record type 0x... - @return whether a valid BSON-object/array was passed to the SAX parser - */ - bool parse_bson_element_internal(const char_int_type element_type, - const std::size_t element_type_parse_position) - { - switch (element_type) - { - case 0x01: // double - { - double number{}; - return get_number(input_format_t::bson, number) && sax->number_float(static_cast(number), ""); - } - - case 0x02: // string - { - std::int32_t len{}; - string_t value; - return get_number(input_format_t::bson, len) && get_bson_string(len, value) && sax->string(value); - } - - case 0x03: // object - { - return parse_bson_internal(); - } - - case 0x04: // array - { - return parse_bson_array(); - } - - case 0x05: // binary - { - std::int32_t len{}; - binary_t value; - return get_number(input_format_t::bson, len) && get_bson_binary(len, value) && sax->binary(value); - } - - case 0x08: // boolean - { - return sax->boolean(get() != 0); - } - - case 0x0A: // null - { - return sax->null(); - } - - case 0x10: // int32 - { - std::int32_t value{}; - return get_number(input_format_t::bson, value) && sax->number_integer(value); - } - - case 0x12: // int64 - { - std::int64_t value{}; - return get_number(input_format_t::bson, value) && sax->number_integer(value); - } - - case 0x11: // uint64 - { - std::uint64_t value{}; - return get_number(input_format_t::bson, value) && sax->number_unsigned(value); - } - - default: // anything else is not supported (yet) - { - std::array cr{{}}; - static_cast((std::snprintf)(cr.data(), cr.size(), "%.2hhX", static_cast(element_type))); // NOLINT(cppcoreguidelines-pro-type-vararg,hicpp-vararg) - const std::string cr_str{cr.data()}; - return sax->parse_error(element_type_parse_position, cr_str, - parse_error::create(114, element_type_parse_position, concat("Unsupported BSON record type 0x", cr_str), nullptr)); - } - } - } - - /*! - @brief Read a BSON element list (as specified in the BSON-spec) - - The same binary layout is used for objects and arrays, hence it must be - indicated with the argument @a is_array which one is expected - (true --> array, false --> object). - - @param[in] is_array Determines if the element list being read is to be - treated as an object (@a is_array == false), or as an - array (@a is_array == true). - @return whether a valid BSON-object/array was passed to the SAX parser - */ - bool parse_bson_element_list(const bool is_array) - { - string_t key; - - while (auto element_type = get()) - { - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::bson, "element list"))) - { - return false; - } - - const std::size_t element_type_parse_position = chars_read; - if (JSON_HEDLEY_UNLIKELY(!get_bson_cstr(key))) - { - return false; - } - - if (!is_array && !sax->key(key)) - { - return false; - } - - if (JSON_HEDLEY_UNLIKELY(!parse_bson_element_internal(element_type, element_type_parse_position))) - { - return false; - } - - // get_bson_cstr only appends - key.clear(); - } - - return true; - } - - /*! - @brief Reads an array from the BSON input and passes it to the SAX-parser. - @return whether a valid BSON-array was passed to the SAX parser - */ - bool parse_bson_array() - { - std::int32_t document_size{}; - get_number(input_format_t::bson, document_size); - - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) - { - return false; - } - - if (JSON_HEDLEY_UNLIKELY(!parse_bson_element_list(/*is_array*/true))) - { - return false; - } - - return sax->end_array(); - } - ////////// // CBOR // ////////// @@ -11034,508 +11020,687 @@ class binary_reader return sax->number_integer(static_cast(-1) - static_cast(number)); } + /*! + @brief a single pending "resume this array/object" entry used by the + iterative container traversal in @ref parse_cbor_internal + + Nested CBOR containers are parsed by pushing one of these onto a + heap-allocated std::vector instead of recursing on the native call stack + for every nesting level (see #5104): parsing a scalar never recurses, + and parsing a container just pushes a frame here and loops back around + to parse its first child; when a container finishes, its frame is + popped and parsing resumes where the parent left off. Native call-stack + depth used while parsing is therefore O(1) regardless of how deeply the + input nests. + */ + struct cbor_container_frame + { + bool is_object; ///< false: array, true: object/map + bool indefinite; ///< true: length is unknown; read until a 0xFF break marker + std::size_t remaining; ///< remaining element count; meaningful only if !indefinite + bool awaiting_key; ///< object only: true if the next thing to read is a key, not a value + }; + bool parse_cbor_internal(const bool get_char, const cbor_tag_handler_t tag_handler) { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - } + std::vector stack; + + // whether the next value-read below should fetch a fresh byte via + // get() (true) or reuse the byte already sitting in `current` (false) + bool fetch_char = get_char; - switch (get_char ? get() : current) + // called whenever a value (scalar, or a container that was just + // completed/immediately-empty) was produced at the position the + // current top-of-stack frame is waiting on; returns true if that + // means the whole parse is finished (the stack is empty, i.e. this + // was the outermost value), false if parsing should continue. + auto produce = [&]() -> bool { - // EOF - case char_traits::eof(): - return unexpect_eof(input_format_t::cbor, "value"); - - // Integer 0x00..0x17 (0..23) - case 0x00: - case 0x01: - case 0x02: - case 0x03: - case 0x04: - case 0x05: - case 0x06: - case 0x07: - case 0x08: - case 0x09: - case 0x0A: - case 0x0B: - case 0x0C: - case 0x0D: - case 0x0E: - case 0x0F: - case 0x10: - case 0x11: - case 0x12: - case 0x13: - case 0x14: - case 0x15: - case 0x16: - case 0x17: - return sax->number_unsigned(static_cast(current)); - - case 0x18: // Unsigned integer (one-byte uint8_t follows) + if (stack.empty()) { - std::uint8_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); + return true; } - - case 0x19: // Unsigned integer (two-byte uint16_t follows) + cbor_container_frame& top = stack.back(); + if (top.is_object) { - std::uint16_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); + top.awaiting_key = true; + } + if (!top.indefinite) + { + --top.remaining; } + return false; + }; - case 0x1A: // Unsigned integer (four-byte uint32_t follows) + // begin parsing a nested array/object of the given length + // (detail::unknown_size() for indefinite length); returns 0 on SAX + // failure or exceeding max_depth (caller should `return false`), 1 + // if a frame was pushed (caller should `continue` the outer loop to + // parse the container's first element), 2 if the container was + // immediately completed because it is empty (caller should treat + // this like any other produced value via `produce()`). + auto enter_container = [&](const bool is_object, const std::size_t len) -> int + { + const bool started = is_object ? sax->start_object(len) : sax->start_array(len); + if (JSON_HEDLEY_UNLIKELY(!started)) { - std::uint32_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); + return 0; + } + if (len == 0) + { + return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(cbor_container_frame{is_object, len == detail::unknown_size(), len, is_object}); + return 1; + }; + +#define JSON_CBOR_VALUE(expr) \ + if (JSON_HEDLEY_UNLIKELY(!(expr))) { return false; } \ + if (produce()) { return true; } \ + continue + +#define JSON_CBOR_ENTER(is_obj, length) \ + switch (enter_container(is_obj, length)) \ + { \ + case 0: return false; \ + case 2: if (produce()) { return true; } continue; \ + default: continue; \ + } - case 0x1B: // Unsigned integer (eight-byte uint64_t follows) + while (true) + { + if (!stack.empty()) { - std::uint64_t number{}; - return get_number(input_format_t::cbor, number) && sax->number_unsigned(number); - } - - // Negative integer -1-0x00..-1-0x17 (-1..-24) - case 0x20: - case 0x21: - case 0x22: - case 0x23: - case 0x24: - case 0x25: - case 0x26: - case 0x27: - case 0x28: - case 0x29: - case 0x2A: - case 0x2B: - case 0x2C: - case 0x2D: - case 0x2E: - case 0x2F: - case 0x30: - case 0x31: - case 0x32: - case 0x33: - case 0x34: - case 0x35: - case 0x36: - case 0x37: - return sax->number_integer(static_cast(0x20 - 1 - current)); - - case 0x38: // Negative integer (one-byte uint8_t follows) - return get_cbor_negative_integer(); - - case 0x39: // Negative integer -1-n (two-byte uint16_t follows) - return get_cbor_negative_integer(); - - case 0x3A: // Negative integer -1-n (four-byte uint32_t follows) - return get_cbor_negative_integer(); - - case 0x3B: // Negative integer -1-n (eight-byte uint64_t follows) - return get_cbor_negative_integer(); + cbor_container_frame& top = stack.back(); - // Binary data (0x00..0x17 bytes follow) - case 0x40: - case 0x41: - case 0x42: - case 0x43: - case 0x44: - case 0x45: - case 0x46: - case 0x47: - case 0x48: - case 0x49: - case 0x4A: - case 0x4B: - case 0x4C: - case 0x4D: - case 0x4E: - case 0x4F: - case 0x50: - case 0x51: - case 0x52: - case 0x53: - case 0x54: - case 0x55: - case 0x56: - case 0x57: - case 0x58: // Binary data (one-byte uint8_t for n follows) - case 0x59: // Binary data (two-byte uint16_t for n follow) - case 0x5A: // Binary data (four-byte uint32_t for n follow) - case 0x5B: // Binary data (eight-byte uint64_t for n follow) - case 0x5F: // Binary data (indefinite length) - { - binary_t b; - return get_cbor_binary(b) && sax->binary(b); - } - - // UTF-8 string (0x00..0x17 bytes follow) - case 0x60: - case 0x61: - case 0x62: - case 0x63: - case 0x64: - case 0x65: - case 0x66: - case 0x67: - case 0x68: - case 0x69: - case 0x6A: - case 0x6B: - case 0x6C: - case 0x6D: - case 0x6E: - case 0x6F: - case 0x70: - case 0x71: - case 0x72: - case 0x73: - case 0x74: - case 0x75: - case 0x76: - case 0x77: - case 0x78: // UTF-8 string (one-byte uint8_t for n follows) - case 0x79: // UTF-8 string (two-byte uint16_t for n follow) - case 0x7A: // UTF-8 string (four-byte uint32_t for n follow) - case 0x7B: // UTF-8 string (eight-byte uint64_t for n follow) - case 0x7F: // UTF-8 string (indefinite length) - { - string_t s; - return get_cbor_string(s) && sax->string(s); - } - - // array (0x00..0x17 data items follow) - case 0x80: - case 0x81: - case 0x82: - case 0x83: - case 0x84: - case 0x85: - case 0x86: - case 0x87: - case 0x88: - case 0x89: - case 0x8A: - case 0x8B: - case 0x8C: - case 0x8D: - case 0x8E: - case 0x8F: - case 0x90: - case 0x91: - case 0x92: - case 0x93: - case 0x94: - case 0x95: - case 0x96: - case 0x97: - return get_cbor_array( - conditional_static_cast(static_cast(current) & 0x1Fu), tag_handler); - - case 0x98: // array (one-byte uint8_t for n follows) - { - std::uint8_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(static_cast(len), tag_handler); - } - - case 0x99: // array (two-byte uint16_t for n follow) - { - std::uint16_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(static_cast(len), tag_handler); - } - - case 0x9A: // array (four-byte uint32_t for n follow) - { - std::uint32_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(conditional_static_cast(len), tag_handler); - } - - case 0x9B: // array (eight-byte uint64_t for n follow) - { - std::uint64_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_array(conditional_static_cast(len), tag_handler); - } - - case 0x9F: // array (indefinite length) - return get_cbor_array(detail::unknown_size(), tag_handler); - - // map (0x00..0x17 pairs of data items follow) - case 0xA0: - case 0xA1: - case 0xA2: - case 0xA3: - case 0xA4: - case 0xA5: - case 0xA6: - case 0xA7: - case 0xA8: - case 0xA9: - case 0xAA: - case 0xAB: - case 0xAC: - case 0xAD: - case 0xAE: - case 0xAF: - case 0xB0: - case 0xB1: - case 0xB2: - case 0xB3: - case 0xB4: - case 0xB5: - case 0xB6: - case 0xB7: - return get_cbor_object(conditional_static_cast(static_cast(current) & 0x1Fu), tag_handler); - - case 0xB8: // map (one-byte uint8_t for n follows) - { - std::uint8_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(static_cast(len), tag_handler); - } - - case 0xB9: // map (two-byte uint16_t for n follow) - { - std::uint16_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(static_cast(len), tag_handler); - } - - case 0xBA: // map (four-byte uint32_t for n follow) - { - std::uint32_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(conditional_static_cast(len), tag_handler); - } - - case 0xBB: // map (eight-byte uint64_t for n follow) - { - std::uint64_t len{}; - return get_number(input_format_t::cbor, len) && get_cbor_object(conditional_static_cast(len), tag_handler); - } - - case 0xBF: // map (indefinite length) - return get_cbor_object(detail::unknown_size(), tag_handler); - - case 0xC6: // tagged item - case 0xC7: - case 0xC8: - case 0xC9: - case 0xCA: - case 0xCB: - case 0xCC: - case 0xCD: - case 0xCE: - case 0xCF: - case 0xD0: - case 0xD1: - case 0xD2: - case 0xD3: - case 0xD4: - case 0xD8: // tagged item (1 byte follows) - case 0xD9: // tagged item (2 bytes follow) - case 0xDA: // tagged item (4 bytes follow) - case 0xDB: // tagged item (8 bytes follow) - { - switch (tag_handler) - { - case cbor_tag_handler_t::error: - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); - } - - case cbor_tag_handler_t::ignore: + if (top.is_object && top.awaiting_key) + { + const bool finished = top.indefinite ? (get() == 0xFF) : (top.remaining == 0); + if (finished) { - // ignore binary subtype - switch (current) + if (JSON_HEDLEY_UNLIKELY(!sax->end_object())) { - case 0xD8: - { - std::uint8_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - case 0xD9: - { - std::uint16_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - case 0xDA: - { - std::uint32_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - case 0xDB: - { - std::uint64_t subtype_to_ignore{}; - get_number(input_format_t::cbor, subtype_to_ignore); - break; - } - default: - break; + return false; + } + stack.pop_back(); + if (produce()) + { + return true; } - return parse_cbor_internal(true, tag_handler); + continue; } - case cbor_tag_handler_t::store: + if (!top.indefinite) + { + get(); + } + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) + { + return false; + } + top.awaiting_key = false; + fetch_char = true; + } + else if (!top.is_object) + { + const bool finished = top.indefinite ? (get() == 0xFF) : (top.remaining == 0); + if (finished) { - binary_t b; - // use binary subtype and store in a binary container - switch (current) + if (JSON_HEDLEY_UNLIKELY(!sax->end_array())) { - case 0xD8: - { - std::uint8_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; - } - case 0xD9: - { - std::uint16_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; - } - case 0xDA: - { - std::uint32_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; - } - case 0xDB: - { - std::uint64_t subtype{}; - get_number(input_format_t::cbor, subtype); - b.set_subtype(detail::conditional_static_cast(subtype)); - break; - } - default: - return parse_cbor_internal(true, tag_handler); + return false; } - get(); - return get_cbor_binary(b) && sax->binary(b); + stack.pop_back(); + if (produce()) + { + return true; + } + continue; } - - default: // LCOV_EXCL_LINE - JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE - return false; // LCOV_EXCL_LINE + fetch_char = !top.indefinite; } + // else: object frame with a key already read; fetch_char is + // already set to true from when the key was read above. } - case 0xF4: // false - return sax->boolean(false); - - case 0xF5: // true - return sax->boolean(true); +read_value: + switch (fetch_char ? get() : current) + { + // EOF + case char_traits::eof(): + return unexpect_eof(input_format_t::cbor, "value"); - case 0xF6: // null - return sax->null(); + // Integer 0x00..0x17 (0..23) + case 0x00: + case 0x01: + case 0x02: + case 0x03: + case 0x04: + case 0x05: + case 0x06: + case 0x07: + case 0x08: + case 0x09: + case 0x0A: + case 0x0B: + case 0x0C: + case 0x0D: + case 0x0E: + case 0x0F: + case 0x10: + case 0x11: + case 0x12: + case 0x13: + case 0x14: + case 0x15: + case 0x16: + case 0x17: + JSON_CBOR_VALUE(sax->number_unsigned(static_cast(current))); - case 0xF9: // Half-Precision Float (two-byte IEEE 754) - { - const auto byte1_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) + case 0x18: // Unsigned integer (one-byte uint8_t follows) { - return false; + std::uint8_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); } - const auto byte2_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) + + case 0x19: // Unsigned integer (two-byte uint16_t follows) { - return false; + std::uint16_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); } - const auto byte1 = static_cast(byte1_raw); - const auto byte2 = static_cast(byte2_raw); - - // Code from RFC 7049, Appendix D, Figure 3: - // As half-precision floating-point numbers were only added - // to IEEE 754 in 2008, today's programming platforms often - // still only have limited support for them. It is very - // easy to include at least decoding support for them even - // without such support. An example of a small decoder for - // half-precision floating-point numbers in the C language - // is shown in Fig. 3. - const auto half = static_cast((byte1 << 8u) + byte2); - const double val = [&half] - { - const int exp = (half >> 10u) & 0x1Fu; - const unsigned int mant = half & 0x3FFu; - JSON_ASSERT(0 <= exp&& exp <= 32); - JSON_ASSERT(mant <= 1024); - switch (exp) - { - case 0: - return std::ldexp(mant, -24); - case 31: - return (mant == 0) - ? std::numeric_limits::infinity() - : std::numeric_limits::quiet_NaN(); - default: - return std::ldexp(mant + 1024, exp - 25); - } - }(); - return sax->number_float((half & 0x8000u) != 0 - ? static_cast(-val) - : static_cast(val), ""); - } - - case 0xFA: // Single-Precision Float (four-byte IEEE 754) - { - float number{}; - return get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), ""); - } + case 0x1A: // Unsigned integer (four-byte uint32_t follows) + { + std::uint32_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); + } - case 0xFB: // Double-Precision Float (eight-byte IEEE 754) - { - double number{}; - return get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), ""); - } + case 0x1B: // Unsigned integer (eight-byte uint64_t follows) + { + std::uint64_t number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_unsigned(number)); + } - default: // anything else (0xFF is handled inside the other types) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); - } - } - } + // Negative integer -1-0x00..-1-0x17 (-1..-24) + case 0x20: + case 0x21: + case 0x22: + case 0x23: + case 0x24: + case 0x25: + case 0x26: + case 0x27: + case 0x28: + case 0x29: + case 0x2A: + case 0x2B: + case 0x2C: + case 0x2D: + case 0x2E: + case 0x2F: + case 0x30: + case 0x31: + case 0x32: + case 0x33: + case 0x34: + case 0x35: + case 0x36: + case 0x37: + JSON_CBOR_VALUE(sax->number_integer(static_cast(0x20 - 1 - current))); - /*! - @brief reads a CBOR string + case 0x38: // Negative integer (one-byte uint8_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); - This function first reads starting bytes to determine the expected - string length and then copies this number of bytes into a string. - Additionally, CBOR's strings with indefinite lengths are supported. + case 0x39: // Negative integer -1-n (two-byte uint16_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); - @param[out] result created string + case 0x3A: // Negative integer -1-n (four-byte uint32_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); - @return whether string creation completed - */ - bool get_cbor_string(string_t& result) - { - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string"))) - { - return false; - } + case 0x3B: // Negative integer -1-n (eight-byte uint64_t follows) + JSON_CBOR_VALUE(get_cbor_negative_integer()); - switch (current) - { - // UTF-8 string (0x00..0x17 bytes follow) - case 0x60: - case 0x61: - case 0x62: - case 0x63: - case 0x64: - case 0x65: + // Binary data (0x00..0x17 bytes follow) + case 0x40: + case 0x41: + case 0x42: + case 0x43: + case 0x44: + case 0x45: + case 0x46: + case 0x47: + case 0x48: + case 0x49: + case 0x4A: + case 0x4B: + case 0x4C: + case 0x4D: + case 0x4E: + case 0x4F: + case 0x50: + case 0x51: + case 0x52: + case 0x53: + case 0x54: + case 0x55: + case 0x56: + case 0x57: + case 0x58: // Binary data (one-byte uint8_t for n follows) + case 0x59: // Binary data (two-byte uint16_t for n follow) + case 0x5A: // Binary data (four-byte uint32_t for n follow) + case 0x5B: // Binary data (eight-byte uint64_t for n follow) + case 0x5F: // Binary data (indefinite length) + { + binary_t b; + JSON_CBOR_VALUE(get_cbor_binary(b) && sax->binary(b)); + } + + // UTF-8 string (0x00..0x17 bytes follow) + case 0x60: + case 0x61: + case 0x62: + case 0x63: + case 0x64: + case 0x65: + case 0x66: + case 0x67: + case 0x68: + case 0x69: + case 0x6A: + case 0x6B: + case 0x6C: + case 0x6D: + case 0x6E: + case 0x6F: + case 0x70: + case 0x71: + case 0x72: + case 0x73: + case 0x74: + case 0x75: + case 0x76: + case 0x77: + case 0x78: // UTF-8 string (one-byte uint8_t for n follows) + case 0x79: // UTF-8 string (two-byte uint16_t for n follow) + case 0x7A: // UTF-8 string (four-byte uint32_t for n follow) + case 0x7B: // UTF-8 string (eight-byte uint64_t for n follow) + case 0x7F: // UTF-8 string (indefinite length) + { + string_t s; + JSON_CBOR_VALUE(get_cbor_string(s) && sax->string(s)); + } + + // array (0x00..0x17 data items follow) + case 0x80: + case 0x81: + case 0x82: + case 0x83: + case 0x84: + case 0x85: + case 0x86: + case 0x87: + case 0x88: + case 0x89: + case 0x8A: + case 0x8B: + case 0x8C: + case 0x8D: + case 0x8E: + case 0x8F: + case 0x90: + case 0x91: + case 0x92: + case 0x93: + case 0x94: + case 0x95: + case 0x96: + case 0x97: + JSON_CBOR_ENTER(false, conditional_static_cast(static_cast(current) & 0x1Fu)); + + case 0x98: // array (one-byte uint8_t for n follows) + { + std::uint8_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(false, static_cast(len)); + } + + case 0x99: // array (two-byte uint16_t for n follow) + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(false, static_cast(len)); + } + + case 0x9A: // array (four-byte uint32_t for n follow) + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(false, conditional_static_cast(len)); + } + + case 0x9B: // array (eight-byte uint64_t for n follow) + { + std::uint64_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(false, conditional_static_cast(len)); + } + + case 0x9F: // array (indefinite length) + JSON_CBOR_ENTER(false, detail::unknown_size()); + + // map (0x00..0x17 pairs of data items follow) + case 0xA0: + case 0xA1: + case 0xA2: + case 0xA3: + case 0xA4: + case 0xA5: + case 0xA6: + case 0xA7: + case 0xA8: + case 0xA9: + case 0xAA: + case 0xAB: + case 0xAC: + case 0xAD: + case 0xAE: + case 0xAF: + case 0xB0: + case 0xB1: + case 0xB2: + case 0xB3: + case 0xB4: + case 0xB5: + case 0xB6: + case 0xB7: + JSON_CBOR_ENTER(true, conditional_static_cast(static_cast(current) & 0x1Fu)); + + case 0xB8: // map (one-byte uint8_t for n follows) + { + std::uint8_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, static_cast(len)); + } + + case 0xB9: // map (two-byte uint16_t for n follow) + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, static_cast(len)); + } + + case 0xBA: // map (four-byte uint32_t for n follow) + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, conditional_static_cast(len)); + } + + case 0xBB: // map (eight-byte uint64_t for n follow) + { + std::uint64_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::cbor, len))) + { + return false; + } + JSON_CBOR_ENTER(true, conditional_static_cast(len)); + } + + case 0xBF: // map (indefinite length) + JSON_CBOR_ENTER(true, detail::unknown_size()); + + case 0xC6: // tagged item + case 0xC7: + case 0xC8: + case 0xC9: + case 0xCA: + case 0xCB: + case 0xCC: + case 0xCD: + case 0xCE: + case 0xCF: + case 0xD0: + case 0xD1: + case 0xD2: + case 0xD3: + case 0xD4: + case 0xD8: // tagged item (1 byte follows) + case 0xD9: // tagged item (2 bytes follow) + case 0xDA: // tagged item (4 bytes follow) + case 0xDB: // tagged item (8 bytes follow) + { + switch (tag_handler) + { + case cbor_tag_handler_t::error: + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); + } + + case cbor_tag_handler_t::ignore: + { + // ignore binary subtype + switch (current) + { + case 0xD8: + { + std::uint8_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + case 0xD9: + { + std::uint16_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + case 0xDA: + { + std::uint32_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + case 0xDB: + { + std::uint64_t subtype_to_ignore{}; + get_number(input_format_t::cbor, subtype_to_ignore); + break; + } + default: + break; + } + fetch_char = true; + goto read_value; + } + + case cbor_tag_handler_t::store: + { + binary_t b; + // use binary subtype and store in a binary container + switch (current) + { + case 0xD8: + { + std::uint8_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + case 0xD9: + { + std::uint16_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + case 0xDA: + { + std::uint32_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + case 0xDB: + { + std::uint64_t subtype{}; + get_number(input_format_t::cbor, subtype); + b.set_subtype(detail::conditional_static_cast(subtype)); + break; + } + default: + fetch_char = true; + goto read_value; + } + get(); + JSON_CBOR_VALUE(get_cbor_binary(b) && sax->binary(b)); + } + + default: // LCOV_EXCL_LINE + JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE + return false; // LCOV_EXCL_LINE + } + } + + case 0xF4: // false + JSON_CBOR_VALUE(sax->boolean(false)); + + case 0xF5: // true + JSON_CBOR_VALUE(sax->boolean(true)); + + case 0xF6: // null + JSON_CBOR_VALUE(sax->null()); + + case 0xF9: // Half-Precision Float (two-byte IEEE 754) + { + const auto byte1_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) + { + return false; + } + const auto byte2_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "number"))) + { + return false; + } + + const auto byte1 = static_cast(byte1_raw); + const auto byte2 = static_cast(byte2_raw); + + // Code from RFC 7049, Appendix D, Figure 3: + // As half-precision floating-point numbers were only added + // to IEEE 754 in 2008, today's programming platforms often + // still only have limited support for them. It is very + // easy to include at least decoding support for them even + // without such support. An example of a small decoder for + // half-precision floating-point numbers in the C language + // is shown in Fig. 3. + const auto half = static_cast((byte1 << 8u) + byte2); + const double val = [&half] + { + const int exp = (half >> 10u) & 0x1Fu; + const unsigned int mant = half & 0x3FFu; + JSON_ASSERT(0 <= exp&& exp <= 32); + JSON_ASSERT(mant <= 1024); + switch (exp) + { + case 0: + return std::ldexp(mant, -24); + case 31: + return (mant == 0) + ? std::numeric_limits::infinity() + : std::numeric_limits::quiet_NaN(); + default: + return std::ldexp(mant + 1024, exp - 25); + } + }(); + JSON_CBOR_VALUE(sax->number_float((half & 0x8000u) != 0 + ? static_cast(-val) + : static_cast(val), "")); + } + + case 0xFA: // Single-Precision Float (four-byte IEEE 754) + { + float number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), "")); + } + + case 0xFB: // Double-Precision Float (eight-byte IEEE 754) + { + double number{}; + JSON_CBOR_VALUE(get_number(input_format_t::cbor, number) && sax->number_float(static_cast(number), "")); + } + + default: // anything else (0xFF is handled inside the other types) + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format_t::cbor, concat("invalid byte: 0x", last_token), "value"), nullptr)); + } + } + } + +#undef JSON_CBOR_VALUE +#undef JSON_CBOR_ENTER + } + + /*! + @brief reads a CBOR string + + This function first reads starting bytes to determine the expected + string length and then copies this number of bytes into a string. + Additionally, CBOR's strings with indefinite lengths are supported. + + @param[out] result created string + + @return whether string creation completed + */ + bool get_cbor_string(string_t& result) + { + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format_t::cbor, "string"))) + { + return false; + } + + switch (current) + { + // UTF-8 string (0x00..0x17 bytes follow) + case 0x60: + case 0x61: + case 0x62: + case 0x63: + case 0x64: + case 0x65: case 0x66: case 0x67: case 0x68: @@ -11705,483 +11870,517 @@ class binary_reader } } + ///////////// + // MsgPack // + ///////////// + /*! - @param[in] len the length of the array or detail::unknown_size() for an - array of indefinite size - @param[in] tag_handler how CBOR tags should be treated - @return whether array creation completed + @brief a single pending "resume this array/object" entry used by the + iterative container traversal in @ref parse_msgpack_internal; + see @ref cbor_container_frame for the general idea. MessagePack + has no indefinite-length containers, so unlike CBOR this frame + does not need an "indefinite" flag. + */ + struct msgpack_container_frame + { + bool is_object; ///< false: array, true: map + std::size_t remaining; ///< remaining element count + bool awaiting_key; ///< map only: true if the next thing to read is a key, not a value + }; + + /*! + @return whether a valid MessagePack value was passed to the SAX parser */ - bool get_cbor_array(const std::size_t len, - const cbor_tag_handler_t tag_handler) + bool parse_msgpack_internal() { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(len))) + std::vector stack; + + auto produce = [&]() -> bool { + if (stack.empty()) + { + return true; + } + msgpack_container_frame& top = stack.back(); + if (top.is_object) + { + top.awaiting_key = true; + } + --top.remaining; return false; - } + }; - if (len != detail::unknown_size()) + auto enter_container = [&](const bool is_object, const std::size_t len) -> int { - for (std::size_t i = 0; i < len; ++i) + const bool started = is_object ? sax->start_object(len) : sax->start_array(len); + if (JSON_HEDLEY_UNLIKELY(!started)) { - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(true, tag_handler))) - { - return false; - } + return 0; } - } - else - { - while (get() != 0xFF) + if (len == 0) { - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(false, tag_handler))) - { - return false; - } + return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; } - } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(msgpack_container_frame{is_object, len, is_object}); + return 1; + }; - return sax->end_array(); - } +#define JSON_MSGPACK_VALUE(expr) \ + if (JSON_HEDLEY_UNLIKELY(!(expr))) { return false; } \ + if (produce()) { return true; } \ + continue - /*! - @param[in] len the length of the object or detail::unknown_size() for an - object of indefinite size - @param[in] tag_handler how CBOR tags should be treated - @return whether object creation completed - */ - bool get_cbor_object(const std::size_t len, - const cbor_tag_handler_t tag_handler) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(len))) - { - return false; - } +#define JSON_MSGPACK_ENTER(is_obj, length) \ + switch (enter_container(is_obj, length)) \ + { \ + case 0: return false; \ + case 2: if (produce()) { return true; } continue; \ + default: continue; \ + } - if (len != 0) + while (true) { - string_t key; - if (len != detail::unknown_size()) + if (!stack.empty()) { - for (std::size_t i = 0; i < len; ++i) + msgpack_container_frame& top = stack.back(); + + if (top.is_object && top.awaiting_key) { - get(); - if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) + if (top.remaining == 0) { - return false; + if (JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; } - - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(true, tag_handler))) + get(); + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key))) { return false; } - key.clear(); + top.awaiting_key = false; } - } - else - { - while (get() != 0xFF) + else if (!top.is_object) { - if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) - { - return false; - } - - if (JSON_HEDLEY_UNLIKELY(!parse_cbor_internal(true, tag_handler))) + if (top.remaining == 0) { - return false; + if (JSON_HEDLEY_UNLIKELY(!sax->end_array())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; } - key.clear(); } + // else: map frame with a key already read; fall through to + // read that key's value. } - } - - return sax->end_object(); - } - - ///////////// - // MsgPack // - ///////////// - - /*! - @return whether a valid MessagePack value was passed to the SAX parser - */ - bool parse_msgpack_internal() - { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - } - switch (get()) - { - // EOF - case char_traits::eof(): - return unexpect_eof(input_format_t::msgpack, "value"); - - // positive fixint - case 0x00: - case 0x01: - case 0x02: - case 0x03: - case 0x04: - case 0x05: - case 0x06: - case 0x07: - case 0x08: - case 0x09: - case 0x0A: - case 0x0B: - case 0x0C: - case 0x0D: - case 0x0E: - case 0x0F: - case 0x10: - case 0x11: - case 0x12: - case 0x13: - case 0x14: - case 0x15: - case 0x16: - case 0x17: - case 0x18: - case 0x19: - case 0x1A: - case 0x1B: - case 0x1C: - case 0x1D: - case 0x1E: - case 0x1F: - case 0x20: - case 0x21: - case 0x22: - case 0x23: - case 0x24: - case 0x25: - case 0x26: - case 0x27: - case 0x28: - case 0x29: - case 0x2A: - case 0x2B: - case 0x2C: - case 0x2D: - case 0x2E: - case 0x2F: - case 0x30: - case 0x31: - case 0x32: - case 0x33: - case 0x34: - case 0x35: - case 0x36: - case 0x37: - case 0x38: - case 0x39: - case 0x3A: - case 0x3B: - case 0x3C: - case 0x3D: - case 0x3E: - case 0x3F: - case 0x40: - case 0x41: - case 0x42: - case 0x43: - case 0x44: - case 0x45: - case 0x46: - case 0x47: - case 0x48: - case 0x49: - case 0x4A: - case 0x4B: - case 0x4C: - case 0x4D: - case 0x4E: - case 0x4F: - case 0x50: - case 0x51: - case 0x52: - case 0x53: - case 0x54: - case 0x55: - case 0x56: - case 0x57: - case 0x58: - case 0x59: - case 0x5A: - case 0x5B: - case 0x5C: - case 0x5D: - case 0x5E: - case 0x5F: - case 0x60: - case 0x61: - case 0x62: - case 0x63: - case 0x64: - case 0x65: - case 0x66: - case 0x67: - case 0x68: - case 0x69: - case 0x6A: - case 0x6B: - case 0x6C: - case 0x6D: - case 0x6E: - case 0x6F: - case 0x70: - case 0x71: - case 0x72: - case 0x73: - case 0x74: - case 0x75: - case 0x76: - case 0x77: - case 0x78: - case 0x79: - case 0x7A: - case 0x7B: - case 0x7C: - case 0x7D: - case 0x7E: - case 0x7F: - return sax->number_unsigned(static_cast(current)); - - // fixmap - case 0x80: - case 0x81: - case 0x82: - case 0x83: - case 0x84: - case 0x85: - case 0x86: - case 0x87: - case 0x88: - case 0x89: - case 0x8A: - case 0x8B: - case 0x8C: - case 0x8D: - case 0x8E: - case 0x8F: - return get_msgpack_object(conditional_static_cast(static_cast(current) & 0x0Fu)); - - // fixarray - case 0x90: - case 0x91: - case 0x92: - case 0x93: - case 0x94: - case 0x95: - case 0x96: - case 0x97: - case 0x98: - case 0x99: - case 0x9A: - case 0x9B: - case 0x9C: - case 0x9D: - case 0x9E: - case 0x9F: - return get_msgpack_array(conditional_static_cast(static_cast(current) & 0x0Fu)); - - // fixstr - case 0xA0: - case 0xA1: - case 0xA2: - case 0xA3: - case 0xA4: - case 0xA5: - case 0xA6: - case 0xA7: - case 0xA8: - case 0xA9: - case 0xAA: - case 0xAB: - case 0xAC: - case 0xAD: - case 0xAE: - case 0xAF: - case 0xB0: - case 0xB1: - case 0xB2: - case 0xB3: - case 0xB4: - case 0xB5: - case 0xB6: - case 0xB7: - case 0xB8: - case 0xB9: - case 0xBA: - case 0xBB: - case 0xBC: - case 0xBD: - case 0xBE: - case 0xBF: - case 0xD9: // str 8 - case 0xDA: // str 16 - case 0xDB: // str 32 + switch (get()) { - string_t s; - return get_msgpack_string(s) && sax->string(s); - } + // EOF + case char_traits::eof(): + return unexpect_eof(input_format_t::msgpack, "value"); - case 0xC0: // nil - return sax->null(); + // positive fixint + case 0x00: + case 0x01: + case 0x02: + case 0x03: + case 0x04: + case 0x05: + case 0x06: + case 0x07: + case 0x08: + case 0x09: + case 0x0A: + case 0x0B: + case 0x0C: + case 0x0D: + case 0x0E: + case 0x0F: + case 0x10: + case 0x11: + case 0x12: + case 0x13: + case 0x14: + case 0x15: + case 0x16: + case 0x17: + case 0x18: + case 0x19: + case 0x1A: + case 0x1B: + case 0x1C: + case 0x1D: + case 0x1E: + case 0x1F: + case 0x20: + case 0x21: + case 0x22: + case 0x23: + case 0x24: + case 0x25: + case 0x26: + case 0x27: + case 0x28: + case 0x29: + case 0x2A: + case 0x2B: + case 0x2C: + case 0x2D: + case 0x2E: + case 0x2F: + case 0x30: + case 0x31: + case 0x32: + case 0x33: + case 0x34: + case 0x35: + case 0x36: + case 0x37: + case 0x38: + case 0x39: + case 0x3A: + case 0x3B: + case 0x3C: + case 0x3D: + case 0x3E: + case 0x3F: + case 0x40: + case 0x41: + case 0x42: + case 0x43: + case 0x44: + case 0x45: + case 0x46: + case 0x47: + case 0x48: + case 0x49: + case 0x4A: + case 0x4B: + case 0x4C: + case 0x4D: + case 0x4E: + case 0x4F: + case 0x50: + case 0x51: + case 0x52: + case 0x53: + case 0x54: + case 0x55: + case 0x56: + case 0x57: + case 0x58: + case 0x59: + case 0x5A: + case 0x5B: + case 0x5C: + case 0x5D: + case 0x5E: + case 0x5F: + case 0x60: + case 0x61: + case 0x62: + case 0x63: + case 0x64: + case 0x65: + case 0x66: + case 0x67: + case 0x68: + case 0x69: + case 0x6A: + case 0x6B: + case 0x6C: + case 0x6D: + case 0x6E: + case 0x6F: + case 0x70: + case 0x71: + case 0x72: + case 0x73: + case 0x74: + case 0x75: + case 0x76: + case 0x77: + case 0x78: + case 0x79: + case 0x7A: + case 0x7B: + case 0x7C: + case 0x7D: + case 0x7E: + case 0x7F: + JSON_MSGPACK_VALUE(sax->number_unsigned(static_cast(current))); + + // fixmap + case 0x80: + case 0x81: + case 0x82: + case 0x83: + case 0x84: + case 0x85: + case 0x86: + case 0x87: + case 0x88: + case 0x89: + case 0x8A: + case 0x8B: + case 0x8C: + case 0x8D: + case 0x8E: + case 0x8F: + JSON_MSGPACK_ENTER(true, conditional_static_cast(static_cast(current) & 0x0Fu)); + + // fixarray + case 0x90: + case 0x91: + case 0x92: + case 0x93: + case 0x94: + case 0x95: + case 0x96: + case 0x97: + case 0x98: + case 0x99: + case 0x9A: + case 0x9B: + case 0x9C: + case 0x9D: + case 0x9E: + case 0x9F: + JSON_MSGPACK_ENTER(false, conditional_static_cast(static_cast(current) & 0x0Fu)); + + // fixstr + case 0xA0: + case 0xA1: + case 0xA2: + case 0xA3: + case 0xA4: + case 0xA5: + case 0xA6: + case 0xA7: + case 0xA8: + case 0xA9: + case 0xAA: + case 0xAB: + case 0xAC: + case 0xAD: + case 0xAE: + case 0xAF: + case 0xB0: + case 0xB1: + case 0xB2: + case 0xB3: + case 0xB4: + case 0xB5: + case 0xB6: + case 0xB7: + case 0xB8: + case 0xB9: + case 0xBA: + case 0xBB: + case 0xBC: + case 0xBD: + case 0xBE: + case 0xBF: + case 0xD9: // str 8 + case 0xDA: // str 16 + case 0xDB: // str 32 + { + string_t s; + JSON_MSGPACK_VALUE(get_msgpack_string(s) && sax->string(s)); + } - case 0xC2: // false - return sax->boolean(false); + case 0xC0: // nil + JSON_MSGPACK_VALUE(sax->null()); - case 0xC3: // true - return sax->boolean(true); + case 0xC2: // false + JSON_MSGPACK_VALUE(sax->boolean(false)); - case 0xC4: // bin 8 - case 0xC5: // bin 16 - case 0xC6: // bin 32 - case 0xC7: // ext 8 - case 0xC8: // ext 16 - case 0xC9: // ext 32 - case 0xD4: // fixext 1 - case 0xD5: // fixext 2 - case 0xD6: // fixext 4 - case 0xD7: // fixext 8 - case 0xD8: // fixext 16 - { - binary_t b; - return get_msgpack_binary(b) && sax->binary(b); - } + case 0xC3: // true + JSON_MSGPACK_VALUE(sax->boolean(true)); - case 0xCA: // float 32 - { - float number{}; - return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), ""); - } + case 0xC4: // bin 8 + case 0xC5: // bin 16 + case 0xC6: // bin 32 + case 0xC7: // ext 8 + case 0xC8: // ext 16 + case 0xC9: // ext 32 + case 0xD4: // fixext 1 + case 0xD5: // fixext 2 + case 0xD6: // fixext 4 + case 0xD7: // fixext 8 + case 0xD8: // fixext 16 + { + binary_t b; + JSON_MSGPACK_VALUE(get_msgpack_binary(b) && sax->binary(b)); + } - case 0xCB: // float 64 - { - double number{}; - return get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), ""); - } + case 0xCA: // float 32 + { + float number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), "")); + } + + case 0xCB: // float 64 + { + double number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_float(static_cast(number), "")); + } - case 0xCC: // uint 8 - { - std::uint8_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCC: // uint 8 + { + std::uint8_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCD: // uint 16 - { - std::uint16_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCD: // uint 16 + { + std::uint16_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCE: // uint 32 - { - std::uint32_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCE: // uint 32 + { + std::uint32_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xCF: // uint 64 - { - std::uint64_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_unsigned(number); - } + case 0xCF: // uint 64 + { + std::uint64_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_unsigned(number)); + } - case 0xD0: // int 8 - { - std::int8_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD0: // int 8 + { + std::int8_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD1: // int 16 - { - std::int16_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD1: // int 16 + { + std::int16_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD2: // int 32 - { - std::int32_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD2: // int 32 + { + std::int32_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xD3: // int 64 - { - std::int64_t number{}; - return get_number(input_format_t::msgpack, number) && sax->number_integer(number); - } + case 0xD3: // int 64 + { + std::int64_t number{}; + JSON_MSGPACK_VALUE(get_number(input_format_t::msgpack, number) && sax->number_integer(number)); + } - case 0xDC: // array 16 - { - std::uint16_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_array(static_cast(len)); - } + case 0xDC: // array 16 + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(false, static_cast(len)); + } - case 0xDD: // array 32 - { - std::uint32_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_array(conditional_static_cast(len)); - } + case 0xDD: // array 32 + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(false, conditional_static_cast(len)); + } - case 0xDE: // map 16 - { - std::uint16_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_object(static_cast(len)); - } + case 0xDE: // map 16 + { + std::uint16_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(true, static_cast(len)); + } - case 0xDF: // map 32 - { - std::uint32_t len{}; - return get_number(input_format_t::msgpack, len) && get_msgpack_object(conditional_static_cast(len)); - } - - // negative fixint - case 0xE0: - case 0xE1: - case 0xE2: - case 0xE3: - case 0xE4: - case 0xE5: - case 0xE6: - case 0xE7: - case 0xE8: - case 0xE9: - case 0xEA: - case 0xEB: - case 0xEC: - case 0xED: - case 0xEE: - case 0xEF: - case 0xF0: - case 0xF1: - case 0xF2: - case 0xF3: - case 0xF4: - case 0xF5: - case 0xF6: - case 0xF7: - case 0xF8: - case 0xF9: - case 0xFA: - case 0xFB: - case 0xFC: - case 0xFD: - case 0xFE: - case 0xFF: - return sax->number_integer(static_cast(current)); - - default: // anything else - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format_t::msgpack, concat("invalid byte: 0x", last_token), "value"), nullptr)); + case 0xDF: // map 32 + { + std::uint32_t len{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format_t::msgpack, len))) + { + return false; + } + JSON_MSGPACK_ENTER(true, conditional_static_cast(len)); + } + + // negative fixint + case 0xE0: + case 0xE1: + case 0xE2: + case 0xE3: + case 0xE4: + case 0xE5: + case 0xE6: + case 0xE7: + case 0xE8: + case 0xE9: + case 0xEA: + case 0xEB: + case 0xEC: + case 0xED: + case 0xEE: + case 0xEF: + case 0xF0: + case 0xF1: + case 0xF2: + case 0xF3: + case 0xF4: + case 0xF5: + case 0xF6: + case 0xF7: + case 0xF8: + case 0xF9: + case 0xFA: + case 0xFB: + case 0xFC: + case 0xFD: + case 0xFE: + case 0xFF: + JSON_MSGPACK_VALUE(sax->number_integer(static_cast(current))); + + default: // anything else + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format_t::msgpack, concat("invalid byte: 0x", last_token), "value"), nullptr)); + } } } + +#undef JSON_MSGPACK_VALUE +#undef JSON_MSGPACK_ENTER } /*! @@ -12286,185 +12485,623 @@ class binary_reader return true; }; - switch (current) - { - case 0xC4: // bin 8 + switch (current) + { + case 0xC4: // bin 8 + { + std::uint8_t len{}; + return get_number(input_format_t::msgpack, len) && + get_binary(input_format_t::msgpack, len, result); + } + + case 0xC5: // bin 16 + { + std::uint16_t len{}; + return get_number(input_format_t::msgpack, len) && + get_binary(input_format_t::msgpack, len, result); + } + + case 0xC6: // bin 32 + { + std::uint32_t len{}; + return get_number(input_format_t::msgpack, len) && + get_binary(input_format_t::msgpack, len, result); + } + + case 0xC7: // ext 8 + { + std::uint8_t len{}; + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, len) && + get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, len, result) && + assign_and_return_true(subtype); + } + + case 0xC8: // ext 16 + { + std::uint16_t len{}; + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, len) && + get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, len, result) && + assign_and_return_true(subtype); + } + + case 0xC9: // ext 32 + { + std::uint32_t len{}; + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, len) && + get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, len, result) && + assign_and_return_true(subtype); + } + + case 0xD4: // fixext 1 + { + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, 1, result) && + assign_and_return_true(subtype); + } + + case 0xD5: // fixext 2 + { + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, 2, result) && + assign_and_return_true(subtype); + } + + case 0xD6: // fixext 4 + { + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, 4, result) && + assign_and_return_true(subtype); + } + + case 0xD7: // fixext 8 + { + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, 8, result) && + assign_and_return_true(subtype); + } + + case 0xD8: // fixext 16 + { + std::int8_t subtype{}; + return get_number(input_format_t::msgpack, subtype) && + get_binary(input_format_t::msgpack, 16, result) && + assign_and_return_true(subtype); + } + + default: // LCOV_EXCL_LINE + return false; // LCOV_EXCL_LINE + } + } + + //////////// + // UBJSON // + //////////// + + /*! + @brief a single pending "resume this array/object" entry used by the + iterative container traversal in @ref parse_ubjson_internal; see + @ref cbor_container_frame for the general idea. + + UBJSON/BJData needs two extra bits of state beyond CBOR/MessagePack: + @ref fixed_type captures the optimized `$type#count` container's + homogeneous element type (0 means "heterogeneous": read a fresh type + marker per element, like an ordinary UBJSON array/object); @ref + extra_close_object is set for the synthetic JData-annotated-array object + ("_ArrayType_"/"_ArraySize_"/"_ArrayData_") BJData wraps an ND-array in, + where finishing the `_ArrayData_` array must also close that wrapping + object. + */ + struct ubjson_container_frame + { + bool is_object; ///< false: array, true: object + bool indefinite; ///< true: no known count; read until a ']'/'}' terminator + std::size_t remaining; ///< remaining element count; meaningful only if !indefinite + bool awaiting_key; ///< object only: true if the next thing to read is a key, not a value + char_int_type fixed_type; ///< 0: heterogeneous; else every element has this known type marker + bool extra_close_object; ///< true: end_array() for this frame must be followed by an end_object() + }; + + /*! + @param[in] get_char whether a new character should be retrieved from the + input (true, default) or whether the last read + character should be considered instead + + @return whether a valid UBJSON value was passed to the SAX parser + */ + bool parse_ubjson_internal(const bool get_char = true) + { + std::vector stack; + bool fetch_char = get_char; + + auto produce = [&]() -> bool + { + if (stack.empty()) + { + return true; + } + ubjson_container_frame& top = stack.back(); + if (top.is_object) + { + top.awaiting_key = true; + } + if (!top.indefinite) + { + --top.remaining; + } + else + { + // indefinite-length containers are terminated by inspecting + // `current`, so advance past the just-produced element/pair + // to whatever byte should be inspected next (mirrors the + // `get_ignore_noop()` at the end of each iteration of the + // original recursive implementation's while-loops) + get_ignore_noop(); + } + return false; + }; + + // returns 0 = failure (caller returns false), 1 = a frame was + // pushed (caller continues the outer loop to parse the first + // element), 2 = a value was produced immediately - either because + // the container turned out to be empty, or (BJData only) because + // the "optimized binary array" special case produced a single + // binary value directly, without ever being a SAX array at all + // (caller should invoke produce() same as for any other value) + auto enter_array = [&]() -> int + { + std::pair size_and_type; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) + { + return 0; + } + + // ND-array: encode as an object in JData annotated array format + // (https://github.com/NeuroJSON/jdata). The wrapping object was + // already start_object()'d (with the "_ArraySize_" key/array + // already emitted) inside get_ubjson_size_type(), so only + // "_ArrayType_" and "_ArrayData_" remain to be added here. + if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) + { + size_and_type.second &= ~(static_cast(1) << 8); + auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), size_and_type.second, [](const bjd_type & p, char_int_type t) + { + return p.first < t; + }); + string_t key = "_ArrayType_"; + if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != size_and_type.second)) + { + auto last_token = get_token_string(); + sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr)); + return 0; + } + + string_t type = it->second; // sax->string() takes a reference + if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type))) + { + return 0; + } + + if (size_and_type.second == 'C' || size_and_type.second == 'B') + { + size_and_type.second = 'U'; + } + + key = "_ArrayData_"; + if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first))) + { + return 0; + } + + if (size_and_type.first == 0) + { + return (sax->end_array() && sax->end_object()) ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, true}); + return 1; + } + + // BJData type marker 'B': decode as binary (not a SAX array at all) + if (input_format == input_format_t::bjdata && size_and_type.first != npos && size_and_type.second == 'B') { - std::uint8_t len{}; - return get_number(input_format_t::msgpack, len) && - get_binary(input_format_t::msgpack, len, result); + binary_t result; + return (get_binary(input_format, size_and_type.first, result) && sax->binary(result)) ? 2 : 0; } - case 0xC5: // bin 16 + if (size_and_type.first != npos) { - std::uint16_t len{}; - return get_number(input_format_t::msgpack, len) && - get_binary(input_format_t::msgpack, len, result); + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) + { + return 0; + } + + // a homogeneous type of 'N' (no-op) means no elements are + // actually read, no matter what count was declared + if (size_and_type.second == 'N' || size_and_type.first == 0) + { + return sax->end_array() ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, false}); + return 1; } - case 0xC6: // bin 32 + // indefinite length + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) { - std::uint32_t len{}; - return get_number(input_format_t::msgpack, len) && - get_binary(input_format_t::msgpack, len, result); + return 0; } - - case 0xC7: // ext 8 + if (current == ']') { - std::uint8_t len{}; - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, len) && - get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, len, result) && - assign_and_return_true(subtype); + return sax->end_array() ? 2 : 0; } - - case 0xC8: // ext 16 + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - std::uint16_t len{}; - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, len) && - get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, len, result) && - assign_and_return_true(subtype); + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; } + stack.push_back(ubjson_container_frame{false, true, 0, false, 0, false}); + return 1; + }; - case 0xC9: // ext 32 + auto enter_object = [&]() -> int + { + std::pair size_and_type; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) { - std::uint32_t len{}; - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, len) && - get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, len, result) && - assign_and_return_true(subtype); + return 0; } - case 0xD4: // fixext 1 + // do not accept ND-array size in objects in BJData + if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) { - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, 1, result) && - assign_and_return_true(subtype); + auto last_token = get_token_string(); + sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, "BJData object does not support ND-array size in optimized format", "object"), nullptr)); + return 0; } - case 0xD5: // fixext 2 + if (size_and_type.first != npos) { - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, 2, result) && - assign_and_return_true(subtype); + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(size_and_type.first))) + { + return 0; + } + if (size_and_type.first == 0) + { + return sax->end_object() ? 2 : 0; + } + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; + } + stack.push_back(ubjson_container_frame{true, false, size_and_type.first, true, size_and_type.second, false}); + return 1; } - case 0xD6: // fixext 4 + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, 4, result) && - assign_and_return_true(subtype); + return 0; } - - case 0xD7: // fixext 8 + if (current == '}') { - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, 8, result) && - assign_and_return_true(subtype); + return sax->end_object() ? 2 : 0; } - - case 0xD8: // fixext 16 + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - std::int8_t subtype{}; - return get_number(input_format_t::msgpack, subtype) && - get_binary(input_format_t::msgpack, 16, result) && - assign_and_return_true(subtype); + sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, + exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + return 0; } + stack.push_back(ubjson_container_frame{true, true, 0, true, 0, false}); + return 1; + }; - default: // LCOV_EXCL_LINE - return false; // LCOV_EXCL_LINE - } - } +#define JSON_UBJSON_VALUE(expr) \ + if (JSON_HEDLEY_UNLIKELY(!(expr))) { return false; } \ + if (produce()) { return true; } \ + continue - /*! - @param[in] len the length of the array - @return whether array creation completed - */ - bool get_msgpack_array(const std::size_t len) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(len))) - { - return false; - } +#define JSON_UBJSON_ENTER(fn) \ + switch (fn()) \ + { \ + case 0: return false; \ + case 2: if (produce()) { return true; } continue; \ + default: continue; \ + } - for (std::size_t i = 0; i < len; ++i) + while (true) { - if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_internal())) + if (!stack.empty()) { - return false; - } - } + ubjson_container_frame& top = stack.back(); - return sax->end_array(); - } + if (top.is_object && top.awaiting_key) + { + const bool finished = top.indefinite ? (current == '}') : (top.remaining == 0); + if (finished) + { + if (JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + stack.pop_back(); + if (produce()) + { + return true; + } + continue; + } - /*! - @param[in] len the length of the object - @return whether object creation completed - */ - bool get_msgpack_object(const std::size_t len) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(len))) - { - return false; - } + string_t key; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key, !top.indefinite) || !sax->key(key))) + { + return false; + } + top.awaiting_key = false; + } + else if (!top.is_object) + { + const bool finished = top.indefinite ? (current == ']') : (top.remaining == 0); + if (finished) + { + if (JSON_HEDLEY_UNLIKELY(!sax->end_array())) + { + return false; + } + const bool extra_close = top.extra_close_object; + stack.pop_back(); + if (extra_close && JSON_HEDLEY_UNLIKELY(!sax->end_object())) + { + return false; + } + if (produce()) + { + return true; + } + continue; + } + } + // else: object frame with a key already read; fall through + // to read that key's value below. + } - string_t key; - for (std::size_t i = 0; i < len; ++i) - { - get(); - if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key))) + char_int_type prefix; + if (!stack.empty()) { - return false; + ubjson_container_frame& top = stack.back(); + if (top.fixed_type != 0) + { + prefix = top.fixed_type; + } + else if (!top.is_object && top.indefinite) + { + prefix = current; + } + else + { + prefix = get_ignore_noop(); + } } - - if (JSON_HEDLEY_UNLIKELY(!parse_msgpack_internal())) + else { - return false; + prefix = fetch_char ? get_ignore_noop() : current; } - key.clear(); - } - return sax->end_object(); - } + switch (prefix) + { + case char_traits::eof(): // EOF + return unexpect_eof(input_format, "value"); - //////////// - // UBJSON // - //////////// + case 'T': // true + JSON_UBJSON_VALUE(sax->boolean(true)); + case 'F': // false + JSON_UBJSON_VALUE(sax->boolean(false)); - /*! - @param[in] get_char whether a new character should be retrieved from the - input (true, default) or whether the last read - character should be considered instead + case 'Z': // null + JSON_UBJSON_VALUE(sax->null()); - @return whether a valid UBJSON value was passed to the SAX parser + case 'B': // byte + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint8_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } - @note Unlike CBOR/MessagePack, UBJSON needs get_ubjson_value() (which - carries the @ref depth_guard) to be reachable both from here *and* - directly with an already-known type marker (the optimized `$type#count` - container path), so the dispatch cannot simply be folded into this - function the way parse_cbor_internal()/parse_msgpack_internal() do. That - leaves this thin wrapper as a genuine extra stack frame on every ordinary - array/object element, on top of get_ubjson_value() and - get_ubjson_array()/get_ubjson_object() - one more frame per nesting level - than CBOR/MessagePack/BSON need for the same @ref max_depth. Force-inlining - it (rather than trusting Debug/-O0 builds to do so) keeps UBJSON's - recursion cost per level in line with the other formats' and was - necessary to stop deeply nested UBJSON input from overflowing the stack - well before max_depth is reached in MSVC Debug builds (#5104). - */ - JSON_HEDLEY_ALWAYS_INLINE - bool parse_ubjson_internal(const bool get_char = true) - { - return get_ubjson_value(get_char ? get_ignore_noop() : current); + case 'U': + { + std::uint8_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'i': + { + std::int8_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'I': + { + std::int16_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'l': + { + std::int32_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'L': + { + std::int64_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_integer(number)); + } + + case 'u': + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint16_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'm': + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint32_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'M': + { + if (input_format != input_format_t::bjdata) + { + break; + } + std::uint64_t number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_unsigned(number)); + } + + case 'h': + { + if (input_format != input_format_t::bjdata) + { + break; + } + const auto byte1_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + { + return false; + } + const auto byte2_raw = get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + { + return false; + } + + const auto byte1 = static_cast(byte1_raw); + const auto byte2 = static_cast(byte2_raw); + + // Code from RFC 7049, Appendix D, Figure 3: + // As half-precision floating-point numbers were only added + // to IEEE 754 in 2008, today's programming platforms often + // still only have limited support for them. It is very + // easy to include at least decoding support for them even + // without such support. An example of a small decoder for + // half-precision floating-point numbers in the C language + // is shown in Fig. 3. + const auto half = static_cast((byte2 << 8u) + byte1); + const double val = [&half] + { + const int exp = (half >> 10u) & 0x1Fu; + const unsigned int mant = half & 0x3FFu; + JSON_ASSERT(0 <= exp&& exp <= 32); + JSON_ASSERT(mant <= 1024); + switch (exp) + { + case 0: + return std::ldexp(mant, -24); + case 31: + return (mant == 0) + ? std::numeric_limits::infinity() + : std::numeric_limits::quiet_NaN(); + default: + return std::ldexp(mant + 1024, exp - 25); + } + }(); + JSON_UBJSON_VALUE(sax->number_float((half & 0x8000u) != 0 + ? static_cast(-val) + : static_cast(val), "")); + } + + case 'd': + { + float number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_float(static_cast(number), "")); + } + + case 'D': + { + double number{}; + JSON_UBJSON_VALUE(get_number(input_format, number) && sax->number_float(static_cast(number), "")); + } + + case 'H': + { + JSON_UBJSON_VALUE(get_ubjson_high_precision_number()); + } + + case 'C': // char + { + get(); + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "char"))) + { + return false; + } + if (JSON_HEDLEY_UNLIKELY(current > 127)) + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, + exception_message(input_format, concat("byte after 'C' must be in range 0x00..0x7F; last byte: 0x", last_token), "char"), nullptr)); + } + string_t s(1, static_cast(current)); + JSON_UBJSON_VALUE(sax->string(s)); + } + + case 'S': // string + { + string_t s; + JSON_UBJSON_VALUE(get_ubjson_string(s) && sax->string(s)); + } + + case '[': // array + JSON_UBJSON_ENTER(enter_array); + + case '{': // object + JSON_UBJSON_ENTER(enter_object); + + default: // anything else + break; + } + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, exception_message(input_format, "invalid byte: 0x" + last_token, "value"), nullptr)); + } + +#undef JSON_UBJSON_VALUE +#undef JSON_UBJSON_ENTER } /*! @@ -12768,594 +13405,194 @@ class binary_reader return true; } - case 'm': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint32_t number{}; - if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) - { - return false; - } - result = conditional_static_cast(number); - return true; - } - - case 'M': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint64_t number{}; - if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) - { - return false; - } - if (!value_in_range_of(number)) - { - return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, - exception_message(input_format, "integer value overflow", "size"), nullptr)); - } - result = detail::conditional_static_cast(number); - return true; - } - - case '[': - { - if (input_format != input_format_t::bjdata) - { - break; - } - if (is_ndarray) // ndarray dimensional vector can only contain integers and cannot embed another array - { - return sax->parse_error(chars_read, get_token_string(), parse_error::create(113, chars_read, exception_message(input_format, "ndarray dimensional vector is not allowed", "size"), nullptr)); - } - std::vector dim; - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_ndarray_size(dim))) - { - return false; - } - if (dim.size() == 1 || (dim.size() == 2 && dim.at(0) == 1)) // return normal array size if 1D row vector - { - result = dim.at(dim.size() - 1); - return true; - } - if (!dim.empty()) // if ndarray, convert to an object in JData annotated array format - { - for (auto i : dim) // test if any dimension in an ndarray is 0, if so, return a 1D empty container - { - if ( i == 0 ) - { - result = 0; - return true; - } - } - - string_t key = "_ArraySize_"; - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size()))) - { - return false; - } - result = 1; - for (auto i : dim) - { - // Pre-multiplication overflow check: if i > 0 and result > SIZE_MAX/i, then result*i would overflow. - // This check must happen before multiplication since overflow detection after the fact is unreliable - // as modular arithmetic can produce any value, not just 0 or SIZE_MAX. - if (JSON_HEDLEY_UNLIKELY(i > 0 && result > (std::numeric_limits::max)() / i)) - { - return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); - } - result *= i; - // Additional post-multiplication check to catch any edge cases the pre-check might miss - if (result == 0 || result == npos) - { - return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); - } - if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast(i)))) - { - return false; - } - } - is_ndarray = true; - return sax->end_array(); - } - result = 0; - return true; - } - - default: - break; - } - auto last_token = get_token_string(); - std::string message; - - if (input_format != input_format_t::bjdata) - { - message = "expected length type specification (U, i, I, l, L) after '#'; last byte: 0x" + last_token; - } - else - { - message = "expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x" + last_token; - } - return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, exception_message(input_format, message, "size"), nullptr)); - } - - /*! - @brief determine the type and size for a container - - In the optimized UBJSON format, a type and a size can be provided to allow - for a more compact representation. - - @param[out] result pair of the size and the type - @param[in] inside_ndarray whether the parser is parsing an ND array dimensional vector - - @return whether pair creation completed - */ - bool get_ubjson_size_type(std::pair& result, bool inside_ndarray = false) - { - result.first = npos; // size - result.second = 0; // type - bool is_ndarray = false; - - get_ignore_noop(); - - if (current == '$') - { - result.second = get(); // must not ignore 'N', because 'N' maybe the type - if (input_format == input_format_t::bjdata - && JSON_HEDLEY_UNLIKELY(std::binary_search(bjd_optimized_type_markers.begin(), bjd_optimized_type_markers.end(), result.second))) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, concat("marker 0x", last_token, " is not a permitted optimized array type"), "type"), nullptr)); - } - - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "type"))) - { - return false; - } - - get_ignore_noop(); - if (JSON_HEDLEY_UNLIKELY(current != '#')) - { - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "value"))) - { - return false; - } - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr)); - } - - const bool is_error = get_ubjson_size_value(result.first, is_ndarray); - if (input_format == input_format_t::bjdata && is_ndarray) - { - if (inside_ndarray) - { - return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, - exception_message(input_format, "ndarray can not be recursive", "size"), nullptr)); - } - result.second |= (1 << 8); // use bit 8 to indicate ndarray, all UBJSON and BJData markers should be ASCII letters - } - return is_error; - } - - if (current == '#') - { - const bool is_error = get_ubjson_size_value(result.first, is_ndarray); - if (input_format == input_format_t::bjdata && is_ndarray) - { - return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, - exception_message(input_format, "ndarray requires both type and size", "size"), nullptr)); - } - return is_error; - } - - return true; - } - - /*! - @param prefix the previously read or set type prefix - @return whether value creation completed - */ - bool get_ubjson_value(const char_int_type prefix) - { - const depth_guard dg(*this); - if (JSON_HEDLEY_UNLIKELY(depth > max_depth)) - { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - } - - switch (prefix) - { - case char_traits::eof(): // EOF - return unexpect_eof(input_format, "value"); - - case 'T': // true - return sax->boolean(true); - case 'F': // false - return sax->boolean(false); - - case 'Z': // null - return sax->null(); - - case 'B': // byte - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint8_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'U': - { - std::uint8_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'i': - { - std::int8_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'I': - { - std::int16_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'l': - { - std::int32_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'L': - { - std::int64_t number{}; - return get_number(input_format, number) && sax->number_integer(number); - } - - case 'u': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint16_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'm': - { - if (input_format != input_format_t::bjdata) - { - break; - } - std::uint32_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); - } - - case 'M': + case 'm': { if (input_format != input_format_t::bjdata) { break; } - std::uint64_t number{}; - return get_number(input_format, number) && sax->number_unsigned(number); + std::uint32_t number{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) + { + return false; + } + result = conditional_static_cast(number); + return true; } - case 'h': + case 'M': { if (input_format != input_format_t::bjdata) { break; } - const auto byte1_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + std::uint64_t number{}; + if (JSON_HEDLEY_UNLIKELY(!get_number(input_format, number))) { return false; } - const auto byte2_raw = get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "number"))) + if (!value_in_range_of(number)) { - return false; + return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, + exception_message(input_format, "integer value overflow", "size"), nullptr)); } - - const auto byte1 = static_cast(byte1_raw); - const auto byte2 = static_cast(byte2_raw); - - // Code from RFC 7049, Appendix D, Figure 3: - // As half-precision floating-point numbers were only added - // to IEEE 754 in 2008, today's programming platforms often - // still only have limited support for them. It is very - // easy to include at least decoding support for them even - // without such support. An example of a small decoder for - // half-precision floating-point numbers in the C language - // is shown in Fig. 3. - const auto half = static_cast((byte2 << 8u) + byte1); - const double val = [&half] - { - const int exp = (half >> 10u) & 0x1Fu; - const unsigned int mant = half & 0x3FFu; - JSON_ASSERT(0 <= exp&& exp <= 32); - JSON_ASSERT(mant <= 1024); - switch (exp) - { - case 0: - return std::ldexp(mant, -24); - case 31: - return (mant == 0) - ? std::numeric_limits::infinity() - : std::numeric_limits::quiet_NaN(); - default: - return std::ldexp(mant + 1024, exp - 25); - } - }(); - return sax->number_float((half & 0x8000u) != 0 - ? static_cast(-val) - : static_cast(val), ""); - } - - case 'd': - { - float number{}; - return get_number(input_format, number) && sax->number_float(static_cast(number), ""); - } - - case 'D': - { - double number{}; - return get_number(input_format, number) && sax->number_float(static_cast(number), ""); - } - - case 'H': - { - return get_ubjson_high_precision_number(); + result = detail::conditional_static_cast(number); + return true; } - case 'C': // char + case '[': { - get(); - if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "char"))) + if (input_format != input_format_t::bjdata) { - return false; + break; } - if (JSON_HEDLEY_UNLIKELY(current > 127)) + if (is_ndarray) // ndarray dimensional vector can only contain integers and cannot embed another array { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, - exception_message(input_format, concat("byte after 'C' must be in range 0x00..0x7F; last byte: 0x", last_token), "char"), nullptr)); + return sax->parse_error(chars_read, get_token_string(), parse_error::create(113, chars_read, exception_message(input_format, "ndarray dimensional vector is not allowed", "size"), nullptr)); } - string_t s(1, static_cast(current)); - return sax->string(s); - } - - case 'S': // string - { - string_t s; - return get_ubjson_string(s) && sax->string(s); - } - - case '[': // array - return get_ubjson_array(); - - case '{': // object - return get_ubjson_object(); - - default: // anything else - break; - } - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, exception_message(input_format, "invalid byte: 0x" + last_token, "value"), nullptr)); - } - - /*! - @return whether array creation completed - */ - bool get_ubjson_array() - { - std::pair size_and_type; - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) - { - return false; - } - - // if bit-8 of size_and_type.second is set to 1, encode bjdata ndarray as an object in JData annotated array format (https://github.com/NeuroJSON/jdata): - // {"_ArrayType_" : "typeid", "_ArraySize_" : [n1, n2, ...], "_ArrayData_" : [v1, v2, ...]} - - if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) - { - size_and_type.second &= ~(static_cast(1) << 8); // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker - auto it = std::lower_bound(bjd_types_map.begin(), bjd_types_map.end(), size_and_type.second, [](const bjd_type & p, char_int_type t) - { - return p.first < t; - }); - string_t key = "_ArrayType_"; - if (JSON_HEDLEY_UNLIKELY(it == bjd_types_map.end() || it->first != size_and_type.second)) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr)); - } - - string_t type = it->second; // sax->string() takes a reference - if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type))) - { - return false; - } - - if (size_and_type.second == 'C' || size_and_type.second == 'B') - { - size_and_type.second = 'U'; - } - - key = "_ArrayData_"; - if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) )) - { - return false; - } - - for (std::size_t i = 0; i < size_and_type.first; ++i) - { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_value(size_and_type.second))) + std::vector dim; + if (JSON_HEDLEY_UNLIKELY(!get_ubjson_ndarray_size(dim))) { return false; } - } - - return (sax->end_array() && sax->end_object()); - } - - // If BJData type marker is 'B' decode as binary - if (input_format == input_format_t::bjdata && size_and_type.first != npos && size_and_type.second == 'B') - { - binary_t result; - return get_binary(input_format, size_and_type.first, result) && sax->binary(result); - } - - if (size_and_type.first != npos) - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) - { - return false; - } - - if (size_and_type.second != 0) - { - if (size_and_type.second != 'N') + if (dim.size() == 1 || (dim.size() == 2 && dim.at(0) == 1)) // return normal array size if 1D row vector { - for (std::size_t i = 0; i < size_and_type.first; ++i) + result = dim.at(dim.size() - 1); + return true; + } + if (!dim.empty()) // if ndarray, convert to an object in JData annotated array format + { + for (auto i : dim) // test if any dimension in an ndarray is 0, if so, return a 1D empty container { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_value(size_and_type.second))) + if ( i == 0 ) { - return false; + result = 0; + return true; } } - } - } - else - { - for (std::size_t i = 0; i < size_and_type.first; ++i) - { - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal())) + + string_t key = "_ArraySize_"; + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size()))) { return false; } + result = 1; + for (auto i : dim) + { + // Pre-multiplication overflow check: if i > 0 and result > SIZE_MAX/i, then result*i would overflow. + // This check must happen before multiplication since overflow detection after the fact is unreliable + // as modular arithmetic can produce any value, not just 0 or SIZE_MAX. + if (JSON_HEDLEY_UNLIKELY(i > 0 && result > (std::numeric_limits::max)() / i)) + { + return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); + } + result *= i; + // Additional post-multiplication check to catch any edge cases the pre-check might miss + if (result == 0 || result == npos) + { + return sax->parse_error(chars_read, get_token_string(), out_of_range::create(408, exception_message(input_format, "excessive ndarray size caused overflow", "size"), nullptr)); + } + if (JSON_HEDLEY_UNLIKELY(!sax->number_unsigned(static_cast(i)))) + { + return false; + } + } + is_ndarray = true; + return sax->end_array(); } + result = 0; + return true; } + + default: + break; + } + auto last_token = get_token_string(); + std::string message; + + if (input_format != input_format_t::bjdata) + { + message = "expected length type specification (U, i, I, l, L) after '#'; last byte: 0x" + last_token; } else { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) - { - return false; - } - - while (current != ']') - { - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal(false))) - { - return false; - } - get_ignore_noop(); - } + message = "expected length type specification (U, i, u, I, m, l, M, L) after '#'; last byte: 0x" + last_token; } - - return sax->end_array(); + return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read, exception_message(input_format, message, "size"), nullptr)); } /*! - @return whether object creation completed + @brief determine the type and size for a container + + In the optimized UBJSON format, a type and a size can be provided to allow + for a more compact representation. + + @param[out] result pair of the size and the type + @param[in] inside_ndarray whether the parser is parsing an ND array dimensional vector + + @return whether pair creation completed */ - bool get_ubjson_object() + bool get_ubjson_size_type(std::pair& result, bool inside_ndarray = false) { - std::pair size_and_type; - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_size_type(size_and_type))) - { - return false; - } + result.first = npos; // size + result.second = 0; // type + bool is_ndarray = false; - // do not accept ND-array size in objects in BJData - if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0) - { - auto last_token = get_token_string(); - return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, - exception_message(input_format, "BJData object does not support ND-array size in optimized format", "object"), nullptr)); - } + get_ignore_noop(); - string_t key; - if (size_and_type.first != npos) + if (current == '$') { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(size_and_type.first))) + result.second = get(); // must not ignore 'N', because 'N' maybe the type + if (input_format == input_format_t::bjdata + && JSON_HEDLEY_UNLIKELY(std::binary_search(bjd_optimized_type_markers.begin(), bjd_optimized_type_markers.end(), result.second))) + { + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, concat("marker 0x", last_token, " is not a permitted optimized array type"), "type"), nullptr)); + } + + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "type"))) { return false; } - if (size_and_type.second != 0) + get_ignore_noop(); + if (JSON_HEDLEY_UNLIKELY(current != '#')) { - for (std::size_t i = 0; i < size_and_type.first; ++i) + if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "value"))) { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key) || !sax->key(key))) - { - return false; - } - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_value(size_and_type.second))) - { - return false; - } - key.clear(); + return false; } + auto last_token = get_token_string(); + return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read, + exception_message(input_format, concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr)); } - else + + const bool is_error = get_ubjson_size_value(result.first, is_ndarray); + if (input_format == input_format_t::bjdata && is_ndarray) { - for (std::size_t i = 0; i < size_and_type.first; ++i) + if (inside_ndarray) { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key) || !sax->key(key))) - { - return false; - } - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal())) - { - return false; - } - key.clear(); + return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, + exception_message(input_format, "ndarray can not be recursive", "size"), nullptr)); } + result.second |= (1 << 8); // use bit 8 to indicate ndarray, all UBJSON and BJData markers should be ASCII letters } + return is_error; } - else - { - if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) - { - return false; - } - while (current != '}') + if (current == '#') + { + const bool is_error = get_ubjson_size_value(result.first, is_ndarray); + if (input_format == input_format_t::bjdata && is_ndarray) { - if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key, false) || !sax->key(key))) - { - return false; - } - if (JSON_HEDLEY_UNLIKELY(!parse_ubjson_internal())) - { - return false; - } - get_ignore_noop(); - key.clear(); + return sax->parse_error(chars_read, get_token_string(), parse_error::create(112, chars_read, + exception_message(input_format, "ndarray requires both type and size", "size"), nullptr)); } + return is_error; } - return sax->end_object(); + return true; } // Note, no reader for UBJSON binary types is implemented because they do @@ -13716,19 +13953,19 @@ class binary_reader static JSON_INLINE_VARIABLE constexpr std::size_t npos = detail::unknown_size(); /// maximum allowed nesting depth of arrays/objects for the binary input - /// formats; guards the recursive-descent parsers against stack overflow - /// on maliciously/accidentally deeply nested input. + /// formats. /// - /// This has to satisfy two constraints in tension: high enough that it - /// doesn't reject legitimate input (the library's own roundtrip test - /// fixture, tests/data/json_testsuite/sample.json, nests 458 levels - /// deep), but low enough that reaching it doesn't itself exhaust the - /// native call stack - each nesting level costs multiple real call - /// frames, and unoptimized/Debug builds (notably MSVC Debug, which - /// crashed CI at max_depth=1024 - see #5104) use substantially more - /// stack per frame than an optimized build. 600 leaves comfortable - /// margin on both sides of that range. - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 600; + /// The container-parsing loops in this file are iterative (see e.g. + /// @ref cbor_container_frame): nesting depth is tracked as the size of a + /// heap-allocated std::vector, not native call-stack recursion, so this + /// limit is no longer a stack-safety mechanism - arbitrarily deep input + /// can no longer overflow the stack regardless of this value. It is kept + /// purely as a sanity/DoS cap on absurd inputs (e.g. a malicious payload + /// nesting billions of levels deep to exhaust memory/time), so it can + /// afford to be generous; it is far above the library's own deepest + /// legitimate test fixture (tests/data/json_testsuite/sample.json, 458 + /// levels). + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 100000; /// input adapter InputAdapterType ia; @@ -13739,9 +13976,6 @@ class binary_reader /// the number of characters read std::size_t chars_read = 0; - /// current container nesting depth, see @ref max_depth and @ref depth_guard - std::size_t depth = 0; - /// whether we can assume little endianness const bool is_little_endian = little_endianness(); diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index 55952caffa..6699f5229c 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -858,34 +858,104 @@ TEST_CASE("BSON regressions") { SECTION("stack overflow via deeply nested input (issue #5104)") { - // A chain of 1100 embedded BSON documents (record type 0x03), each - // wrapping the previous one under key "a". The recursive-descent - // parser must reject this with a clean parse_error once the nesting - // exceeds the depth limit, rather than exhausting the native call - // stack. - std::vector inner = {5, 0, 0, 0, 0}; // innermost empty document - for (int i = 0; i < 1100; ++i) + // A chain of 100010 embedded BSON documents (record type 0x03), each + // wrapping the previous one under key "a" - comfortably past + // max_depth (100000) and far beyond any depth a recursive-descent + // parser could ever have survived. The (now iterative, heap-stack + // based) parser must reject this with a clean parse_error once the + // nesting exceeds the depth limit, rather than exhausting the native + // call stack. + // + // Built directly in O(N) rather than by repeatedly wrapping and + // copying the previous document (which would be O(N^2) and much too + // slow at this depth). Layout, front to back: + // [header_N]...[header_1][innermost 5-byte empty doc][term_1]...[term_N] + // where header_i is a 4-byte little-endian size_i followed by + // {0x03,'a',0x00}, and size_i = size_{i-1} + 8, size_0 = 5. + // Terminators are all 0x00, already the buffer's default value. + const int n = 100010; + const std::size_t header_size = 7; // 4 (size) + 3 (0x03, 'a', 0x00) + const std::size_t inner_size = 5; + const std::size_t total = static_cast(n) * header_size + inner_size + static_cast(n); + std::vector inner(total, 0x00); + + std::vector size_at(static_cast(n) + 1); + size_at[0] = static_cast(inner_size); + for (int i = 1; i <= n; ++i) { - std::vector elem = {0x03, 'a', 0x00}; // embedded-document element, key "a" - elem.insert(elem.end(), inner.begin(), inner.end()); + size_at[static_cast(i)] = size_at[static_cast(i) - 1] + 8; + } - const auto doc_size = static_cast(4 + elem.size() + 1); - std::vector doc = - { - static_cast(doc_size & 0xFF), - static_cast((doc_size >> 8) & 0xFF), - static_cast((doc_size >> 16) & 0xFF), - static_cast((doc_size >> 24) & 0xFF), - }; - doc.insert(doc.end(), elem.begin(), elem.end()); - doc.push_back(0x00); // terminator + std::size_t pos = 0; + for (int i = n; i >= 1; --i) + { + const auto sz = static_cast(size_at[static_cast(i)]); + inner[pos + 0] = static_cast(sz & 0xFF); + inner[pos + 1] = static_cast((sz >> 8) & 0xFF); + inner[pos + 2] = static_cast((sz >> 16) & 0xFF); + inner[pos + 3] = static_cast((sz >> 24) & 0xFF); + inner[pos + 4] = 0x03; + inner[pos + 5] = 'a'; + inner[pos + 6] = 0x00; + pos += header_size; + } + // innermost empty document: size=5 (little-endian) + terminator 0x00 + inner[pos + 0] = 5; + inner[pos + 1] = 0; + inner[pos + 2] = 0; + inner[pos + 3] = 0; + inner[pos + 4] = 0x00; + // trailing n terminator bytes are already 0x00 from initialization + + json _; + CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), + "[json.exception.parse_error.116] parse error at byte 700004: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + json::parse_error&); + CHECK(json::from_bson(inner, true, false).is_discarded()); + } + + SECTION("array nesting depth is also capped (issue #5104)") + { + // The original depth guard only protected nested *objects* (record + // type 0x03); nested *arrays* (record type 0x04) went through a + // separate code path with no depth check of their own at all. The + // iterative rewrite unifies object/array container handling into one + // loop, so this gap is now closed too - verify it here explicitly. + const int n = 100010; + const std::size_t header_size = 7; // 4 (size) + 3 (0x04, '0', 0x00) + const std::size_t inner_size = 5; + const std::size_t total = static_cast(n) * header_size + inner_size + static_cast(n); + std::vector inner(total, 0x00); + + std::vector size_at(static_cast(n) + 1); + size_at[0] = static_cast(inner_size); + for (int i = 1; i <= n; ++i) + { + size_at[static_cast(i)] = size_at[static_cast(i) - 1] + 8; + } - inner = doc; + std::size_t pos = 0; + for (int i = n; i >= 1; --i) + { + const auto sz = static_cast(size_at[static_cast(i)]); + inner[pos + 0] = static_cast(sz & 0xFF); + inner[pos + 1] = static_cast((sz >> 8) & 0xFF); + inner[pos + 2] = static_cast((sz >> 16) & 0xFF); + inner[pos + 3] = static_cast((sz >> 24) & 0xFF); + inner[pos + 4] = 0x04; // array element + inner[pos + 5] = '0'; + inner[pos + 6] = 0x00; + pos += header_size; } + inner[pos + 0] = 5; + inner[pos + 1] = 0; + inner[pos + 2] = 0; + inner[pos + 3] = 0; + inner[pos + 4] = 0x00; json _; CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), - "[json.exception.parse_error.116] parse error at byte 4200: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + "[json.exception.parse_error.116] parse error at byte 700004: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", json::parse_error&); CHECK(json::from_bson(inner, true, false).is_discarded()); } diff --git a/tests/src/unit-cbor.cpp b/tests/src/unit-cbor.cpp index b1e96c8cf4..f284c282d2 100644 --- a/tests/src/unit-cbor.cpp +++ b/tests/src/unit-cbor.cpp @@ -1999,14 +1999,17 @@ TEST_CASE("CBOR regressions") SECTION("stack overflow via deeply nested input (issue #5104)") { - // 2000 nested CBOR arrays of length 1 (0x81 each); the recursive-descent - // parser must reject this with a clean parse_error once the nesting - // exceeds the depth limit, rather than exhausting the native call stack. - std::vector vec(2000, 0x81); + // 100010 nested CBOR arrays of length 1 (0x81 each) - comfortably past + // max_depth (100000) and far beyond any depth a recursive-descent + // parser could ever have survived; the (now iterative, heap-stack + // based) parser must reject this with a clean parse_error once the + // nesting exceeds the depth limit, rather than exhausting the native + // call stack. + std::vector vec(100010, 0x81); vec.push_back(0x00); json _; // NOLINT(readability-identifier-naming) CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), - "[json.exception.parse_error.116] parse error at byte 600: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 100001: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_cbor(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-msgpack.cpp b/tests/src/unit-msgpack.cpp index 56485ee493..17c65aa0c7 100644 --- a/tests/src/unit-msgpack.cpp +++ b/tests/src/unit-msgpack.cpp @@ -1573,14 +1573,17 @@ TEST_CASE("MessagePack") SECTION("stack overflow via deeply nested input (issue #5104)") { - // 2000 nested fixarrays of length 1 (0x91 each); the recursive-descent - // parser must reject this with a clean parse_error once the nesting - // exceeds the depth limit, rather than exhausting the native call stack. - std::vector vec(2000, 0x91); + // 100010 nested fixarrays of length 1 (0x91 each) - comfortably past + // max_depth (100000) and far beyond any depth a recursive-descent + // parser could ever have survived; the (now iterative, heap-stack + // based) parser must reject this with a clean parse_error once the + // nesting exceeds the depth limit, rather than exhausting the + // native call stack. + std::vector vec(100010, 0x91); vec.push_back(0x00); json _; CHECK_THROWS_WITH_AS(_ = json::from_msgpack(vec), - "[json.exception.parse_error.116] parse error at byte 600: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 100001: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_msgpack(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-ubjson.cpp b/tests/src/unit-ubjson.cpp index 69da1e571b..9dece8b533 100644 --- a/tests/src/unit-ubjson.cpp +++ b/tests/src/unit-ubjson.cpp @@ -1641,19 +1641,22 @@ TEST_CASE("UBJSON") SECTION("stack overflow via deeply nested input (issue #5104)") { - // 2000 nested UBJSON arrays; the recursive-descent parser must reject - // this with a clean parse_error once the nesting exceeds the depth - // limit, rather than exhausting the native call stack. - std::string payload(2000, '['); + // 100010 nested UBJSON arrays - comfortably past max_depth (100000) + // and far beyond any depth a recursive-descent parser could ever + // have survived; the (now iterative, heap-stack based) parser must + // reject this with a clean parse_error once the nesting exceeds the + // depth limit, rather than exhausting the native call stack. + const int nesting = 100010; + std::string payload(static_cast(nesting), '['); payload += "Z"; - for (int i = 0; i < 2000; ++i) + for (int i = 0; i < nesting; ++i) { payload += ']'; } std::vector const v(payload.begin(), payload.end()); json _; CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), - "[json.exception.parse_error.116] parse error at byte 601: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 100002: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_ubjson(v, true, false).is_discarded()); } From a0b8940eaf969031d81d9f85801c2e93146fe78d Mon Sep 17 00:00:00 2001 From: Kyue Date: Thu, 23 Jul 2026 16:27:35 +0100 Subject: [PATCH 07/14] Fix clang-tidy nested-ternary and const-correctness warnings in binary_reader Signed-off-by: Kyue --- include/nlohmann/detail/input/binary_reader.hpp | 8 +++++--- single_include/nlohmann/json.hpp | 8 +++++--- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index b3a1dcfc97..cf66239f73 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -540,7 +540,8 @@ class binary_reader } if (len == 0) { - return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; + const bool ended = is_object ? sax->end_object() : sax->end_array(); + return ended ? 2 : 0; } if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { @@ -1370,7 +1371,8 @@ class binary_reader } if (len == 0) { - return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; + const bool ended = is_object ? sax->end_object() : sax->end_array(); + return ended ? 2 : 0; } if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { @@ -2072,7 +2074,7 @@ class binary_reader bool parse_ubjson_internal(const bool get_char = true) { std::vector stack; - bool fetch_char = get_char; + const bool fetch_char = get_char; auto produce = [&]() -> bool { diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 11f88b60f6..cac0875e5c 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -11089,7 +11089,8 @@ class binary_reader } if (len == 0) { - return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; + const bool ended = is_object ? sax->end_object() : sax->end_array(); + return ended ? 2 : 0; } if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { @@ -11919,7 +11920,8 @@ class binary_reader } if (len == 0) { - return (is_object ? sax->end_object() : sax->end_array()) ? 2 : 0; + const bool ended = is_object ? sax->end_object() : sax->end_array(); + return ended ? 2 : 0; } if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { @@ -12621,7 +12623,7 @@ class binary_reader bool parse_ubjson_internal(const bool get_char = true) { std::vector stack; - bool fetch_char = get_char; + const bool fetch_char = get_char; auto produce = [&]() -> bool { From 2fa472ff38268b5a0d7239742bd4e3983d41ed35 Mon Sep 17 00:00:00 2001 From: Kyue Date: Thu, 23 Jul 2026 17:52:12 +0100 Subject: [PATCH 08/14] Fix remaining clang-tidy const-reference and precedence warnings Signed-off-by: Kyue --- include/nlohmann/detail/input/binary_reader.hpp | 4 ++-- single_include/nlohmann/json.hpp | 4 ++-- tests/src/unit-bson.cpp | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index cf66239f73..c45d4acf96 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -208,7 +208,7 @@ class binary_reader while (true) { - bson_container_frame& top = stack.back(); + const bson_container_frame& top = stack.back(); const auto element_type = get(); if (element_type == 0) @@ -2346,7 +2346,7 @@ class binary_reader char_int_type prefix; if (!stack.empty()) { - ubjson_container_frame& top = stack.back(); + const ubjson_container_frame& top = stack.back(); if (top.fixed_type != 0) { prefix = top.fixed_type; diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index cac0875e5c..8e4b5d1537 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -10757,7 +10757,7 @@ class binary_reader while (true) { - bson_container_frame& top = stack.back(); + const bson_container_frame& top = stack.back(); const auto element_type = get(); if (element_type == 0) @@ -12895,7 +12895,7 @@ class binary_reader char_int_type prefix; if (!stack.empty()) { - ubjson_container_frame& top = stack.back(); + const ubjson_container_frame& top = stack.back(); if (top.fixed_type != 0) { prefix = top.fixed_type; diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index 6699f5229c..eb4aefd8dd 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -876,7 +876,7 @@ TEST_CASE("BSON regressions") const int n = 100010; const std::size_t header_size = 7; // 4 (size) + 3 (0x03, 'a', 0x00) const std::size_t inner_size = 5; - const std::size_t total = static_cast(n) * header_size + inner_size + static_cast(n); + const std::size_t total = (static_cast(n) * header_size) + inner_size + static_cast(n); std::vector inner(total, 0x00); std::vector size_at(static_cast(n) + 1); @@ -924,7 +924,7 @@ TEST_CASE("BSON regressions") const int n = 100010; const std::size_t header_size = 7; // 4 (size) + 3 (0x04, '0', 0x00) const std::size_t inner_size = 5; - const std::size_t total = static_cast(n) * header_size + inner_size + static_cast(n); + const std::size_t total = (static_cast(n) * header_size) + inner_size + static_cast(n); std::vector inner(total, 0x00); std::vector size_at(static_cast(n) + 1); From a47902ead4b0ad717b76321a62fbc58f82c69199 Mon Sep 17 00:00:00 2001 From: Kyue <164024549+Gooh456@users.noreply.github.com> Date: Fri, 24 Jul 2026 13:44:20 +0100 Subject: [PATCH 09/14] Move result instead of copying on binary parse return Copying a deeply-nested basic_json on return is exactly as recursive as the stack overflow this PR is meant to fix, since basic_json's copy constructor recurses over the tree. Use std::move instead across all from_cbor/from_msgpack/from_ubjson/from_bjdata/from_bson overloads. Also add a CI-only workaround for the MinGW clang linker failing with "relocation truncated to fit" on test-regression2, which appears to be tipped over by this PR's added instantiated code (-fuse-ld=lld). Signed-off-by: Kyue <164024549+Gooh456@users.noreply.github.com> --- include/nlohmann/json.hpp | 28 +++--- single_include/nlohmann/json.hpp | 154 +++++++++++++++---------------- tests/CMakeLists.txt | 9 ++ 3 files changed, 100 insertions(+), 91 deletions(-) diff --git a/include/nlohmann/json.hpp b/include/nlohmann/json.hpp index a460bb29ff..d2ee375f6d 100644 --- a/include/nlohmann/json.hpp +++ b/include/nlohmann/json.hpp @@ -4461,7 +4461,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in CBOR format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -4478,7 +4478,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -4504,7 +4504,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in MessagePack format @@ -4519,7 +4519,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in MessagePack format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -4535,7 +4535,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -4559,7 +4559,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in UBJSON format @@ -4574,7 +4574,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in UBJSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -4590,7 +4590,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -4614,7 +4614,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BJData format @@ -4629,7 +4629,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BJData format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -4645,7 +4645,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BSON format @@ -4660,7 +4660,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -4676,7 +4676,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -4700,7 +4700,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @} diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index 8e4b5d1537..ebc1cf3c30 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -3722,71 +3722,71 @@ NLOHMANN_JSON_NAMESPACE_END // SPDX-License-Identifier: MIT #ifndef INCLUDE_NLOHMANN_JSON_FWD_HPP_ - #define INCLUDE_NLOHMANN_JSON_FWD_HPP_ +#define INCLUDE_NLOHMANN_JSON_FWD_HPP_ - #include // int64_t, uint64_t - #include // map - #include // allocator - #include // string - #include // vector +#include // int64_t, uint64_t +#include // map +#include // allocator +#include // string +#include // vector - // #include +// #include - /*! - @brief namespace for Niels Lohmann - @see https://github.com/nlohmann - @since version 1.0.0 - */ - NLOHMANN_JSON_NAMESPACE_BEGIN +/*! +@brief namespace for Niels Lohmann +@see https://github.com/nlohmann +@since version 1.0.0 +*/ +NLOHMANN_JSON_NAMESPACE_BEGIN - /*! - @brief default JSONSerializer template argument +/*! +@brief default JSONSerializer template argument - This serializer ignores the template arguments and uses ADL - ([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) - for serialization. - */ - template - struct adl_serializer; - - /// a class to store JSON values - /// @sa https://json.nlohmann.me/api/basic_json/ - template class ObjectType = - std::map, - template class ArrayType = std::vector, - class StringType = std::string, class BooleanType = bool, - class NumberIntegerType = std::int64_t, - class NumberUnsignedType = std::uint64_t, - class NumberFloatType = double, - template class AllocatorType = std::allocator, - template class JSONSerializer = - adl_serializer, - class BinaryType = std::vector, // cppcheck-suppress syntaxError - class CustomBaseClass = void> - class basic_json; - - /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document - /// @sa https://json.nlohmann.me/api/json_pointer/ - template - class json_pointer; +This serializer ignores the template arguments and uses ADL +([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) +for serialization. +*/ +template +struct adl_serializer; + +/// a class to store JSON values +/// @sa https://json.nlohmann.me/api/basic_json/ +template class ObjectType = + std::map, + template class ArrayType = std::vector, + class StringType = std::string, class BooleanType = bool, + class NumberIntegerType = std::int64_t, + class NumberUnsignedType = std::uint64_t, + class NumberFloatType = double, + template class AllocatorType = std::allocator, + template class JSONSerializer = + adl_serializer, + class BinaryType = std::vector, // cppcheck-suppress syntaxError + class CustomBaseClass = void> +class basic_json; - /*! - @brief default specialization - @sa https://json.nlohmann.me/api/json/ - */ - using json = basic_json<>; +/// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document +/// @sa https://json.nlohmann.me/api/json_pointer/ +template +class json_pointer; - /// @brief a minimal map-like container that preserves insertion order - /// @sa https://json.nlohmann.me/api/ordered_map/ - template - struct ordered_map; +/*! +@brief default specialization +@sa https://json.nlohmann.me/api/json/ +*/ +using json = basic_json<>; - /// @brief specialization that maintains the insertion order of object keys - /// @sa https://json.nlohmann.me/api/ordered_json/ - using ordered_json = basic_json; +/// @brief a minimal map-like container that preserves insertion order +/// @sa https://json.nlohmann.me/api/ordered_map/ +template +struct ordered_map; - NLOHMANN_JSON_NAMESPACE_END +/// @brief specialization that maintains the insertion order of object keys +/// @sa https://json.nlohmann.me/api/ordered_json/ +using ordered_json = basic_json; + +NLOHMANN_JSON_NAMESPACE_END #endif // INCLUDE_NLOHMANN_JSON_FWD_HPP_ @@ -5857,7 +5857,7 @@ NLOHMANN_JSON_NAMESPACE_END // #include -// JSON_HAS_CPP_17 + // JSON_HAS_CPP_17 #ifdef JSON_HAS_CPP_17 #include // optional #endif @@ -21591,10 +21591,10 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bool allow_exceptions = true, const bool ignore_comments = false, const bool ignore_trailing_commas = false - ) + ) { return ::nlohmann::detail::parser(std::move(adapter), - std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); + std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); } private: @@ -22292,8 +22292,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::enable_if_t < !detail::is_basic_json::value && detail::is_compatible_type::value, int > = 0 > basic_json(CompatibleType && val) noexcept(noexcept( // NOLINT(bugprone-forwarding-reference-overload,bugprone-exception-escape) - JSONSerializer::to_json(std::declval(), - std::forward(val)))) + JSONSerializer::to_json(std::declval(), + std::forward(val)))) { JSONSerializer::to_json(*this, std::forward(val)); set_parents(); @@ -23096,7 +23096,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<0> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), std::declval()))) + JSONSerializer::from_json(std::declval(), std::declval()))) { auto ret = ValueType(); JSONSerializer::from_json(*this, ret); @@ -23138,7 +23138,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_non_default_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<1> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval()))) + JSONSerializer::from_json(std::declval()))) { return JSONSerializer::from_json(*this); } @@ -23288,7 +23288,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType & get_to(ValueType& v) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), v))) + JSONSerializer::from_json(std::declval(), v))) { JSONSerializer::from_json(*this, v); return v; @@ -25887,7 +25887,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in CBOR format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -25904,7 +25904,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -25930,7 +25930,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::cbor).sax_parse(input_format_t::cbor, &sdp, strict, tag_handler); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in MessagePack format @@ -25945,7 +25945,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in MessagePack format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -25961,7 +25961,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -25985,7 +25985,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::msgpack).sax_parse(input_format_t::msgpack, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in UBJSON format @@ -26000,7 +26000,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in UBJSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -26016,7 +26016,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -26040,7 +26040,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::ubjson).sax_parse(input_format_t::ubjson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BJData format @@ -26055,7 +26055,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BJData format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -26071,7 +26071,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bjdata).sax_parse(input_format_t::bjdata, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BSON format @@ -26086,7 +26086,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::forward(i)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @brief create a JSON value from an input in BSON format (iterator pair, or iterator+sentinel pair for C++20 ranges support) @@ -26102,7 +26102,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec auto ia = detail::input_adapter(std::move(first), std::move(last)); detail::json_sax_dom_parser sdp(result, allow_exceptions); const bool res = binary_reader(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } template @@ -26126,7 +26126,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::json_sax_dom_parser sdp(result, allow_exceptions); // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) const bool res = binary_reader(std::move(ia), input_format_t::bson).sax_parse(input_format_t::bson, &sdp, strict); // cppcheck-suppress[accessMoved] - return res ? result : basic_json(value_t::discarded); + return res ? std::move(result) : basic_json(value_t::discarded); } /// @} diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 4383b582c3..55bb6f074c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -104,6 +104,15 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") json_test_set_test_options("test-cbor;test-msgpack;test-ubjson;test-bjdata;test-binary_formats" LINK_OPTIONS /STACK:4000000) endif() +if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang" AND MINGW) + # test-regression2 is already the largest test TU and exercises CBOR/BSON/ + # MessagePack/UBJSON parsing; GNU ld's COFF backend can exhaust addressable + # range for IMAGE_REL_AMD64_REL32 relocations against .rdata once template + # instantiation grows far enough ("relocation truncated to fit"). lld lays + # out COFF sections differently and doesn't hit this ceiling for the same input. + json_test_set_test_options(test-regression2 LINK_OPTIONS -fuse-ld=lld) +endif() + # disable exceptions for test-disabled_exceptions json_test_set_test_options(test-disabled_exceptions COMPILE_DEFINITIONS From bc8a9cb1a28e79e7d935aee3477b93afc5514fd9 Mon Sep 17 00:00:00 2001 From: Kyue <164024549+Gooh456@users.noreply.github.com> Date: Fri, 24 Jul 2026 21:02:51 +0100 Subject: [PATCH 10/14] reamalgamate after merge Signed-off-by: Kyue <164024549+Gooh456@users.noreply.github.com> --- single_include/nlohmann/json.hpp | 126 +++++++++++++++---------------- 1 file changed, 63 insertions(+), 63 deletions(-) diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index dda9fdca36..ec0a65ce9d 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -3722,71 +3722,71 @@ NLOHMANN_JSON_NAMESPACE_END // SPDX-License-Identifier: MIT #ifndef INCLUDE_NLOHMANN_JSON_FWD_HPP_ -#define INCLUDE_NLOHMANN_JSON_FWD_HPP_ + #define INCLUDE_NLOHMANN_JSON_FWD_HPP_ -#include // int64_t, uint64_t -#include // map -#include // allocator -#include // string -#include // vector - -// #include + #include // int64_t, uint64_t + #include // map + #include // allocator + #include // string + #include // vector + // #include -/*! -@brief namespace for Niels Lohmann -@see https://github.com/nlohmann -@since version 1.0.0 -*/ -NLOHMANN_JSON_NAMESPACE_BEGIN -/*! -@brief default JSONSerializer template argument + /*! + @brief namespace for Niels Lohmann + @see https://github.com/nlohmann + @since version 1.0.0 + */ + NLOHMANN_JSON_NAMESPACE_BEGIN -This serializer ignores the template arguments and uses ADL -([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) -for serialization. -*/ -template -struct adl_serializer; - -/// a class to store JSON values -/// @sa https://json.nlohmann.me/api/basic_json/ -template class ObjectType = - std::map, - template class ArrayType = std::vector, - class StringType = std::string, class BooleanType = bool, - class NumberIntegerType = std::int64_t, - class NumberUnsignedType = std::uint64_t, - class NumberFloatType = double, - template class AllocatorType = std::allocator, - template class JSONSerializer = - adl_serializer, - class BinaryType = std::vector, // cppcheck-suppress syntaxError - class CustomBaseClass = void> -class basic_json; + /*! + @brief default JSONSerializer template argument -/// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document -/// @sa https://json.nlohmann.me/api/json_pointer/ -template -class json_pointer; + This serializer ignores the template arguments and uses ADL + ([argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)) + for serialization. + */ + template + struct adl_serializer; + + /// a class to store JSON values + /// @sa https://json.nlohmann.me/api/basic_json/ + template class ObjectType = + std::map, + template class ArrayType = std::vector, + class StringType = std::string, class BooleanType = bool, + class NumberIntegerType = std::int64_t, + class NumberUnsignedType = std::uint64_t, + class NumberFloatType = double, + template class AllocatorType = std::allocator, + template class JSONSerializer = + adl_serializer, + class BinaryType = std::vector, // cppcheck-suppress syntaxError + class CustomBaseClass = void> + class basic_json; + + /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document + /// @sa https://json.nlohmann.me/api/json_pointer/ + template + class json_pointer; -/*! -@brief default specialization -@sa https://json.nlohmann.me/api/json/ -*/ -using json = basic_json<>; + /*! + @brief default specialization + @sa https://json.nlohmann.me/api/json/ + */ + using json = basic_json<>; -/// @brief a minimal map-like container that preserves insertion order -/// @sa https://json.nlohmann.me/api/ordered_map/ -template -struct ordered_map; + /// @brief a minimal map-like container that preserves insertion order + /// @sa https://json.nlohmann.me/api/ordered_map/ + template + struct ordered_map; -/// @brief specialization that maintains the insertion order of object keys -/// @sa https://json.nlohmann.me/api/ordered_json/ -using ordered_json = basic_json; + /// @brief specialization that maintains the insertion order of object keys + /// @sa https://json.nlohmann.me/api/ordered_json/ + using ordered_json = basic_json; -NLOHMANN_JSON_NAMESPACE_END + NLOHMANN_JSON_NAMESPACE_END #endif // INCLUDE_NLOHMANN_JSON_FWD_HPP_ @@ -5857,7 +5857,7 @@ NLOHMANN_JSON_NAMESPACE_END // #include - // JSON_HAS_CPP_17 +// JSON_HAS_CPP_17 #ifdef JSON_HAS_CPP_17 #include // optional #endif @@ -21632,10 +21632,10 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec const bool allow_exceptions = true, const bool ignore_comments = false, const bool ignore_trailing_commas = false - ) + ) { return ::nlohmann::detail::parser(std::move(adapter), - std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); + std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); } private: @@ -22333,8 +22333,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::enable_if_t < !detail::is_basic_json::value && detail::is_compatible_type::value, int > = 0 > basic_json(CompatibleType && val) noexcept(noexcept( // NOLINT(bugprone-forwarding-reference-overload,bugprone-exception-escape) - JSONSerializer::to_json(std::declval(), - std::forward(val)))) + JSONSerializer::to_json(std::declval(), + std::forward(val)))) { JSONSerializer::to_json(*this, std::forward(val)); set_parents(); @@ -23137,7 +23137,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<0> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), std::declval()))) + JSONSerializer::from_json(std::declval(), std::declval()))) { auto ret = ValueType(); JSONSerializer::from_json(*this, ret); @@ -23179,7 +23179,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_non_default_from_json::value, int > = 0 > ValueType get_impl(detail::priority_tag<1> /*unused*/) const noexcept(noexcept( - JSONSerializer::from_json(std::declval()))) + JSONSerializer::from_json(std::declval()))) { return JSONSerializer::from_json(*this); } @@ -23329,7 +23329,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec detail::has_from_json::value, int > = 0 > ValueType & get_to(ValueType& v) const noexcept(noexcept( - JSONSerializer::from_json(std::declval(), v))) + JSONSerializer::from_json(std::declval(), v))) { JSONSerializer::from_json(*this, v); return v; From f748671e16c0d911f9635747e2d218bc0f12345f Mon Sep 17 00:00:00 2001 From: Gooh456 <164024549+Gooh456@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:09:27 +0100 Subject: [PATCH 11/14] Address review: shared reached_max_depth() helper, lower max_depth to 4096 Nine near-identical depth-check blocks (2 BSON, 1 CBOR, 1 MessagePack, 5 UBJSON) collapse into one reached_max_depth(format) helper, which also fixes the BSON "objects/arrays" vs the other formats' "arrays/objects" wording mismatch by construction. Every depth check now runs before the corresponding SAX start_object()/start_array() event fires, instead of after - a custom SAX consumer no longer sees an unmatched start immediately before the parse_error. BSON's begin_document() is split into a byte-consuming read_document_header() and a separate SAX-emitting step so the depth check can sit between them; CBOR/MessagePack/UBJSON's enter_container()/ enter_array()/enter_object() are reordered the same way, using the fact that the "did we actually push a frame" condition (len != 0, or current != the container's close marker for indefinite/BSON-terminated forms) is already known before the SAX call. max_depth drops from 100000 to 4096: the iterative parser itself is O(1) in native stack regardless of this value, but dump(), the copy constructor, to_cbor()/to_msgpack()/to_ubjson()/to_bson(), and operator== on the parsed result are all still recursive and can overflow well below 100000 (measured: dump() around ~12000 on a 1 MiB stack). 4096 keeps ~9x headroom over the deepest real fixture in the repo (458 levels) while staying safe for those operations too. Also: the BSON parser's `top` reference into the frame stack no longer outlives a stack.push_back() that could reallocate it (was safe today only because a continue always followed each push, which is a fragile invariant to leave implicit); key strings are hoisted outside their element loops again in all four formats and cleared per-iteration instead of being reconstructed, restoring the capacity reuse the iterative rewrite had dropped; and three comments describing an unreachable "is_object && !awaiting_key" branch state are reworded to say so instead of implying a stale-fetch_char case that can't happen. --- .../nlohmann/detail/input/binary_reader.hpp | 211 ++++++++++-------- single_include/nlohmann/json.hpp | 211 ++++++++++-------- 2 files changed, 242 insertions(+), 180 deletions(-) diff --git a/include/nlohmann/detail/input/binary_reader.hpp b/include/nlohmann/detail/input/binary_reader.hpp index 7bdf7a3d95..0eb67de5c6 100644 --- a/include/nlohmann/detail/input/binary_reader.hpp +++ b/include/nlohmann/detail/input/binary_reader.hpp @@ -220,38 +220,41 @@ class binary_reader { std::vector stack; - // reads the 4-byte document-size prefix, records where it started - // (both are needed by @ref check_bson_document_size once this - // document/array's terminator is reached), and emits the - // corresponding SAX start_object()/start_array() event - auto begin_document = [&](const bool is_object, std::size_t& document_start, std::int32_t& document_size) -> bool + // reads the 4-byte document-size prefix and records where it + // started (both are needed by @ref check_bson_document_size once + // this document/array's terminator is reached); deliberately does + // not touch the SAX consumer, so callers can run the max_depth + // check before the corresponding start_object()/start_array() + // event fires + auto read_document_header = [&](std::size_t& document_start, std::int32_t& document_size) { document_start = chars_read; get_number(input_format_t::bson, document_size); - return is_object ? sax->start_object(detail::unknown_size()) : sax->start_array(detail::unknown_size()); }; std::size_t document_start{}; std::int32_t document_size{}; - if (JSON_HEDLEY_UNLIKELY(!begin_document(true, document_start, document_size))) + read_document_header(document_start, document_size); + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { return false; } stack.push_back(bson_container_frame{true, document_start, document_size}); + string_t key; while (true) { - const bson_container_frame& top = stack.back(); + const std::size_t top_index = stack.size() - 1; const auto element_type = get(); if (element_type == 0) { // 0x00 terminates the current document/array - if (JSON_HEDLEY_UNLIKELY(!check_bson_document_size(top.document_start, top.document_size))) + if (JSON_HEDLEY_UNLIKELY(!check_bson_document_size(stack[top_index].document_start, stack[top_index].document_size))) { return false; } - if (JSON_HEDLEY_UNLIKELY(!(top.is_object ? sax->end_object() : sax->end_array()))) + if (JSON_HEDLEY_UNLIKELY(!(stack[top_index].is_object ? sax->end_object() : sax->end_array()))) { return false; } @@ -269,12 +272,12 @@ class binary_reader } const std::size_t element_type_parse_position = chars_read; - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_bson_cstr(key))) { return false; } - if (top.is_object && JSON_HEDLEY_UNLIKELY(!sax->key(key))) + if (stack[top_index].is_object && JSON_HEDLEY_UNLIKELY(!sax->key(key))) { return false; } @@ -306,15 +309,14 @@ class binary_reader { std::size_t nested_document_start{}; std::int32_t nested_document_size{}; - if (JSON_HEDLEY_UNLIKELY(!begin_document(true, nested_document_start, nested_document_size))) + read_document_header(nested_document_start, nested_document_size); + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - return false; + return reached_max_depth(input_format_t::bson); } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + return false; } stack.push_back(bson_container_frame{true, nested_document_start, nested_document_size}); continue; @@ -324,15 +326,14 @@ class binary_reader { std::size_t nested_document_start{}; std::int32_t nested_document_size{}; - if (JSON_HEDLEY_UNLIKELY(!begin_document(false, nested_document_start, nested_document_size))) + read_document_header(nested_document_start, nested_document_size); + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - return false; + return reached_max_depth(input_format_t::bson); } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + return false; } stack.push_back(bson_container_frame{false, nested_document_start, nested_document_size}); continue; @@ -574,6 +575,15 @@ class binary_reader // this like any other produced value via `produce()`). auto enter_container = [&](const bool is_object, const std::size_t len) -> int { + // len == 0 (definite-length, empty) never pushes a frame, so it + // never counts against max_depth; every other case - definite + // non-zero or indefinite - always does, so the check can run + // before the SAX start event fires + if (len != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format_t::cbor); + return 0; + } const bool started = is_object ? sax->start_object(len) : sax->start_array(len); if (JSON_HEDLEY_UNLIKELY(!started)) { @@ -584,13 +594,6 @@ class binary_reader const bool ended = is_object ? sax->end_object() : sax->end_array(); return ended ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(cbor_container_frame{is_object, len == detail::unknown_size(), len, is_object}); return 1; }; @@ -608,6 +611,7 @@ class binary_reader default: continue; \ } + string_t key; while (true) { if (!stack.empty()) @@ -635,7 +639,7 @@ class binary_reader { get(); } - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) { return false; @@ -661,8 +665,11 @@ class binary_reader } fetch_char = !top.indefinite; } - // else: object frame with a key already read; fetch_char is - // already set to true from when the key was read above. + // else: top is an object frame with !awaiting_key, i.e. a + // key was just read above in this same iteration (no branch + // above sets awaiting_key back to false except that one); + // fetch_char is already true from that path, so just fall + // through to read the value. } read_value: @@ -1405,6 +1412,14 @@ class binary_reader auto enter_container = [&](const bool is_object, const std::size_t len) -> int { + // MessagePack containers are always definite-length, so len == 0 + // is the only case that never pushes a frame / never counts + // against max_depth + if (len != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format_t::msgpack); + return 0; + } const bool started = is_object ? sax->start_object(len) : sax->start_array(len); if (JSON_HEDLEY_UNLIKELY(!started)) { @@ -1415,13 +1430,6 @@ class binary_reader const bool ended = is_object ? sax->end_object() : sax->end_array(); return ended ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(msgpack_container_frame{is_object, len, is_object}); return 1; }; @@ -1439,6 +1447,7 @@ class binary_reader default: continue; \ } + string_t key; while (true) { if (!stack.empty()) @@ -1461,7 +1470,7 @@ class binary_reader continue; } get(); - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key))) { return false; @@ -1484,8 +1493,9 @@ class binary_reader continue; } } - // else: map frame with a key already read; fall through to - // read that key's value. + // else: top is a map frame with !awaiting_key, i.e. a key + // was just read above in this same iteration; fall through + // to read that key's value. } switch (get()) @@ -2191,6 +2201,12 @@ class binary_reader size_and_type.second = 'U'; } + if (size_and_type.first != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } + key = "_ArrayData_"; if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first))) { @@ -2201,12 +2217,6 @@ class binary_reader { return (sax->end_array() && sax->end_object()) ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, true}); return 1; } @@ -2220,28 +2230,32 @@ class binary_reader if (size_and_type.first != npos) { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) - { - return 0; - } - // a homogeneous type of 'N' (no-op) means no elements are // actually read, no matter what count was declared - if (size_and_type.second == 'N' || size_and_type.first == 0) + const bool empty = size_and_type.second == 'N' || size_and_type.first == 0; + if (!empty && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - return sax->end_array() ? 2 : 0; + reached_max_depth(input_format); + return 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); return 0; } + if (empty) + { + return sax->end_array() ? 2 : 0; + } stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, false}); return 1; } // indefinite length + if (current != ']' && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) { return 0; @@ -2250,12 +2264,6 @@ class binary_reader { return sax->end_array() ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{false, true, 0, false, 0, false}); return 1; }; @@ -2279,6 +2287,11 @@ class binary_reader if (size_and_type.first != npos) { + if (size_and_type.first != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } if (JSON_HEDLEY_UNLIKELY(!sax->start_object(size_and_type.first))) { return 0; @@ -2287,16 +2300,15 @@ class binary_reader { return sax->end_object() ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{true, false, size_and_type.first, true, size_and_type.second, false}); return 1; } + if (current != '}' && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { return 0; @@ -2305,12 +2317,6 @@ class binary_reader { return sax->end_object() ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{true, true, 0, true, 0, false}); return 1; }; @@ -2328,6 +2334,7 @@ class binary_reader default: continue; \ } + string_t key; while (true) { if (!stack.empty()) @@ -2351,7 +2358,7 @@ class binary_reader continue; } - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key, !top.indefinite) || !sax->key(key))) { return false; @@ -2380,8 +2387,9 @@ class binary_reader continue; } } - // else: object frame with a key already read; fall through - // to read that key's value below. + // else: top is an object frame with !awaiting_key, i.e. a + // key was just read above in this same iteration; fall + // through to read that key's value below. } char_int_type prefix; @@ -3443,6 +3451,25 @@ class binary_reader return concat(error_msg, ' ', context, ": ", detail); } + /*! + @brief report a max_depth parse_error(116) for the given format + + Shared by every container entry point in CBOR/MessagePack/UBJSON/BSON so + there is exactly one place that can forget to guard a new one (see the + BSON 0x04/array gap this PR closes, which happened precisely because the + check was duplicated instead of shared). + + @param[in] format the current format + @return whether parsing should continue (always false: this always + reports an error) + */ + bool reached_max_depth(const input_format_t format) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + private: static JSON_INLINE_VARIABLE constexpr std::size_t npos = detail::unknown_size(); @@ -3451,15 +3478,19 @@ class binary_reader /// /// The container-parsing loops in this file are iterative (see e.g. /// @ref cbor_container_frame): nesting depth is tracked as the size of a - /// heap-allocated std::vector, not native call-stack recursion, so this - /// limit is no longer a stack-safety mechanism - arbitrarily deep input - /// can no longer overflow the stack regardless of this value. It is kept - /// purely as a sanity/DoS cap on absurd inputs (e.g. a malicious payload - /// nesting billions of levels deep to exhaust memory/time), so it can - /// afford to be generous; it is far above the library's own deepest - /// legitimate test fixture (tests/data/json_testsuite/sample.json, 458 - /// levels). - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 100000; + /// heap-allocated std::vector, not native call-stack recursion, so + /// *parsing* can no longer overflow the stack regardless of this value. + /// However, operations on the resulting value - @ref + /// basic_json(const basic_json&) "the copy constructor", @ref dump(), + /// to_cbor()/to_msgpack()/to_ubjson()/to_bson(), and @ref operator== in + /// particular - are still recursive, and a value nested deeper than the + /// low thousands can overflow the native stack in *those* regardless of + /// how it was produced. This limit therefore stays low enough that a + /// successfully parsed value remains safe to copy/dump/compare even on a + /// constrained (1 MiB) stack, with generous headroom over the library's + /// own deepest legitimate test fixture + /// (tests/data/json_testsuite/sample.json, 458 levels). + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 4096; /// input adapter InputAdapterType ia; diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index ec0a65ce9d..22b973c5d4 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -10769,38 +10769,41 @@ class binary_reader { std::vector stack; - // reads the 4-byte document-size prefix, records where it started - // (both are needed by @ref check_bson_document_size once this - // document/array's terminator is reached), and emits the - // corresponding SAX start_object()/start_array() event - auto begin_document = [&](const bool is_object, std::size_t& document_start, std::int32_t& document_size) -> bool + // reads the 4-byte document-size prefix and records where it + // started (both are needed by @ref check_bson_document_size once + // this document/array's terminator is reached); deliberately does + // not touch the SAX consumer, so callers can run the max_depth + // check before the corresponding start_object()/start_array() + // event fires + auto read_document_header = [&](std::size_t& document_start, std::int32_t& document_size) { document_start = chars_read; get_number(input_format_t::bson, document_size); - return is_object ? sax->start_object(detail::unknown_size()) : sax->start_array(detail::unknown_size()); }; std::size_t document_start{}; std::int32_t document_size{}; - if (JSON_HEDLEY_UNLIKELY(!begin_document(true, document_start, document_size))) + read_document_header(document_start, document_size); + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { return false; } stack.push_back(bson_container_frame{true, document_start, document_size}); + string_t key; while (true) { - const bson_container_frame& top = stack.back(); + const std::size_t top_index = stack.size() - 1; const auto element_type = get(); if (element_type == 0) { // 0x00 terminates the current document/array - if (JSON_HEDLEY_UNLIKELY(!check_bson_document_size(top.document_start, top.document_size))) + if (JSON_HEDLEY_UNLIKELY(!check_bson_document_size(stack[top_index].document_start, stack[top_index].document_size))) { return false; } - if (JSON_HEDLEY_UNLIKELY(!(top.is_object ? sax->end_object() : sax->end_array()))) + if (JSON_HEDLEY_UNLIKELY(!(stack[top_index].is_object ? sax->end_object() : sax->end_array()))) { return false; } @@ -10818,12 +10821,12 @@ class binary_reader } const std::size_t element_type_parse_position = chars_read; - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_bson_cstr(key))) { return false; } - if (top.is_object && JSON_HEDLEY_UNLIKELY(!sax->key(key))) + if (stack[top_index].is_object && JSON_HEDLEY_UNLIKELY(!sax->key(key))) { return false; } @@ -10855,15 +10858,14 @@ class binary_reader { std::size_t nested_document_start{}; std::int32_t nested_document_size{}; - if (JSON_HEDLEY_UNLIKELY(!begin_document(true, nested_document_start, nested_document_size))) + read_document_header(nested_document_start, nested_document_size); + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - return false; + return reached_max_depth(input_format_t::bson); } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + return false; } stack.push_back(bson_container_frame{true, nested_document_start, nested_document_size}); continue; @@ -10873,15 +10875,14 @@ class binary_reader { std::size_t nested_document_start{}; std::int32_t nested_document_size{}; - if (JSON_HEDLEY_UNLIKELY(!begin_document(false, nested_document_start, nested_document_size))) + read_document_header(nested_document_start, nested_document_size); + if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - return false; + return reached_max_depth(input_format_t::bson); } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) { - return sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::bson, "maximum depth of nested objects/arrays exceeded", "value"), nullptr)); + return false; } stack.push_back(bson_container_frame{false, nested_document_start, nested_document_size}); continue; @@ -11123,6 +11124,15 @@ class binary_reader // this like any other produced value via `produce()`). auto enter_container = [&](const bool is_object, const std::size_t len) -> int { + // len == 0 (definite-length, empty) never pushes a frame, so it + // never counts against max_depth; every other case - definite + // non-zero or indefinite - always does, so the check can run + // before the SAX start event fires + if (len != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format_t::cbor); + return 0; + } const bool started = is_object ? sax->start_object(len) : sax->start_array(len); if (JSON_HEDLEY_UNLIKELY(!started)) { @@ -11133,13 +11143,6 @@ class binary_reader const bool ended = is_object ? sax->end_object() : sax->end_array(); return ended ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::cbor, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(cbor_container_frame{is_object, len == detail::unknown_size(), len, is_object}); return 1; }; @@ -11157,6 +11160,7 @@ class binary_reader default: continue; \ } + string_t key; while (true) { if (!stack.empty()) @@ -11184,7 +11188,7 @@ class binary_reader { get(); } - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_cbor_string(key) || !sax->key(key))) { return false; @@ -11210,8 +11214,11 @@ class binary_reader } fetch_char = !top.indefinite; } - // else: object frame with a key already read; fetch_char is - // already set to true from when the key was read above. + // else: top is an object frame with !awaiting_key, i.e. a + // key was just read above in this same iteration (no branch + // above sets awaiting_key back to false except that one); + // fetch_char is already true from that path, so just fall + // through to read the value. } read_value: @@ -11954,6 +11961,14 @@ class binary_reader auto enter_container = [&](const bool is_object, const std::size_t len) -> int { + // MessagePack containers are always definite-length, so len == 0 + // is the only case that never pushes a frame / never counts + // against max_depth + if (len != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format_t::msgpack); + return 0; + } const bool started = is_object ? sax->start_object(len) : sax->start_array(len); if (JSON_HEDLEY_UNLIKELY(!started)) { @@ -11964,13 +11979,6 @@ class binary_reader const bool ended = is_object ? sax->end_object() : sax->end_array(); return ended ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), - parse_error::create(116, chars_read, - exception_message(input_format_t::msgpack, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(msgpack_container_frame{is_object, len, is_object}); return 1; }; @@ -11988,6 +11996,7 @@ class binary_reader default: continue; \ } + string_t key; while (true) { if (!stack.empty()) @@ -12010,7 +12019,7 @@ class binary_reader continue; } get(); - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_msgpack_string(key) || !sax->key(key))) { return false; @@ -12033,8 +12042,9 @@ class binary_reader continue; } } - // else: map frame with a key already read; fall through to - // read that key's value. + // else: top is a map frame with !awaiting_key, i.e. a key + // was just read above in this same iteration; fall through + // to read that key's value. } switch (get()) @@ -12740,6 +12750,12 @@ class binary_reader size_and_type.second = 'U'; } + if (size_and_type.first != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } + key = "_ArrayData_"; if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first))) { @@ -12750,12 +12766,6 @@ class binary_reader { return (sax->end_array() && sax->end_object()) ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, true}); return 1; } @@ -12769,28 +12779,32 @@ class binary_reader if (size_and_type.first != npos) { - if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) - { - return 0; - } - // a homogeneous type of 'N' (no-op) means no elements are // actually read, no matter what count was declared - if (size_and_type.second == 'N' || size_and_type.first == 0) + const bool empty = size_and_type.second == 'N' || size_and_type.first == 0; + if (!empty && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) { - return sax->end_array() ? 2 : 0; + reached_max_depth(input_format); + return 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + if (JSON_HEDLEY_UNLIKELY(!sax->start_array(size_and_type.first))) { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); return 0; } + if (empty) + { + return sax->end_array() ? 2 : 0; + } stack.push_back(ubjson_container_frame{false, false, size_and_type.first, false, size_and_type.second, false}); return 1; } // indefinite length + if (current != ']' && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } if (JSON_HEDLEY_UNLIKELY(!sax->start_array(detail::unknown_size()))) { return 0; @@ -12799,12 +12813,6 @@ class binary_reader { return sax->end_array() ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{false, true, 0, false, 0, false}); return 1; }; @@ -12828,6 +12836,11 @@ class binary_reader if (size_and_type.first != npos) { + if (size_and_type.first != 0 && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } if (JSON_HEDLEY_UNLIKELY(!sax->start_object(size_and_type.first))) { return 0; @@ -12836,16 +12849,15 @@ class binary_reader { return sax->end_object() ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{true, false, size_and_type.first, true, size_and_type.second, false}); return 1; } + if (current != '}' && JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) + { + reached_max_depth(input_format); + return 0; + } if (JSON_HEDLEY_UNLIKELY(!sax->start_object(detail::unknown_size()))) { return 0; @@ -12854,12 +12866,6 @@ class binary_reader { return sax->end_object() ? 2 : 0; } - if (JSON_HEDLEY_UNLIKELY(stack.size() >= max_depth)) - { - sax->parse_error(chars_read, get_token_string(), parse_error::create(116, chars_read, - exception_message(input_format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); - return 0; - } stack.push_back(ubjson_container_frame{true, true, 0, true, 0, false}); return 1; }; @@ -12877,6 +12883,7 @@ class binary_reader default: continue; \ } + string_t key; while (true) { if (!stack.empty()) @@ -12900,7 +12907,7 @@ class binary_reader continue; } - string_t key; + key.clear(); if (JSON_HEDLEY_UNLIKELY(!get_ubjson_string(key, !top.indefinite) || !sax->key(key))) { return false; @@ -12929,8 +12936,9 @@ class binary_reader continue; } } - // else: object frame with a key already read; fall through - // to read that key's value below. + // else: top is an object frame with !awaiting_key, i.e. a + // key was just read above in this same iteration; fall + // through to read that key's value below. } char_int_type prefix; @@ -13992,6 +14000,25 @@ class binary_reader return concat(error_msg, ' ', context, ": ", detail); } + /*! + @brief report a max_depth parse_error(116) for the given format + + Shared by every container entry point in CBOR/MessagePack/UBJSON/BSON so + there is exactly one place that can forget to guard a new one (see the + BSON 0x04/array gap this PR closes, which happened precisely because the + check was duplicated instead of shared). + + @param[in] format the current format + @return whether parsing should continue (always false: this always + reports an error) + */ + bool reached_max_depth(const input_format_t format) + { + return sax->parse_error(chars_read, get_token_string(), + parse_error::create(116, chars_read, + exception_message(format, "maximum depth of nested arrays/objects exceeded", "value"), nullptr)); + } + private: static JSON_INLINE_VARIABLE constexpr std::size_t npos = detail::unknown_size(); @@ -14000,15 +14027,19 @@ class binary_reader /// /// The container-parsing loops in this file are iterative (see e.g. /// @ref cbor_container_frame): nesting depth is tracked as the size of a - /// heap-allocated std::vector, not native call-stack recursion, so this - /// limit is no longer a stack-safety mechanism - arbitrarily deep input - /// can no longer overflow the stack regardless of this value. It is kept - /// purely as a sanity/DoS cap on absurd inputs (e.g. a malicious payload - /// nesting billions of levels deep to exhaust memory/time), so it can - /// afford to be generous; it is far above the library's own deepest - /// legitimate test fixture (tests/data/json_testsuite/sample.json, 458 - /// levels). - static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 100000; + /// heap-allocated std::vector, not native call-stack recursion, so + /// *parsing* can no longer overflow the stack regardless of this value. + /// However, operations on the resulting value - @ref + /// basic_json(const basic_json&) "the copy constructor", @ref dump(), + /// to_cbor()/to_msgpack()/to_ubjson()/to_bson(), and @ref operator== in + /// particular - are still recursive, and a value nested deeper than the + /// low thousands can overflow the native stack in *those* regardless of + /// how it was produced. This limit therefore stays low enough that a + /// successfully parsed value remains safe to copy/dump/compare even on a + /// constrained (1 MiB) stack, with generous headroom over the library's + /// own deepest legitimate test fixture + /// (tests/data/json_testsuite/sample.json, 458 levels). + static JSON_INLINE_VARIABLE constexpr std::size_t max_depth = 4096; /// input adapter InputAdapterType ia; From b4fed3e1a1e5626059737a1468bacebf192c1620 Mon Sep 17 00:00:00 2001 From: Gooh456 <164024549+Gooh456@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:09:36 +0100 Subject: [PATCH 12/14] Update depth regression tests for max_depth=4096; dedupe BSON, add BJData Resize the five stack-overflow regression tests from n=100010 (sized for the old max_depth=100000) down to n=4200, comfortably past the new 4096 limit; updated the expected byte offsets and exception text (now "arrays/objects" everywhere, matching the shared helper) to match. The smaller payloads also make these tests noticeably cheaper to build and run. unit-bson.cpp: extracted the object-depth and array-depth sections' ~50 lines of near-identical payload construction into one make_deeply_nested_bson(n, record_type, key_char) lambda shared by both. unit-bjdata.cpp: added a "BJData regressions" test case with the same plain-nested-array depth test as UBJSON, run under the bjdata format tag specifically - BJData reaches enter_array() through paths UBJSON doesn't have (the ND-array _ArrayData_ frame, the optimized $type#count container), so it had no dedicated depth coverage of its own before this. --- tests/src/unit-bjdata.cpp | 32 ++++++++++++++ tests/src/unit-bson.cpp | 85 ++++++++++++++------------------------ tests/src/unit-cbor.cpp | 15 ++++--- tests/src/unit-msgpack.cpp | 15 ++++--- tests/src/unit-ubjson.cpp | 14 +++---- 5 files changed, 84 insertions(+), 77 deletions(-) diff --git a/tests/src/unit-bjdata.cpp b/tests/src/unit-bjdata.cpp index a60d0a1641..782895cba1 100644 --- a/tests/src/unit-bjdata.cpp +++ b/tests/src/unit-bjdata.cpp @@ -3712,6 +3712,38 @@ TEST_CASE("Universal Binary JSON Specification Examples 1") } } +TEST_CASE("BJData regressions") +{ + SECTION("stack overflow via deeply nested input (issue #5104)") + { + // BJData shares UBJSON's container-parsing code path (see the + // equivalent test in unit-ubjson.cpp), but also reaches enter_array + // through paths UBJSON doesn't have at all (the ND-array + // "_ArrayData_" frame, the optimized $type#count container) - a + // dedicated depth test here is cheap insurance against a regression + // that's specific to the bjdata format tag. + // + // 4200 nested arrays - comfortably past max_depth (4096, see + // binary_reader.hpp); the (now iterative, heap-stack based) parser + // must reject this with a clean parse_error once the nesting + // exceeds the depth limit, rather than exhausting the native call + // stack. + const int nesting = 4200; + std::string payload(static_cast(nesting), '['); + payload += "Z"; + for (int i = 0; i < nesting; ++i) + { + payload += ']'; + } + std::vector const v(payload.begin(), payload.end()); + json _; + CHECK_THROWS_WITH_AS(_ = json::from_bjdata(v), + "[json.exception.parse_error.116] parse error at byte 4098: syntax error while parsing BJData value: maximum depth of nested arrays/objects exceeded", + json::parse_error&); + CHECK(json::from_bjdata(v, true, false).is_discarded()); + } +} + TEST_CASE("Parse BJData directly from a file using iterator and sentinel") { std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.bjdata"; diff --git a/tests/src/unit-bson.cpp b/tests/src/unit-bson.cpp index 193205ed2a..d581b86025 100644 --- a/tests/src/unit-bson.cpp +++ b/tests/src/unit-bson.cpp @@ -856,25 +856,21 @@ TEST_CASE("Unsupported BSON input") TEST_CASE("BSON regressions") { - SECTION("stack overflow via deeply nested input (issue #5104)") + // Shared by both sections below: builds a chain of n embedded BSON + // containers of the given record_type (0x03 object / 0x04 array), each + // wrapping the previous one under a 1-byte key - differing from the + // other section only in those two parameters. + // + // Built directly in O(N) rather than by repeatedly wrapping and copying + // the previous document (which would be O(N^2) and much too slow at + // this depth). Layout, front to back: + // [header_N]...[header_1][innermost 5-byte empty doc][term_1]...[term_N] + // where header_i is a 4-byte little-endian size_i followed by + // {record_type, key_char, 0x00}, and size_i = size_{i-1} + 8, size_0 = 5. + // Terminators are all 0x00, already the buffer's default value. + auto make_deeply_nested_bson = [](int n, std::uint8_t record_type, char key_char) -> std::vector { - // A chain of 100010 embedded BSON documents (record type 0x03), each - // wrapping the previous one under key "a" - comfortably past - // max_depth (100000) and far beyond any depth a recursive-descent - // parser could ever have survived. The (now iterative, heap-stack - // based) parser must reject this with a clean parse_error once the - // nesting exceeds the depth limit, rather than exhausting the native - // call stack. - // - // Built directly in O(N) rather than by repeatedly wrapping and - // copying the previous document (which would be O(N^2) and much too - // slow at this depth). Layout, front to back: - // [header_N]...[header_1][innermost 5-byte empty doc][term_1]...[term_N] - // where header_i is a 4-byte little-endian size_i followed by - // {0x03,'a',0x00}, and size_i = size_{i-1} + 8, size_0 = 5. - // Terminators are all 0x00, already the buffer's default value. - const int n = 100010; - const std::size_t header_size = 7; // 4 (size) + 3 (0x03, 'a', 0x00) + const std::size_t header_size = 7; // 4 (size) + 3 (record_type, key_char, 0x00) const std::size_t inner_size = 5; const std::size_t total = (static_cast(n) * header_size) + inner_size + static_cast(n); std::vector inner(total, 0x00); @@ -894,8 +890,8 @@ TEST_CASE("BSON regressions") inner[pos + 1] = static_cast((sz >> 8) & 0xFF); inner[pos + 2] = static_cast((sz >> 16) & 0xFF); inner[pos + 3] = static_cast((sz >> 24) & 0xFF); - inner[pos + 4] = 0x03; - inner[pos + 5] = 'a'; + inner[pos + 4] = record_type; + inner[pos + 5] = static_cast(key_char); inner[pos + 6] = 0x00; pos += header_size; } @@ -906,10 +902,22 @@ TEST_CASE("BSON regressions") inner[pos + 3] = 0; inner[pos + 4] = 0x00; // trailing n terminator bytes are already 0x00 from initialization + return inner; + }; + + // n chosen comfortably past max_depth (4096, see binary_reader.hpp) with + // real margin, while staying far smaller than a stress test needs to be + // - unlike a recursive-descent parser, the iterative parser's memory use + // is the only thing scaling with n, so there's no need to pick a value + // "far beyond any depth a recursive parser could survive". + const int n = 4200; + SECTION("stack overflow via deeply nested input (issue #5104)") + { + const auto inner = make_deeply_nested_bson(n, 0x03, 'a'); json _; CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), - "[json.exception.parse_error.116] parse error at byte 700004: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + "[json.exception.parse_error.116] parse error at byte 28676: syntax error while parsing BSON value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_bson(inner, true, false).is_discarded()); } @@ -921,41 +929,10 @@ TEST_CASE("BSON regressions") // separate code path with no depth check of their own at all. The // iterative rewrite unifies object/array container handling into one // loop, so this gap is now closed too - verify it here explicitly. - const int n = 100010; - const std::size_t header_size = 7; // 4 (size) + 3 (0x04, '0', 0x00) - const std::size_t inner_size = 5; - const std::size_t total = (static_cast(n) * header_size) + inner_size + static_cast(n); - std::vector inner(total, 0x00); - - std::vector size_at(static_cast(n) + 1); - size_at[0] = static_cast(inner_size); - for (int i = 1; i <= n; ++i) - { - size_at[static_cast(i)] = size_at[static_cast(i) - 1] + 8; - } - - std::size_t pos = 0; - for (int i = n; i >= 1; --i) - { - const auto sz = static_cast(size_at[static_cast(i)]); - inner[pos + 0] = static_cast(sz & 0xFF); - inner[pos + 1] = static_cast((sz >> 8) & 0xFF); - inner[pos + 2] = static_cast((sz >> 16) & 0xFF); - inner[pos + 3] = static_cast((sz >> 24) & 0xFF); - inner[pos + 4] = 0x04; // array element - inner[pos + 5] = '0'; - inner[pos + 6] = 0x00; - pos += header_size; - } - inner[pos + 0] = 5; - inner[pos + 1] = 0; - inner[pos + 2] = 0; - inner[pos + 3] = 0; - inner[pos + 4] = 0x00; - + const auto inner = make_deeply_nested_bson(n, 0x04, '0'); json _; CHECK_THROWS_WITH_AS(_ = json::from_bson(inner), - "[json.exception.parse_error.116] parse error at byte 700004: syntax error while parsing BSON value: maximum depth of nested objects/arrays exceeded", + "[json.exception.parse_error.116] parse error at byte 28676: syntax error while parsing BSON value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_bson(inner, true, false).is_discarded()); } diff --git a/tests/src/unit-cbor.cpp b/tests/src/unit-cbor.cpp index f284c282d2..316ff57016 100644 --- a/tests/src/unit-cbor.cpp +++ b/tests/src/unit-cbor.cpp @@ -1999,17 +1999,16 @@ TEST_CASE("CBOR regressions") SECTION("stack overflow via deeply nested input (issue #5104)") { - // 100010 nested CBOR arrays of length 1 (0x81 each) - comfortably past - // max_depth (100000) and far beyond any depth a recursive-descent - // parser could ever have survived; the (now iterative, heap-stack - // based) parser must reject this with a clean parse_error once the - // nesting exceeds the depth limit, rather than exhausting the native - // call stack. - std::vector vec(100010, 0x81); + // 4200 nested CBOR arrays of length 1 (0x81 each) - comfortably past + // max_depth (4096, see binary_reader.hpp); the (now iterative, + // heap-stack based) parser must reject this with a clean parse_error + // once the nesting exceeds the depth limit, rather than exhausting + // the native call stack. + std::vector vec(4200, 0x81); vec.push_back(0x00); json _; // NOLINT(readability-identifier-naming) CHECK_THROWS_WITH_AS(_ = json::from_cbor(vec), - "[json.exception.parse_error.116] parse error at byte 100001: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 4097: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_cbor(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-msgpack.cpp b/tests/src/unit-msgpack.cpp index 17c65aa0c7..2f46e464d3 100644 --- a/tests/src/unit-msgpack.cpp +++ b/tests/src/unit-msgpack.cpp @@ -1573,17 +1573,16 @@ TEST_CASE("MessagePack") SECTION("stack overflow via deeply nested input (issue #5104)") { - // 100010 nested fixarrays of length 1 (0x91 each) - comfortably past - // max_depth (100000) and far beyond any depth a recursive-descent - // parser could ever have survived; the (now iterative, heap-stack - // based) parser must reject this with a clean parse_error once the - // nesting exceeds the depth limit, rather than exhausting the - // native call stack. - std::vector vec(100010, 0x91); + // 4200 nested fixarrays of length 1 (0x91 each) - comfortably past + // max_depth (4096, see binary_reader.hpp); the (now iterative, + // heap-stack based) parser must reject this with a clean + // parse_error once the nesting exceeds the depth limit, rather + // than exhausting the native call stack. + std::vector vec(4200, 0x91); vec.push_back(0x00); json _; CHECK_THROWS_WITH_AS(_ = json::from_msgpack(vec), - "[json.exception.parse_error.116] parse error at byte 100001: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 4097: syntax error while parsing MessagePack value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_msgpack(vec, true, false).is_discarded()); } diff --git a/tests/src/unit-ubjson.cpp b/tests/src/unit-ubjson.cpp index 9dece8b533..18c12a67c7 100644 --- a/tests/src/unit-ubjson.cpp +++ b/tests/src/unit-ubjson.cpp @@ -1641,12 +1641,12 @@ TEST_CASE("UBJSON") SECTION("stack overflow via deeply nested input (issue #5104)") { - // 100010 nested UBJSON arrays - comfortably past max_depth (100000) - // and far beyond any depth a recursive-descent parser could ever - // have survived; the (now iterative, heap-stack based) parser must - // reject this with a clean parse_error once the nesting exceeds the - // depth limit, rather than exhausting the native call stack. - const int nesting = 100010; + // 4200 nested UBJSON arrays - comfortably past max_depth (4096, + // see binary_reader.hpp); the (now iterative, heap-stack based) + // parser must reject this with a clean parse_error once the + // nesting exceeds the depth limit, rather than exhausting the + // native call stack. + const int nesting = 4200; std::string payload(static_cast(nesting), '['); payload += "Z"; for (int i = 0; i < nesting; ++i) @@ -1656,7 +1656,7 @@ TEST_CASE("UBJSON") std::vector const v(payload.begin(), payload.end()); json _; CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), - "[json.exception.parse_error.116] parse error at byte 100002: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", + "[json.exception.parse_error.116] parse error at byte 4098: syntax error while parsing UBJSON value: maximum depth of nested arrays/objects exceeded", json::parse_error&); CHECK(json::from_ubjson(v, true, false).is_discarded()); } From eb3acd8087108750f9600994ee22bfddb2cf492b Mon Sep 17 00:00:00 2001 From: Gooh456 <164024549+Gooh456@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:09:42 +0100 Subject: [PATCH 13/14] Guard -fuse-ld=lld behind find_program instead of hard-requiring it The Clang+MinGW -fuse-ld=lld addition for test-regression2's linker error was an untested hypothesis when it went in, with no availability check - it would break the build for anyone building the tests with Clang+MinGW who doesn't happen to have lld installed. find_program degrades instead of failing when it's missing. Left as a guarded opt-in rather than removed outright: I don't have a Clang+MinGW toolchain in this environment to confirm the original linker error is actually gone now that develop has been merged in twice since. --- tests/CMakeLists.txt | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 55bb6f074c..0d8b0cda88 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -110,7 +110,15 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang" AND MINGW) # range for IMAGE_REL_AMD64_REL32 relocations against .rdata once template # instantiation grows far enough ("relocation truncated to fit"). lld lays # out COFF sections differently and doesn't hit this ceiling for the same input. - json_test_set_test_options(test-regression2 LINK_OPTIONS -fuse-ld=lld) + # + # Guarded rather than unconditional: this was originally added as an + # untested hypothesis for a CI linker error, and hard-requiring lld with + # no fallback would break the build for anyone building the tests with + # Clang+MinGW who doesn't happen to have it installed. + find_program(JSON_TEST_LLD_PROGRAM lld) + if (JSON_TEST_LLD_PROGRAM) + json_test_set_test_options(test-regression2 LINK_OPTIONS -fuse-ld=lld) + endif() endif() # disable exceptions for test-disabled_exceptions From 5c88c051325b44ef837429d19f86391dd64eba31 Mon Sep 17 00:00:00 2001 From: Gooh456 <164024549+Gooh456@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:09:47 +0100 Subject: [PATCH 14/14] Update parse_error.116 docs for max_depth=4096 and add recursive-ops caveat Reflects the lowered limit and its example byte offset, and adds the caveat @nlohmann's review asked for: the O(1)-native-stack guarantee is specific to parsing, not to dump()/copy/to_cbor() and friends on the result. --- docs/mkdocs/docs/home/exceptions.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/mkdocs/docs/home/exceptions.md b/docs/mkdocs/docs/home/exceptions.md index c66a3f0d91..525cca9465 100644 --- a/docs/mkdocs/docs/home/exceptions.md +++ b/docs/mkdocs/docs/home/exceptions.md @@ -369,18 +369,22 @@ A UBJSON high-precision number could not be parsed. ### json.exception.parse_error.116 A binary input format (CBOR, MessagePack, UBJSON/BJData, or BSON) contained -arrays/objects nested deeper than the library's internal depth limit -(100000). The binary format parsers track container nesting on an explicit, -heap-allocated stack rather than the native call stack, so nesting depth can -no longer cause a stack overflow regardless of this limit; the limit exists -purely as a sanity check against absurd/malicious inputs (for example, one -designed to exhaust memory or time rather than the call stack), and inputs -nested beyond it are rejected with this parse error instead. +arrays/objects nested deeper than the library's internal depth limit (4096). +The binary format parsers track container nesting on an explicit, +heap-allocated stack rather than the native call stack, so *parsing* alone +can no longer cause a stack overflow regardless of this limit. The limit +itself stays conservative rather than being raised as high as that guarantee +would allow, because operations on the resulting value - the copy +constructor, `dump()`, `to_cbor()`/`to_msgpack()`/`to_ubjson()`/`to_bson()`, +and `operator==` - are still recursive, and a successfully parsed value +nested deeper than the low thousands can overflow the native stack in +*those* regardless of how it was produced. Inputs nested beyond the limit +are rejected with this parse error instead. !!! failure "Example message" ``` - [json.exception.parse_error.116] parse error at byte 100001: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded + [json.exception.parse_error.116] parse error at byte 4097: syntax error while parsing CBOR value: maximum depth of nested arrays/objects exceeded ``` ## Iterator errors