[SPARK-59417][SQL] Stream multiline top-level JSON arrays

[tdcmeehan]

What changes were proposed in this pull request?

This PR adds the internal Spark SQL configuration spark.sql.json.enableStreamingTopLevelArray. When enabled, multiline JSON reading consumes a top-level array of structs lazily from one Jackson parser instead of materializing the entire array before returning rows. The configuration defaults to false so the behavior can be rolled out safely.

It also adds an opt-in lazy path to FailureSafeParser for failures raised while advancing the parser iterator, preserves rows emitted before later structural corruption, and adds benchmark cases for both many small elements and fewer large elements. Existing eager FailureSafeParser callers remain unchanged.

Why are the changes needed?

The existing parser materializes every element of a multiline top-level JSON array in memory. Large arrays can therefore require memory proportional to the complete document even though Spark consumes the result as an iterator. Streaming elements bounds parser-side memory while retaining the existing valid-row and parse-mode behavior. The SQL configuration permits staged adoption of the new parsing path.

Does this PR introduce any user-facing change?

Yes, when spark.sql.json.enableStreamingTopLevelArray is enabled. Reading a multiline top-level JSON array no longer materializes the complete array before producing rows. If malformed structure is encountered after valid rows have already been emitted, those rows remain available and the malformed document is then handled according to the configured parse mode. The configuration is disabled by default.

How was this patch tested?

Added regression coverage for both SQL configuration values, lazy top-level array parsing, and malformed input after partial output under PERMISSIVE, DROPMALFORMED, and FAILFAST, and extended the existing SPARK-3308 top-level-array coverage.

Ran:

• sql/Test/compile
• sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV1Suite (162 tests)
• sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV2Suite (163 tests)
• catalyst/scalastyle
• sql/scalastyle
• sql/Test/scalastyle
• git diff --check

The focused OSS benchmark can be run with:

build/sbt "sql/Test/runMain org.apache.spark.sql.execution.datasources.json.JsonBenchmark top-level-array"

The benchmark reads multiline JSON through the data source with spark.sql.json.enableStreamingTopLevelArray disabled and enabled. On OpenJDK 17.0.15 and Intel Xeon 6975P:

• 100,000 rows with empty payloads: disabled 67 ms; enabled 43 ms.
• 1,000 rows with 64 KiB payloads: disabled 114 ms; enabled 108 ms.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[HyukjinKwon]

Mind taking a look at https://github.com/apache/spark/pull/58704/checks?check_run_id=103066275835?

[tdcmeehan]

@HyukjinKwon done, thanks!