refactor(spark): resolve format options through a VortexOptions type

[jackylee-ch]

Follow-up to the discussion on #9170: rather than fixing the case-insensitive lookup at one call site, this gives the Vortex format options a type.

Why

Spark lower-cases the keys of the CaseInsensitiveStringMap it hands to a table, so an option a user spelled vortex.workerThreads arrives as vortex.workerthreads. Every call site that read an option had to know that, and none did:

• VortexPartitionReaderFactory looked the worker-thread count up case-sensitively, so any spelling other than the exact camelCase one silently fell back to the default.
• VortexTable.newScanBuilder merged scan options over table options with a plain putAll, so the same option could end up present twice under two spellings, and VortexScanBuilder seeded the camelCase spelling to compensate.
• vortex.session.provider and vortex.write.batch.size had the same exposure, each parsed inline with a string literal.

What

VortexOptions follows the shape of Spark's own ParquetOptions: Serializable, holding its CaseInsensitiveStringMap as transient and rebuilding it on demand, so it crosses the serialization boundary into VortexFilePartition and the reader/writer factories without carrying a non-serializable field. Every option now has one place that names it, documents its default and validates it:

Constant	Key	Accessor
WORKER_THREADS	vortex.workerThreads	workerThreads() — default 4, rejects a negative count
WRITE_BATCH_SIZE	vortex.write.batch.size	writeBatchSize() — default 2048, falls back outside [1, 65536]
LEGACY_WRITE_BATCH_SIZE	batch.size	same, as a fallback
SESSION_PROVIDER	vortex.session.provider	sessionProvider()

withOverrides folds keys with toLowerCase(Locale.ROOT) — the same rule CaseInsensitiveStringMap uses — so a scan-level override replaces the table-level option rather than sitting beside it. It is threaded through the read and write paths in place of the raw map, converting back with asMap() only where the native bindings are called.

Two details worth flagging:

• The worker-thread count is now resolved once on the driver in VortexBatchExec.createReaderFactory, so an invalid value fails the query once instead of once per task on the executors.
• rejectedWriteBatchSize() returns the key and the value that were ignored, so the warning names whichever key the user actually set (previously it could not distinguish batch.size from vortex.write.batch.size, and dropped the offending value).

Tests

New VortexOptionsTest (19 cases): case-insensitive resolution for each option, the legacy batch-size key and its precedence, out-of-range fallback reported under the key the user set, validation messages, override semantics through Spark's own map, asMap() immutability, defensive copying, and a Java-serialization round trip that resolves an option before serializing so the transient view is actually populated.

VortexTableTest gains a case pinning the table-vs-scan option merge end to end.

Each of these was checked for vacuity by mutating the implementation and confirming the test fails: dropping transient, returning a new instance from the withOverrides fast path, and making newScanBuilder ignore its argument all produce failures.

• ./gradlew :vortex-spark_2.13:test — 176 tests pass
• ./gradlew :vortex-spark_2.12:test — 176 tests pass
• spotlessCheck (both Scala versions) and javadoc — clean

VortexDataSourceS3MockTest fails locally for lack of a Docker environment, unrelated to this change.

[codspeed-hq]

Merging this PR will regress 1 benchmark

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 1891 untouched benchmarks
⏩ 43 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	decompress[u32, (1000, 16)]	45.1 µs	57.7 µs	-21.91%
⚡	Simulation	decode_varbin[(1000, 2)]	78.3 µs	61.8 µs	+26.63%
⚡	Simulation	decompress[u64, (1000, 16)]	74 µs	65.6 µs	+12.65%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing jackylee-ch:spark-vortex-options (0f69e66) with develop (7ad0d9d)}

Footnotes

1. 43 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩