[rustyconover]

Rationale for this change

Closes #1380. Part of apache/arrow#49704 and apache/arrow#54958.

In IPC a dictionary-encoded field is one Field whose type is the value type, so dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialize identically. In arrow-go:

• a dictionary of arrow.uuid values is written with the UUID keys on the Field, and reading it back fails (invalid storage type for UUIDType: dictionary);
• writing the schema of an extension type with dictionary storage panics.

The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding custom_metadata: [KeyValue] to DictionaryEncoding for the dictionary value type's metadata.

Companion drafts: Rust apache/arrow-rs#11622, nanoarrow apache/arrow-nanoarrow#972.

This depends on the format change in apache/arrow#54959 (Schema.fbs and the columnar, C Data Interface and integration JSON specs), which is pending a dev@ vote. The C++ implementation and the dictionary_extension archery integration case are in apache/arrow#53994, which is stacked on it. It is the second reference implementation for that vote, so it should not be merged before the format change is accepted.

What changes are included in this PR?

• Generated code: arrow/internal/flatbuf/DictionaryEncoding.go regenerated (flatc 25.12.19) with the new DictionaryEncoding.custom_metadata member.
• IPC writer:
  • The ARROW:extension:name/ARROW:extension:metadata keys of a dictionary's extension value type are written to DictionaryEncoding.custom_metadata instead of Field.custom_metadata.
  • An extension type with dictionary storage keeps writing its keys to Field.custom_metadata, and its schema no longer panics on write.
• IPC reader:
  • Applies DictionaryEncoding.custom_metadata to the value type, wraps the result in the dictionary type, then applies Field.custom_metadata as before.
  • Only registered extension types are applied. Go's DictionaryType has nowhere to keep other value-type metadata, so other keys and unregistered extension names are dropped on read. That is a known limitation of this implementation; the format gives the member the same semantics as Field.custom_metadata.
  • Removing the extension keys from field metadata no longer mutates slices shared with the caller's arrow.Metadata.
• Integration JSON: an optional "metadata" list in a field's "dictionary" object. The reader also now honours "isOrdered".
• dictutils.Mapper: finds nested dictionaries inside extension value types.
• New arrow/internal/integration package: registers the canonical extension types. It is imported by arrow-json-integration-test, arrow-file-to-stream, arrow-stream-to-file and the Flight integration client and server. The C Data integration library doesn't import it, because its importer does not use the extension registry.

Are these changes tested?

Yes.

• IPC unit tests:
  • schema serialization, checking the keys sit in DictionaryEncoding and not Field;
  • stream and file data round trips of dictionaries of UUID and of an extension type with metadata;
  • extension types with dictionary storage, including a dictionary of extension values as storage;
  • a dictionary nested inside an extension value type;
  • delta dictionaries of dict<uuid>;
  • unregistered, non-extension and wrong-storage dictionary metadata on read;
  • legacy field-level metadata.
• Integration JSON round trips: dictionary metadata, opaque<dict<uuid>>, list<dict<uuid>> and ordered dictionaries.
• Archery, run manually:
  • C++ (GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types arrow#53994) ⇄ Go passes over IPC and C Data arrays for the extension, dictionary_extension, dictionary, nested_dictionary and extension_union cases.
  • Go's C schema import is skipped there, because the C Data importer does not use the extension registry.
  • Go-to-Go Flight integration passes for the same cases.

Are there any user-facing changes?

Yes:

• Changed IPC output: for a dictionary whose values are an extension type, the extension keys move from Field.custom_metadata to the new DictionaryEncoding.custom_metadata.
  • Such columns now round-trip as dictionary<extension<...>> in Go and in other implementations that support the new member.
  • Readers that predate the format change ignore the member and see a plain dictionary of the storage type. Previously they saw the keys on the field, which typically failed or produced an extension type with dictionary storage.
• Integration JSON format: gains an optional "metadata" key in the "dictionary" object.
• Integration programs: the IPC JSON integration, file/stream conversion and Flight integration client/server now register the canonical extension types.

This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer and an AI reviewer.

🤖 Generated with Claude Code

[zeroshade]

Thanks — reviewed while still a draft, since it's gated on the apache/arrow#54959 vote. The overall approach looks right to me:

• Writer/reader symmetry is clean: extension keys of a dictionary's value type go to DictionaryEncoding.custom_metadata, and the reader applies them to the value type before wrapping in DictionaryType, so dict<ext> and ext<dict> are no longer conflated. fieldVisitor is per-field, so dictMeta can't leak, and an empty dictMeta writes no member.
• extensionTypeFromMetadata now always builds fresh key/value slices — good catch on the old append(mdkeys[:i], ...) aliasing the caller's metadata.
• Tests pin the behaviour: disabling the reader's use of DictionaryEncoding metadata fails 13 tests in arrow/ipc.

One thing we'd like changed before this lands:

1. DictionaryEncoding.custom_metadata should survive a Go read→write round trip. Today, at field level, an unregistered extension name (and any other key) stays in Field.Metadata, so Go passes it through unchanged. In the new path, anything extensionTypeFromMetadata doesn't consume — unregistered extension names, their serialized metadata, and any non-extension keys — is dropped on read, so reading and re-writing a dict<unregistered-ext> file silently strips the annotation. We'd rather not introduce that pass-through regression, even as a documented limitation. Moving the leftover keys into Field.Metadata wouldn't work, since that reintroduces the dict<ext>/ext<dict> ambiguity on the next write. The leftover metadata needs to be carried with the dictionary's value type so the writer can emit it back into DictionaryEncoding.custom_metadata. How to do that is open — e.g. a field on arrow.DictionaryType for value-type metadata, or an internal placeholder value type that keeps the unregistered name and serialized metadata — happy to discuss whichever you think fits best. A round-trip test with an unregistered extension name plus an extra non-extension key would pin it.
2. Minor: dictutils.Mapper.ImportField unwraps one level of extension value type; an extension whose storage is itself an extension wrapping a nested type wouldn't be descended. Likely unreachable today.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.