[Parquet][Python][Dataset] Allow attaching a known FileMetaData to a ParquetFileFragment #55325

Describe the enhancement requested

When a dataset has a _metadata file, every file's row-group metadata is already available in a single object. From Python there are only two ways to use it, and they sit at opposite extremes:

1. ds.parquet_dataset(metadata_path) — ParquetDatasetFactory pre-populates every fragment, so fragment.metadata is free and no file footer is ever read. But all of that metadata stays resident for the dataset's lifetime.
2. Build fragments yourself with ParquetFileFormat.make_fragment(path, filesystem, partition_expression, row_groups) — memory stays flat, but each fragment re-reads that file's footer on first scan, re-fetching data the caller already had.

There is no middle option: a caller cannot say "here is the FileMetaData for this one file, don't go and read it again".

Measured on a 689-file dataset over S3 (each file ~190 row groups, 13 columns, ~240 KB footer):

Over a query touching 200 files that is ~25 s of latency against ~1.7 GB of permanent RSS. For a job running under a memory cap the fast path is simply unavailable, even though the metadata had already been read once in order to build the fragment list at all.

Why this cannot be worked around in Python today

• pq.ParquetFile(source, metadata=...) does accept a pre-loaded FileMetaData and skips the footer read — 11.9 ms versus 66.1 ms in my measurements, i.e. no I/O at all. But that is the low-level reader, outside the dataset API, so it gives up filter pushdown and row-group selection.
• ParquetFileFragment's constructor and SetMetadata() are private, with ParquetDatasetFactory as the only friend (cpp/src/arrow/dataset/file_parquet.h).
• FileMetaData exposes append_row_groups, row_group(i), set_file_path and write_metadata_file, but no way to slice one file's row groups out of a combined _metadata into a new FileMetaData. So even if an API existed to accept per-file metadata, a caller holding _metadata could not construct the object to pass it.

The C++ pieces already exist and are public — ParquetFileFormat::GetReader(source, options, metadata) takes a FileMetaData, and ParquetFileFragment::EnsureCompleteMetadata(FileReader*) takes a reader — they are simply not wired together through Python.

Suggested shape

Either of these would close the gap:

# a) accept metadata at fragment construction
fmt.make_fragment(path, filesystem, partition_expression, row_groups, metadata=file_metadata)

# b) or let an existing fragment adopt it
fragment.ensure_complete_metadata(metadata=file_metadata)

To make either usable from a combined _metadata, a way to obtain a single file's FileMetaData from it would also be needed — for example FileMetaData.subset(row_group_indices), or a helper yielding per-file FileMetaData keyed by file_path.

With both, a caller could parse _metadata once in a short-lived subprocess, keep only the serialised per-file blobs (FileMetaData.write_metadata_file round-trips through a buffer and re-injects in ~14 ms), and get parquet_dataset()'s zero-footer-read behaviour at the hand-built path's memory profile.

Related: #55324 reduces the cost of the footer read when it does happen; this issue is about not needing it at all.

Component(s)

Python, Parquet