Decouple the `FileSource` trait from the concrete `FileScanConfig` `DataSource`

[mitchsw]

Is your feature request related to a problem or challenge?

I'd like to build a custom DataSource that reuses an existing FileSource (e.g. ParquetSource) instead of reimplementing all its internals. Today this is not possible without dragging along a FileScanConfig that my custom DataSource doesn't otherwise need.

The architecture diagram in datafusion/datasource/src/source.rs shows a clean top-down layering:

DataSourceExec ──► DataSource (trait) ──► FileScanConfig ──► FileSource (trait) ──► ParquetSource / ...

Roughly speaking, theDataSource handles execution and partitioning and the FileSource handles file-type specifics. But FileSource is currently coupled to FileScanConfig (the concrete DataSource). Its open-time methods take a &FileScanConfig and use it as a "bag of args" (e.g. ParquetSource pulls expr_adapter_factory, limit, preserve_order, ...).

fn create_file_opener(&self, object_store: Arc<dyn ObjectStore>, base_config: &FileScanConfig, partition: usize) -> ...;
fn create_morselizer(&self, object_store: Arc<dyn ObjectStore>, base_config: &FileScanConfig, partition: usize) -> ...;

Decoupling the two would let a custom DataSource reuse any FileSource directly.

Describe the solution you'd like

Pass those inputs explicitly via a small, freely-constructible args struct, replacing retrieval direct from the &FileScanConfig:

#[derive(Debug, Clone)]
#[non_exhaustive]
pub struct FileSourceArgs {
    pub object_store: Arc<dyn ObjectStore>,
    pub batch_size: usize,
    pub limit: Option<usize>,
    pub preserve_order: bool,
    pub file_compression_type: FileCompressionType,
    pub expr_adapter_factory: Option<Arc<dyn PhysicalExprAdapterFactory>>,
}

fn create_file_opener(&self, args: &FileSourceArgs, partition: usize) -> ...;
fn create_morselizer(&self, args: &FileSourceArgs, partition: usize) -> ...;
// with_batch_size removed — batch size now flows through FileSourceArgs

FileScanConfig builds FileSourceArgs inline (behavior unchanged). However, now a custom DataSource can build the same struct and reuse any FileSource too.

This is a breaking change to the public FileSource trait, but the migration path is simple.

I have a draft PR ready for this issue.

Describe alternatives you've considered

1. Do nothing. I can construct a dummy FileScanConfig within my custom DataSource - simply to satisfy the interface - but the construction is quite messy.
2. Extend the DataSource trait with all getter fns that a FileSource may - might(?) avoid a breaking change, but adds coupling that may not make sense for other custom DataSources.
3. Fork ParquetSource, add a custom fn create_morselizer() that doesn't depend on FileSourceArgs - I'd prefer not to diverge from upstream

Additional context

The goal of my custom DataSource is to increase I/O parallelism on an I/O bound workload with plenty of CPU to spare. To achieve this, I am diverging from the FileStream currently tightly coupled in a FileScanConfig.

P.S. this is my first datafusion issue. Thanks!

[Udaytaneja]

Hi mantainrs , I'd like to work on this . Please assign this to me .