Feature Request: `dy.DataFrame` Support in Collections

[gab23r]

Allow Collections to declare dy.DataFrame members alongside dy.LazyFrame:

class MyCollection(dy.Collection):
    users: dy.DataFrame[UserSchema]      # Eager
    orders: dy.DataFrame[OrderSchema]    # Eager

Motivation

• Small datasets: When working with small data, .collect() is unnecessary ceremony
• EDA workflows: Inspecting results in notebooks/REPLs is more convenient with DataFrames
• Consistency: Schema validation already supports both df and lf - Collections should too

# Current (verbose for small data)
collection = MyCollection.validate(data)
collection.users.collect()  # extra step just to see data

# Proposed (direct)
collection = MyCollection.validate(data)
collection.users  # already a DataFrame

[gab23r]

What do you think about the feature request ? I can propose a PR if this is something you want.

[borchero]

We originally went for lazy frame because that's a functional "superset" of data frames and you can't have a generic over a generic type (i.e. having dy.Collection itself be fully generic over lazy/eager). I don't see a good reason why we can't have the user decide that they'd rather store data frames than lazy frames though.

[gab23r]

OK, I went ahead with an implementation to test if we could add the feature without changing too much the internals.