Nullable column missing fails validate(..., cast=True)

[mattijsdp]

If a column is nullable, why do we not allow dataframely to create the column if it doesn't exist yet (filled with nulls) when casting? I realise you've probably thought about this but I couldn't find an explicit mention of this in the docs or GitHub issues.

import dataframely as dy
import polars as pl

class TableSchema(dy.Schema):
    column_a = dy.String(nullable=False)
    column_b = dy.String(nullable=True)

df = pl.DataFrame({"column_a": 0})
TableSchema.validate(df, cast=True)

# Raises SchemaError
# SchemaError: 1 missing columns for schema 'TableSchema': 'column_b'

[borchero]

One key design decision in dataframely today is that is doesn't ever alter inputs to validate. We are currently re-evaluating this decision though (also in the context of #237) and will revisit this once we've reached a conclusion :)

[brendancooley]

One key design decision in dataframely today is that is doesn't ever alter inputs to validate. We are currently re-evaluating this decision though (also in the context of #237) and will revisit this once we've reached a conclusion :)

Is this in tension with the behavior noted by Ion Koutsouris (@ion-elgreco) in #258 (silently dropping extraneous columns)? Should operations that mutate the data frame be handled explicitly by different methods? filter, derive, etc?

[borchero]

Yeah, definitely related, there is also #100... all of those are in the same vein and likely warrant an extension to the API in one or multiple ways