Skip to content

feat: add Opaque type and allow planner disengagement - #49

Draft
lu-pl wants to merge 2 commits into
mainfrom
lupl/opaque
Draft

feat: add Opaque type and allow planner disengagement#49
lu-pl wants to merge 2 commits into
mainfrom
lupl/opaque

Conversation

@lu-pl

@lu-pl lu-pl commented Jul 23, 2026

Copy link
Copy Markdown
Owner

The feature adds an Opaque type which allows to annotate/flag a model field as 'opaque' to the
planner and causes the planner to disengage; for Opaque-annotated fields, the entire
projection of the current context is forwarded as a pl.Struct. I.e. planner semantics do not
apply, recursion and aggregation code paths are NOT triggered.

This e.g. allows a before-validator to act on the raw bindings on the Python level.

Example:

data = [
    {"x": 1, "y": 2, "z": 3},
    {"x": 1, "y": 3, "z": 4},
    {"x": 2, "y": 4, "z": 5},
]

class Model(BaseModel):
    model_config = ConfigDict(group_by="x")

    x: int
    y: Annotated[list[int], Opaque()]

Here, the Opaque flag disengages the planner, i.e. aggregation is not triggered for the y
field. Instead, y will receive the entire struct projection of the current context; the above
produces the following bindings:

[
    {
        "x": 1,
        "y": [
            {
                "y": 2,
                "x": 1,
                "z": 3
            },
            {
                "y": 3,
                "x": 1,
                "z": 4
            }
        ]
    },
    {
        "x": 2,
        "y": [
            {
                "y": 4,
                "x": 2,
                "z": 5
            }
        ]
    }
]

Note that Opaque can also be used to manually handle arbitrary model unions; GraphTy allows only
discriminated union types, because those can be resolved at planning time, i.e. the winning union member
is knowable on the Polars-level and not just on the Pydantic-level.

Closes #6.

@lu-pl
lu-pl marked this pull request as draft July 23, 2026 15:13
@lu-pl

lu-pl commented Jul 23, 2026

Copy link
Copy Markdown
Owner Author

TODO: Tests, tests, tests. 🤖🧪

@lu-pl

lu-pl commented Aug 4, 2026

Copy link
Copy Markdown
Owner Author

I cannot currently gauge the effect the Opaque feature will have on custom reduction strategies as proposed in #36 .

Currently the behavior is like so:

data = [
    {"x": 1, "y": 2, "z": 3},
    {"x": 1, "y": 3, "z": 4},
    {"x": 2, "y": 4, "z": 5},
]

class Nested(BaseModel):
    model_config = ConfigDict(group_by="x")

    raw: Annotated[Any, Opaque()]

    x: int
    y: list[int]

class Model(BaseModel):
    model_config = ConfigDict(group_by="x")

    raw: Annotated[Any, Opaque()]

    x: int

    z: int
    nested: list[Nested]

will produce the following bindings:

{
    "x": 1,
    "z": 3,
    "raw": [
        {
            "y": 2,
            "x": 1,
            "z": 3
        },
        {
            "y": 3,
            "x": 1,
            "z": 4
        }
    ],
    "nested": [
        {
            "x": 1,
            "raw": {
                "y": 2,
                "x": 1,
                "z": 3
            },
            "y": [
                2,
                3
            ]
        },
        {
            "x": 1,
            "raw": {
                "y": 3,
                "x": 1,
                "z": 4
            },
            "y": [
                2,
                3
            ]
        }
    ]
},
{
    "x": 2,
    "z": 5,
    "raw": [
        {
            "y": 4,
            "x": 2,
            "z": 5
        }
    ],
    "nested": [
        {
            "x": 2,
            "raw": {
                "y": 4,
                "x": 2,
                "z": 5
            },
            "y": [
                4
            ]
        }
    ]
}

I think this is correct: Opaque fields receive the raw bindings of the current partition context.

Nonetheless, this needs careful thinking and testing.

@lu-pl

lu-pl commented Aug 18, 2026

Copy link
Copy Markdown
Owner Author

The following model

data = [{"x": 1, "y": 2}, {"x": 1, "y": 3}, {"x": 3, "y": 4}]

class Model(BaseModel):
    model_config = ConfigDict(group_by="y")

    x: int
    y: Annotated[list[int], Opaque()]

currently crashes with

graphty.utils.exceptions.InvalidGroupByError: 
Invalid grouping key 'y' for 'Model'. Grouping keys must reference scalar model fields.

Should it?

@lu-pl

lu-pl commented Aug 18, 2026

Copy link
Copy Markdown
Owner Author

The following model

data = [{"x": 1, "y": 2}, {"x": 1, "y": 3}, {"x": 3, "y": 4}]

class Model(BaseModel):
    model_config = ConfigDict(group_by="x")

    x: Annotated[int, Opaque]
    y: list[int]

currently produces

{
    "x": 1,
    "y": [
        2,
        3
    ]
}
{
    "x": 3,
    "y": [
        4
    ]
}

I.e. the planner does not recognize Opaque for non-structured fields. Should this be the behavior of Opaque in scalar fields/grouping key fields?

I think so, Opaque disengages the planner for structured field types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider final/terminal/leaf indicator for arbitrary multi model unions

1 participant