Skip to content

PipelineTestModel cannot be constructed: pipeline_integration_infrastructure.py written against an API that does not exist #435

Description

@jeremymanning

Symptom

Every test in tests/integration/test_pipeline_integration_infrastructure.py fails — 27 at HEAD, all with a TypeError from constructing PipelineTestModel.

Root cause

src/orchestrator/testing/pipeline_integration_infrastructure.py passes keyword arguments that the core dataclasses do not define. Fixing one simply reveals the next; I peeled four before stopping:

Line Passes Actual field
89 ModelCapabilities(max_output_tokens=...) max_tokens
96–97 ModelRequirements(gpu_memory_gb=0, network_access=True) network_access does not exist; requires_gpu expresses the intent
102 ModelMetrics(tokens_per_second=...) not a field
112–114 ModelCost(input_cost_per_token=..., fixed_cost_per_request=...) input_cost_per_1k_tokens, base_cost_per_request

max_output_tokens is Google's generation-config key (used correctly in google_model.py and langchain_adapter.py) — it was evidently copied into a ModelCapabilities call where it means nothing.

This module has therefore never run successfully. It is not a regression; it appears to have been written against an imagined API and never executed.

How it stayed invisible

Two methods in TestPipelineTestProvider shared the name test_pipeline_test_provider_initialization (lines 238 and 335), so the second silently replaced the first and one test never ran at all. Ruff's F811 catches this, but CI lints only src/orchestrator, not tests/.

Fixed in #434 by renaming the second method — which is what surfaced the TypeError chain above.

What this needs

A decision before any code: is this module wanted?

If it is wanted, the four constructor calls need correcting and the mock-response design reconsidering. If not, delete it and its test file rather than leaving 27 permanently red tests that suggest coverage that does not exist.

I deliberately did not half-fix it: correcting two kwargs left it still 100% broken while making the diff look like progress.

Reproduce

ORCHESTRATOR_RUN_INTEGRATION=1 pytest tests/integration/test_pipeline_integration_infrastructure.py -q
# 27 failed

Suggestion beyond this issue

Extend the CI correctness lint to tests/F811 there means a test that silently does not run, which is worse than a failing one. There are currently ~73 correctness findings under tests/.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions