Symptom
Every test in tests/integration/test_pipeline_integration_infrastructure.py fails — 27 at HEAD, all with a TypeError from constructing PipelineTestModel.
Root cause
src/orchestrator/testing/pipeline_integration_infrastructure.py passes keyword arguments that the core dataclasses do not define. Fixing one simply reveals the next; I peeled four before stopping:
| Line |
Passes |
Actual field |
| 89 |
ModelCapabilities(max_output_tokens=...) |
max_tokens |
| 96–97 |
ModelRequirements(gpu_memory_gb=0, network_access=True) |
network_access does not exist; requires_gpu expresses the intent |
| 102 |
ModelMetrics(tokens_per_second=...) |
not a field |
| 112–114 |
ModelCost(input_cost_per_token=..., fixed_cost_per_request=...) |
input_cost_per_1k_tokens, base_cost_per_request |
max_output_tokens is Google's generation-config key (used correctly in google_model.py and langchain_adapter.py) — it was evidently copied into a ModelCapabilities call where it means nothing.
This module has therefore never run successfully. It is not a regression; it appears to have been written against an imagined API and never executed.
How it stayed invisible
Two methods in TestPipelineTestProvider shared the name test_pipeline_test_provider_initialization (lines 238 and 335), so the second silently replaced the first and one test never ran at all. Ruff's F811 catches this, but CI lints only src/orchestrator, not tests/.
Fixed in #434 by renaming the second method — which is what surfaced the TypeError chain above.
What this needs
A decision before any code: is this module wanted?
If it is wanted, the four constructor calls need correcting and the mock-response design reconsidering. If not, delete it and its test file rather than leaving 27 permanently red tests that suggest coverage that does not exist.
I deliberately did not half-fix it: correcting two kwargs left it still 100% broken while making the diff look like progress.
Reproduce
ORCHESTRATOR_RUN_INTEGRATION=1 pytest tests/integration/test_pipeline_integration_infrastructure.py -q
# 27 failed
Suggestion beyond this issue
Extend the CI correctness lint to tests/ — F811 there means a test that silently does not run, which is worse than a failing one. There are currently ~73 correctness findings under tests/.
Symptom
Every test in
tests/integration/test_pipeline_integration_infrastructure.pyfails — 27 at HEAD, all with aTypeErrorfrom constructingPipelineTestModel.Root cause
src/orchestrator/testing/pipeline_integration_infrastructure.pypasses keyword arguments that the core dataclasses do not define. Fixing one simply reveals the next; I peeled four before stopping:ModelCapabilities(max_output_tokens=...)max_tokensModelRequirements(gpu_memory_gb=0, network_access=True)network_accessdoes not exist;requires_gpuexpresses the intentModelMetrics(tokens_per_second=...)ModelCost(input_cost_per_token=..., fixed_cost_per_request=...)input_cost_per_1k_tokens,base_cost_per_requestmax_output_tokensis Google's generation-config key (used correctly ingoogle_model.pyandlangchain_adapter.py) — it was evidently copied into aModelCapabilitiescall where it means nothing.This module has therefore never run successfully. It is not a regression; it appears to have been written against an imagined API and never executed.
How it stayed invisible
Two methods in
TestPipelineTestProvidershared the nametest_pipeline_test_provider_initialization(lines 238 and 335), so the second silently replaced the first and one test never ran at all. Ruff'sF811catches this, but CI lints onlysrc/orchestrator, nottests/.Fixed in #434 by renaming the second method — which is what surfaced the
TypeErrorchain above.What this needs
A decision before any code: is this module wanted?
src/, and it is built aroundmock_responses— which the contributing guide forbids outright ("No mock objects. Ever.").tests/test_infrastructure.py, which does work.If it is wanted, the four constructor calls need correcting and the mock-response design reconsidering. If not, delete it and its test file rather than leaving 27 permanently red tests that suggest coverage that does not exist.
I deliberately did not half-fix it: correcting two kwargs left it still 100% broken while making the diff look like progress.
Reproduce
ORCHESTRATOR_RUN_INTEGRATION=1 pytest tests/integration/test_pipeline_integration_infrastructure.py -q # 27 failedSuggestion beyond this issue
Extend the CI correctness lint to
tests/—F811there means a test that silently does not run, which is worse than a failing one. There are currently ~73 correctness findings undertests/.