Make the new-format templater output set granularity-invariant#125
Open
nick-gorman wants to merge 6 commits into
Open
Make the new-format templater output set granularity-invariant#125nick-gorman wants to merge 6 commits into
nick-gorman wants to merge 6 commits into
Conversation
The new-format templater now emits the same table set at every granularity: timeslices and costs_connection are wired in, and the custom-constraint tables are emitted header-only at nem_regions / single_region instead of being absent. This reverses the gating from 9366945 — the PLEXOS constraints are sub-regional export-group limits with no meaningful representation once sub-regions are collapsed, but writing "all columns, no rows" tables means list_templater_output_files needs no granularity awareness and downstream consumers never check for missing tables. costs_connection is also now tracked in the output list — it was being written but untracked, the same class of bug 9366945 fixed for the constraint tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
The three _stub_* helpers built their DataFrames by hand (dict-of-lists), the outliers in a file that otherwise uses the csv_str_to_df convention. Convert them so the fixtures read as row-oriented tables — the timeslice calendar especially, since test_timeslices.py already writes that exact shape with csv_str_to_df. The helpers are module-level, so the fixture is threaded in as a parameter rather than imported. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nem_regions and single_region tests both assert the new-format output at a collapsed granularity, but had diverged: single_region only checked the custom-constraint tables had *some* columns where nem_regions pinned the exact column set, and neither asserted timeslices at all. timeslices is granularity-invariant (decoded from the calendar + reference_year_sequence, independent of regional_granularity), so omitting it left the invariance this PR establishes unchecked at the coarse granularities. Pin the exact custom-constraint column sets in single_region and assert timeslices (two patterns, same as sub_regions) in both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new-format CLI test pins a row count plus referential integrity for every output table except custom_constraints, which only checked that the LHS/RHS rows carried no orphan constraint_ids. Add a principled constraint count (one per CONSTRAINT_NAMES in scripts/extract_plexos_constraints.py) and a drift-detection count for the LHS coefficient rows, so a regression in the populated sub_regions output is caught here rather than only in the per-module tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
empty_custom_constraint_tables() declared the three tables' column sets as module constants, but the populated builders still hardcoded their own copies of the same lists. Nothing coupled the two, so a future change to a builder's output columns would silently diverge from the header-only tables emitted at coarse granularities, breaking the granularity-invariance this branch establishes. Project every builder onto the shared constants so the populated and empty tables have one definition, and split the CLI test's empty-case assertion so a failure names the offending table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The same explanation now lives in empty_custom_constraint_tables()'s docstring, which the else-branch calls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR containts a few small fixes aiming at making sure the templater output is consistent.