Integrate with PSY/IS that supports time series in InfraStore - #38
Draft
daniel-thom wants to merge 32 commits into
Draft
Integrate with PSY/IS that supports time series in InfraStore#38daniel-thom wants to merge 32 commits into
daniel-thom wants to merge 32 commits into
Conversation
A BRANCH record whose CKT is prefixed '@' or '*' is a breaker or switch, and _split_breakers_and_branches! routes it to SWITCHES_AS_BRANCHES. The RAW System Switching Device record carries only a reactance below the zero-impedance threshold -- it has no GI/BI/GJ/BJ columns at all -- and a breaker or disconnect has no line-connected reactor or capacitor to describe. A switching-device row carrying line shunts is therefore malformed input. That admittance was previously dropped without a trace. Each non-zero end now becomes a shunt at the bus it was declared on, and the relocation is warned about rather than silent: a branch with real line shunts is a transmission line, not a breaker or switch, so the source data is worth checking. GI/BI/GJ/BJ are pu on the system base -- a BRANCH record declares no MVA base of its own, unlike a transformer winding with its SBASE1-2 -- while a FIXED SHUNT's GL/BL are MW/MVAr at unity voltage and _make_per_unit! divides every shunt gs/bs by the case base. The value is scaled by baseMVA so that division lands back on the declared pu value. The shunt inherits the device's ST because the RAW defines these shunts as "connected to and switched with the line". The salvage runs before _build_switch_breaker_sub_data, which pops I and J.
These are Claude Code working notes, not project documentation, and should never be committed.
The 12 copies of this file across the psy6 workspace had drifted into 6 distinct versions. Its content now lives in the user-scope `sienna-psy6` skill, which loads on invocation instead of being read in full every session.
The lowercase name only loaded because macOS is case-insensitive; on Linux and in CI the file was silently ignored. Pure rename, no content change.
Mirrors PowerTableDataParser's completed refactor (8cbf8a4): OpenAPISystem now composes PC.SystemDocument + IdRegistry(document) + an always-empty time_series slot instead of duplicating the document's state, IdRegistry delegates id allocation to PC.next_id!(document), and serialize.jl's hand-rolled envelope is replaced by PC.validate_document + PC.write_document. Adds src/openapi/topology.jl (ACBus, Area, LoadZone, Arc-minting scaffolding) and src/openapi/build.jl (build_openapi_system entry point), porting PowerSystemCaseBuilder's read_bus!/read_loadzones! oracle with unit conversion at the set_value! boundary: PowerModelsData's system per-unit power quantities are multiplied back by baseMVA before every set_value! that touches MW/MVAr, since the schemas' ActivePower/ReactivePower have no pu row. Voltage magnitude/ limits and angle need no conversion (already pu-on-base_kv and radians respectively). Later reader stages (load, generation, branch, ...) are same-named stubs that error on demand; build_openapi_system calls only the implemented topology stage.
build_openapi_system was silently dropping every non-topology pm dict section
(gen, load, branch, shunt, facts, dcline, ...) with no warning or error - the
silent-partial-output pattern this effort exists to kill. Add
_warn_unconsumed_sections, called at the end of build_openapi_system: walks the
pm dict, skips scalar/metadata keys and the sections in _CONSUMED_PM_SECTIONS
("bus" only today), and emits one @warn naming every remaining non-empty
section by name and row count. As later sub-tasks add real readers, each
consumed section joins the tuple and leaves the warning; once every section is
accounted for, this becomes a hard error for genuinely unknown pm-dict shapes.
Also fix test_openapi_units.jl's two failing tests, which turned out to be
fallout of this same effort's PowerOpenAPIModels regen (f460cc7) rather than
unrelated drift: TwoTerminalLCCLine.parameter_units dropped "SYSTEM_BASE" down
to a 2-value enum (DEVICE_BASE now plays that role), and ConstantReserve was
replaced by OnlineReserve/OfflineReserve/GroupReserve.
Ports PSCB power_models_data.jl's load, generator (thermal, hydro, renewable, synchronous condenser, storage), and cost-curve readers into the openapi/ pipeline, bug-compatible with the three known oracle defects (renewable rating double base_conversion, generic battery's unconverted thermal_rating, hydro reservoir emitted as HydroDispatch). Wires read_loads!/read_generation! into build_openapi_system in place of their prior stubs.
…it layer Ports PSCB's read_branch!/read_3w_transformer!/read_dcline!/read_vscline!/ read_shunt!/read_switched_shunt!/read_facts! plus the interarea_transfer AreaInterchange block into PFFP's emit layer: Line, TwoWindingTransformer + TransformerCircuit, ThreeWindingTransformer, TwoTerminalLCCLine/ TwoTerminalGenericHVDCLine, TwoTerminalVSCLine, AreaInterchange (bug-compatible with the oracle's D5 #4 unscaled power_transfer), FixedAdmittance, SwitchedAdmittance, and FACTSControlDevice. Wires all four stages into build_openapi_system and extends _CONSUMED_PM_SECTIONS accordingly.
…LTAGE arithmetic Converts the 16 ternary operators flagged in review (branch.jl, dc_branch.jl) to if/else, since the oracle itself uses if/else at those sites and they don't qualify for the verbatim-port exception. Inline-marks the 5 remaining ternaries that ARE verbatim oracle ports (including 2 more found beyond the review's list in read_area_interchanges!). Also removes the dimensionally-nonsensical sys_mbase-scaled dc_setpoint_from/to computation in the unreachable DC_VOLTAGE branch, leaving the loud set_value! error path as the only outcome there.
…lip warn to error Task 13d (final PFFP-readers sub-task): read_switch_breaker! (switch/breaker/ generic_connector -> DiscreteControlledACBranch) and read_attributes! (impedance correction tables -> shared ImpedanceCorrectionData supplemental attributes, deduplicated per (table, winding) to match the oracle's object sharing). Also consumes area_interchange (Area.ext), found while auditing every remaining pm dict section; the rest (substation, areas, owner, zone) are recorded on KNOWN_UNCONSUMED_PM_SECTIONS with reasons. _warn_unconsumed_sections is now _check_unconsumed_sections and errors on any section neither consumed nor allow-listed. Adds the mandatory CZ/CW/CM/MDC discriminator fixture (device base != system base, closing the untestable gap flagged in test_openapi_branch.jl) and regenerates inspect_14bus_json.jl's output via the now-complete emit layer. Full report: SiennaSchemas .superpowers/sdd/.../reports/task-13d-report.md
…/pu discriminator debt
…ries DEVICE_BASE; dual-mode canary; AC-side gap pointers
Trim task-narration and paragraph-padding from the src/openapi/ headers and docstrings, keeping only non-obvious WHY (units conventions, oracle deviations, bug-compatible markers, the VSC RECORDED GAP). Structural cleanups: - attach make_vscline!'s docstring to make_vscline! instead of the helper above it - merge the duplicated dc/ac voltage-control conditions in make_vscline! - drop the redundant validate_document call in to_json (write_document validates) - inline the single-call-site impedance-correction association and cost slope helpers - hoist the six copy-pasted 14-bus fixture consts and _matches_nt into runtests.jl
unit_system = "DEVICE_BASE" previously stamped the flag but left every value in natural units, identical to NATURAL_UNITS. Add a post-build conversion pass (src/openapi/device_base.jl) that walks the built document and divides each power-family field by the component's own device base (or the document's system base, for the few types with none of their own) whenever the document is DEVICE_BASE, mirroring the inverse of PowerSystems' own NaturalUnit importer. Classification is mechanical (PowerCoreOpenAPIModels' declared unit/quantity metadata), with two hand-diffed exceptions (Area/LoadZone peak fields are schema-fixed-natural; FACTSControlDevice has no base_power of its own) and a loud error for anything the pass cannot classify. Extend generation/load/branch tests with DEVICE_BASE assertions for a generator with mbase != sys_mbase, a load, a line rating, and a transformer circuit, hand-deriving the expected per-unit values.
The PSY equivalence oracle (from_openapi round-trip comparison) caught ThermalStandard/HydroDispatch/EnergyReservoirStorage's ramp_limits (quantity ActivePowerChangeRate, MW/min) missing from the power-family quantity set, leaving it in natural units in a DEVICE_BASE document while every sibling MW/MVAr/MVA field converted correctly. Add a regression assertion cross-checked against the oracle's finding.
EnergyReservoirStorage.storage_capacity was silently left in natural units under DEVICE_BASE: its quantity (ElectricalEnergy) is only resolvable through energy_units' instance-level discriminator, and the previous rule treated every instance-dispatched field as :skip, conflating a genuine pu-vs-natural representation switch (correctly untouched) with a natural-unit-choice discriminator that still needs conversion (PSY's own converter divides storage_capacity by device base_power regardless of which energy_units branch is active). Replace the blanket skip with an explicit registry (_DEVICEBASE_INSTANCE_DISPATCHED) that every instance-dispatched (key, prop) must appear in, erroring by name otherwise -- falling through silently is no longer possible by construction. Populated by empirically auditing every real (unit, quantity) pair PFFP's readers produce, which also caught two more previously-mis-skipped fields: TransformerCircuit.controlled_quantity_limits and TwoTerminalLCCLine.transfer_setpoint both genuinely switch quantity with their own discriminator (control_objective / power_mode) and are resolved per component via a second registry (_DEVICEBASE_DYNAMIC_QUANTITIES). Add a storage DEVICE_BASE test (device base != system base, math in comments) and a test pinning the loud-error guarantee on an unregistered instance-dispatched field.
…:dynamic Round 1 classified this field :dynamic, converting it by the circuit's own base_power whenever control_objective resolves to a power-flow quantity (ActivePower/ReactivePower). Wrong: PowerSystems' own to_openapi calls the identical, unscaled _minmax_po(get_controlled_quantity_limits(circuit)) in both DeviceBaseUnit and NaturalUnit (export_handwritten.jl:166-167, :195-196) -- this field never scales with the document convention, regardless of control_objective. Invisible on the 14-bus fixture because every circuit there is control_objective = "FIXED" (already :skip either way). Static :skip now, matching control_limits, with the PSY citation inlined. Mark TwoTerminalLCCLine.transfer_setpoint's kept :dynamic disposition as explicitly parked: PSY has no converter for that type to check against yet, unlike every other registry entry. Add a regression test: a synthetic ACTIVE_POWER_FLOW-objective transformer circuit (base_power != sys_mbase) asserting controlled_quantity_limits passes through unscaled while its sibling power fields still convert.
…two-terminal HVDC, family convention governs
…ue helpers, dedup tests
Matches the document's un-consolidated association tables: a service is a component, not a supplemental attribute, so the membership belongs in service_associations where the document can validate each end against the set it can legally point into. `attribute_type` went with it -- ServiceAssociation has no such column, and the service's own type is already on the component. Nothing in src/ calls this yet; the 14-bus case emits no memberships. The container test is the only caller, and it now asserts the row lands in service_associations and that supplemental_attribute_associations stays empty. The direct push in attributes.jl is untouched: a shared ImpedanceCorrectionData row genuinely is a supplemental attribute link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two assertions this replaces checked that group_index and role were unset on the SupplementalAttributeAssociation row. Those columns no longer exist, so the row type enforces that structurally and the assertions could only throw a FieldError. What is still worth asserting is the intent behind them: an ImpedanceCorrectionData link emits nothing into the plant-family or service tables that replaced those columns. Suite is green: 2369 passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
psy6 carries a rebased copy of this branch's OpenAPI lineage plus six commits
on top, so most conflicts resolve to psy6's side:
- oneOf members emitted wrapped (PC.ValueCurve, PC.InputOutputCurveFunctionData,
PC.TwoTerminalLoss) in cost.jl and dc_branch.jl
- the VSC voltage-control gap closed via setpoint_voltage_units = "DEVICE_BASE",
deleting _vsc_voltage_control_unsupported
- read_substations! and add_supplemental_attribute_association!, moving
"substation" out of KNOWN_UNCONSUMED_PM_SECTIONS
- base_power emitted on FixedAdmittance/FACTSControlDevice, so the latter drops
out of _DEVICEBASE_SYSTEM_BASE_TYPES
- the generator mapping inlined as GENERATOR_MAPPING_ENTRIES_PM, dropping the
YAML dependency and src/openapi/generator_mapping_pm.yaml
Kept from this branch: .claude/CLAUDE.md (psy6 still carries the pre-rewrite doc
referencing the deleted .claude/Sienna.md and PowerFlowData) and the plant/service
table assertions in test_openapi_attributes.jl. test_openapi_container.jl takes
psy6's text plus this branch's supplemental_attribute_associations assertion.
Two fixes git's auto-merge did not flag: Project.toml ended up with two [sources]
tables, and generator_mapping_pm.yaml survived psy6's deletion because neither
side's merge base had it.
[sources] now pins both PowerOpenAPIModels packages by git rev
feat/infrastore-integration across the root, test, and docs environments, matching
the existing InfrastructureSystems pin and the models branch that actually carries
the wrapped-oneOf and hvdc changes.
Suite is green: 2376 passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
SystemDocument and the document functions moved out of PowerCoreOpenAPIModels into the umbrella PowerOpenAPIModels, so this package no longer compiled; they are now reached through the PD alias. Core keeps the value types, the type registry, and DocumentFormatError. DEVICE_BASE becomes COMPONENT_BASE and DEVICE_MVAR becomes COMPONENT_MVAR in the document strings and the unit-basis discriminators. src/pm_io is untouched: its UNIT_SYSTEM_MAPPING targets IS's own UnitSystem scoped enum, which is a different vocabulary and keeps its spelling. test/Project.toml gains the umbrella source alongside the two it already pinned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.