Continuous data quality and cross-check system for ENTSOG (European Network of Transmission System Operators for Gas) data (now only Germany).
ENTSOG's Transparency Platform aggregates data that European gas TSOs are required to publish under Regulation (EC) No 715/2009 (Annex I, Chapter 3) and the Network Code on Interoperability and Data Exchange Rules (Regulation (EU) 2015/703). At every interconnection point (IP), both adjacent TSOs report independently — their own flows, nominations, and allocated quantities on their own side of the border. The two sides are not automatically checked against each other: unit conventions differ (kWh vs. Nm³, differing gross calorific values), gas-day boundaries and revision timing diverge, and — for the 91 German "pipe-in-pipe" points — a single physical interconnection is split into several virtual points by capacity type and direction that must be recombined before the two sides can even be compared.
This project:
- ingests the ENTSOG API on a schedule and stores every revision as immutable history, not just the latest published value,
- reconciles both sides of each German interconnection point reported by two or more operators, matching each detector to its counterpart and quantifying the discrepancy per point and per operator pair; a mismatched unit between the two sides is detected and excluded from comparison rather than converted, and points where more than one operator shares a single counterpart (the "pipe-in-pipe" case) are flagged for manual review instead of being aggregated automatically,
- computes and publishes an ongoing data quality report — completeness, publication timeliness, frequency of retroactive revisions, unit/type consistency — both platform-wide and per operator.
Each adjacent operator reports independently through its own detector — one row per operator, per point, per flow direction, per day. Two detectors are only comparable when they're a genuine counterpart pair: one operator's exit reading matched against the other operator's entry reading for the same physical flow.
A point can have more than one detector pair if gas can flow either way. Mallnow (Germany/Poland), for example, has two: a DE->PL pair (GASCADE exit / GAZ-SYSTEM entry) and a separate PL->DE pair (GAZ-SYSTEM exit / GASCADE entry) — each reconciled independently, not lumped together as "the two operators at Mallnow."
v1 covers Germany only. Expanding to all of Europe is a longer-term goal.
- Detector pairs reconciliation (core) — pair up operators per interconnection point, surface discrepancies; unit mismatches and multi-operator ("pipe-in-pipe") points are detected and flagged, not yet automatically resolved.
- Data quality scorecard — late publishers, gaps, retroactive edits, broken units/types.
- Hydrogen monitoring — track hydrogen content per point.
- Anomaly detection (secondary) — simple baseline, evaluated against known events.
- Map — geolocate interconnection points.
Apache License 2.0 — see LICENSE.