Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
9ee98c4
feat: use config master files for all versions of supported plexos ve…
May 20, 2026
e3fac81
feat: add new master files to handle new schema or versioning
May 20, 2026
20145b3
test: update test to handle new schema creation
May 20, 2026
11c1d4b
docs: update documentation on new approach of schema creation
May 20, 2026
7e75b15
fix: format xml files
May 20, 2026
59545a1
feat: update db convertion from plexos solution
May 20, 2026
b84ee0b
fix: improve approach to convert plexos solutions to sqlite files
May 20, 2026
e8d6ebd
feat: implement fast reading for large dbs
May 20, 2026
fc38e9f
test: increase test coverage
May 20, 2026
bc49e48
fix: update solution parsing logic and formatter issues
May 21, 2026
20758b7
merge: resolve conflicts with origin/main
Copilot May 21, 2026
0c97e00
fix: resolve merge conflicts with origin/main and fix utils exception…
Copilot May 21, 2026
5ef5651
fix: fix complexity on function and address formatter issues
May 21, 2026
fbeca18
docs: include steps of use for reading solution
May 21, 2026
3f58de3
Merge branch 'main' into ml/plexos-reader
mcllerena May 22, 2026
ea91512
Merge branch 'main' into ml/plexos-reader
mcllerena Jun 8, 2026
d5f4b88
fix: remove plain-string bypass and resolve helper issue
mcllerena Jun 22, 2026
a8bcfcd
fix: reject conditional on seeds_default when a master template is re…
mcllerena Jun 22, 2026
e8bef11
fix: add use transactional mode on import path to fail and roll back
mcllerena Jun 22, 2026
a40d13d
fix: stream xml parsing to avoid bottlenecks
mcllerena Jun 22, 2026
a054c3b
refactor: implement solution reader as an internal module
mcllerena Jun 22, 2026
da8ad7b
docs: add docstring on missing fuctnions
mcllerena Jun 22, 2026
14b23b1
fix: normalize master files to fit format correctly
mcllerena Jun 22, 2026
11f9283
fix: remove unused setuptools entry on pyproject
mcllerena Jun 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
*.sdf
*.tax2010
*.vcf
*.xml

### Database ###
*.accdb
Expand Down Expand Up @@ -315,6 +314,7 @@ Sessionx.vim
[._]*.un~

### VisualStudioCode ###
.vscode
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
Expand Down
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ repos:
language: node
additional_dependencies: ["prettier@3"]
types: [markdown]
exclude: ^CHANGELOG\.md$

- repo: builtin
hooks:
Expand Down
28 changes: 20 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,22 @@
# AGENTS.md

- Assume concurrent human/agent work; never revert or overwrite changes you did not author.
- This repo processes model data into SQLite; watch query/transform cost, memory churn, and fixture representativeness.
- Prefer minimal, behavior-proven changes; add regression coverage for bug fixes and use existing fixtures under `tests/fixtures` or `tests/data` when practical.
- No breadcrumbs after deleting or moving code; remove the old code/comment instead of leaving relocation notes.
- Use `uv sync --all-groups` for setup. For changed areas, run the narrow relevant check first; before broad handoff use `uv run prek run --show-diff-on-failure --color=always --all-files --hook-stage pre-push` when feasible.
- Type-check package changes with `uv run ty check --output-format github ./src/plexosdb`.
- Test package changes with `uv run pytest --cov --cov-report=xml`; benchmark-sensitive changes may need `uv run pytest benchmarks/ -k "not xlarge_300k" --benchmark-only --benchmark-json=benchmark-results.json --no-cov`.
- Build docs for user/operator-facing documentation changes with `uv run sphinx-build docs/source/ docs/_build/`.
- Assume concurrent human/agent work; never revert or overwrite changes you did
not author.
- This repo processes model data into SQLite; watch query/transform cost, memory
churn, and fixture representativeness.
- Prefer minimal, behavior-proven changes; add regression coverage for bug fixes
and use existing fixtures under `tests/fixtures` or `tests/data` when
practical.
- No breadcrumbs after deleting or moving code; remove the old code/comment
instead of leaving relocation notes.
- Use `uv sync --all-groups` for setup. For changed areas, run the narrow
relevant check first; before broad handoff use
`uv run prek run --show-diff-on-failure --color=always --all-files --hook-stage pre-push`
when feasible.
- Type-check package changes with
`uv run ty check --output-format github ./src/plexosdb`.
- Test package changes with `uv run pytest --cov --cov-report=xml`;
benchmark-sensitive changes may need
`uv run pytest benchmarks/ -k "not xlarge_300k" --benchmark-only --benchmark-json=benchmark-results.json --no-cov`.
- Build docs for user/operator-facing documentation changes with
`uv run sphinx-build docs/source/ docs/_build/`.
55 changes: 39 additions & 16 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,38 +36,61 @@ All notable changes to this project will be documented in this file.

## [1.3.4](https://github.com/NatLabRockies/plexosdb/compare/v1.3.3...v1.3.4) (2026-03-27)


### 🧩 CI

* use release/v1 tag for pypa/gh-action-pypi-publish ([#107](https://github.com/NatLabRockies/plexosdb/issues/107)) ([c4e58b8](https://github.com/NatLabRockies/plexosdb/commit/c4e58b8dc062f3302216b3caa7c9c6c1cc423c86))

- use release/v1 tag for pypa/gh-action-pypi-publish
([#107](https://github.com/NatLabRockies/plexosdb/issues/107))
([c4e58b8](https://github.com/NatLabRockies/plexosdb/commit/c4e58b8dc062f3302216b3caa7c9c6c1cc423c86))

### 📦 Build

* **deps:** bump actions/cache from 5.0.3 to 5.0.4 ([#115](https://github.com/NatLabRockies/plexosdb/issues/115)) ([1ff162a](https://github.com/NatLabRockies/plexosdb/commit/1ff162afe66dfe3bcad7d0dbb0a534b4a9d3374a))
* **deps:** bump astral-sh/setup-uv from 7.5.0 to 7.6.0 ([#117](https://github.com/NatLabRockies/plexosdb/issues/117)) ([13fb3cf](https://github.com/NatLabRockies/plexosdb/commit/13fb3cfb4c7a2affd7df504a2e152c9a2b0c1295))
* **deps:** bump astral-sh/setup-uv from b75dde52aef63a238519e7aecbbe79a4a52e4315 to e06108dd0aef18192324c70427afc47652e63a82 ([#114](https://github.com/NatLabRockies/plexosdb/issues/114)) ([96f3975](https://github.com/NatLabRockies/plexosdb/commit/96f397540b06e68e45075a5d700d2b0a91ebe112))
* **deps:** bump codecov/codecov-action from 5.5.2 to 5.5.3 ([#116](https://github.com/NatLabRockies/plexosdb/issues/116)) ([c86c8e2](https://github.com/NatLabRockies/plexosdb/commit/c86c8e254a85909043c8a7b25a10ff7d169d1e02))
* **deps:** bump googleapis/release-please-action from c3fc4de07084f75a2b61a5b933069bda6edf3d5c to 16a9c90856f42705d54a6fda1823352bdc62cf38 ([#112](https://github.com/NatLabRockies/plexosdb/issues/112)) ([4150d56](https://github.com/NatLabRockies/plexosdb/commit/4150d56065f41cf916912daf9cda39281cf4e3df))
* **deps:** bump peaceiris/actions-gh-pages from e9c66a37f080288a11235e32cbe2dc5fb3a679cc to 4f9cc6602d3f66b9c108549d475ec49e8ef4d45e ([#113](https://github.com/NatLabRockies/plexosdb/issues/113)) ([c697f1d](https://github.com/NatLabRockies/plexosdb/commit/c697f1d7642dac4c16cd5ea9e11e327d684ff548))
- **deps:** bump actions/cache from 5.0.3 to 5.0.4
([#115](https://github.com/NatLabRockies/plexosdb/issues/115))
([1ff162a](https://github.com/NatLabRockies/plexosdb/commit/1ff162afe66dfe3bcad7d0dbb0a534b4a9d3374a))
- **deps:** bump astral-sh/setup-uv from 7.5.0 to 7.6.0
([#117](https://github.com/NatLabRockies/plexosdb/issues/117))
([13fb3cf](https://github.com/NatLabRockies/plexosdb/commit/13fb3cfb4c7a2affd7df504a2e152c9a2b0c1295))
- **deps:** bump astral-sh/setup-uv from
b75dde52aef63a238519e7aecbbe79a4a52e4315 to
e06108dd0aef18192324c70427afc47652e63a82
([#114](https://github.com/NatLabRockies/plexosdb/issues/114))
([96f3975](https://github.com/NatLabRockies/plexosdb/commit/96f397540b06e68e45075a5d700d2b0a91ebe112))
- **deps:** bump codecov/codecov-action from 5.5.2 to 5.5.3
([#116](https://github.com/NatLabRockies/plexosdb/issues/116))
([c86c8e2](https://github.com/NatLabRockies/plexosdb/commit/c86c8e254a85909043c8a7b25a10ff7d169d1e02))
- **deps:** bump googleapis/release-please-action from
c3fc4de07084f75a2b61a5b933069bda6edf3d5c to
16a9c90856f42705d54a6fda1823352bdc62cf38
([#112](https://github.com/NatLabRockies/plexosdb/issues/112))
([4150d56](https://github.com/NatLabRockies/plexosdb/commit/4150d56065f41cf916912daf9cda39281cf4e3df))
- **deps:** bump peaceiris/actions-gh-pages from
e9c66a37f080288a11235e32cbe2dc5fb3a679cc to
4f9cc6602d3f66b9c108549d475ec49e8ef4d45e
([#113](https://github.com/NatLabRockies/plexosdb/issues/113))
([c697f1d](https://github.com/NatLabRockies/plexosdb/commit/c697f1d7642dac4c16cd5ea9e11e327d684ff548))

## [1.3.3](https://github.com/NatLabRockies/plexosdb/compare/v1.3.2...v1.3.3) (2026-03-16)


### 🐛 Bug Fixes

* **ci:** harden all workflows per zizmor audit ([#105](https://github.com/NatLabRockies/plexosdb/issues/105)) ([67ca845](https://github.com/NatLabRockies/plexosdb/commit/67ca84584d1e66410dc66b014a9b710a24b00b95))

- **ci:** harden all workflows per zizmor audit
([#105](https://github.com/NatLabRockies/plexosdb/issues/105))
([67ca845](https://github.com/NatLabRockies/plexosdb/commit/67ca84584d1e66410dc66b014a9b710a24b00b95))

### ⚡ Performance

* Improving performance of adding memberships from records ([#104](https://github.com/NatLabRockies/plexosdb/issues/104)) ([1ea4a39](https://github.com/NatLabRockies/plexosdb/commit/1ea4a39612a1bef1a0f290eaeb40441874a2b8f0))

- Improving performance of adding memberships from records
([#104](https://github.com/NatLabRockies/plexosdb/issues/104))
([1ea4a39](https://github.com/NatLabRockies/plexosdb/commit/1ea4a39612a1bef1a0f290eaeb40441874a2b8f0))

### 📦 Build

* **deps:** bump actions/download-artifact from 7 to 8 ([#101](https://github.com/NatLabRockies/plexosdb/issues/101)) ([0e572a0](https://github.com/NatLabRockies/plexosdb/commit/0e572a07a930f6e25f196e98fc879f65f7dd9daa))
* **deps:** bump actions/upload-artifact from 6 to 7 ([#102](https://github.com/NatLabRockies/plexosdb/issues/102)) ([22b8374](https://github.com/NatLabRockies/plexosdb/commit/22b8374aa7ed9d29eb36258a6a5ad16feb2e21c5))
- **deps:** bump actions/download-artifact from 7 to 8
([#101](https://github.com/NatLabRockies/plexosdb/issues/101))
([0e572a0](https://github.com/NatLabRockies/plexosdb/commit/0e572a07a930f6e25f196e98fc879f65f7dd9daa))
- **deps:** bump actions/upload-artifact from 6 to 7
([#102](https://github.com/NatLabRockies/plexosdb/issues/102))
([22b8374](https://github.com/NatLabRockies/plexosdb/commit/22b8374aa7ed9d29eb36258a6a5ad16feb2e21c5))

## [1.3.2](https://github.com/NatLabRockies/plexosdb/compare/v1.3.1...v1.3.2) (2026-02-12)

Expand Down
46 changes: 46 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,57 @@ db.add_membership(
db.to_xml("modified_model.xml")
```

## Versioned Schema Initialization

When creating a brand-new database (not importing XML), you can preload the
matching PLEXOS master template by version:

```python
from plexosdb import PlexosDB

db = PlexosDB()
db.create_schema(version=10)
```

Supported versions: 9, 10, 11, 12.

Accepted inputs include integers, strings, and tuples. For example:

```python
db.create_schema(version="v11.0R4")
db.create_schema(version=(12, 0, 3))
```

## Read PLEXOS Solution ZIP Tables with pandas

```python
from plexosdb import PLEXOS2SQLite
import pandas as pd

PLEXOS_SOLUTION = "/path/to/solution.zip"

client = PLEXOS2SQLite(PLEXOS_SOLUTION, force=True, materialize_on_enter=False)
client.convert()

table = "ST__Interval__Regions__Fixed_Load"
with client as db:
db.materialize_table(table, schema="report")
df_table = pd.read_sql_query(f'SELECT * FROM report."{table}"', db.connection)
```

This pattern is useful when you only need a few report/data tables from a large
solution ZIP, because it materializes tables on demand.

## Documentation

Full documentation is available at
[natlabrockies.github.io/plexosdb](https://natlabrockies.github.io/plexosdb/).

## Related Work

For related previous/current work on querying PLEXOS outputs with DuckDB, see
[plexos2duckdb](https://github.com/NatLabRockies/plexos2duckdb).

## Developer Setup

plexosdb uses [uv](https://docs.astral.sh/uv/) for dependency management.
Expand Down
76 changes: 61 additions & 15 deletions docs/source/howtos/create_db.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,67 @@ from plexosdb import PlexosDB
db = PlexosDB.from_xml("/path/to/model.xml")
```

When using `from_xml(...)`, the schema is created automatically from the XML
content and metadata in that file.

## Create a New Empty Database with a Versioned Master Template

When starting from an empty database (instead of an existing XML), you can now
preload the versioned master template during schema creation.

```python
from plexosdb import PlexosDB

db_base = PlexosDB()

# Load default SQL schema + master template for PLEXOS v10
ok = db_base.create_schema(version=10)
assert ok
```

`create_schema(...)` returns a boolean status. It does not return a new
`PlexosDB` instance.

You can also pass custom SQL with `schema=...`:

```python
custom_schema = """
CREATE TABLE IF NOT EXISTS my_table (
id INTEGER PRIMARY KEY,
name TEXT
);
"""

db_custom = PlexosDB()
db_custom.create_schema(schema=custom_schema)
```

Supported template versions are:

- 9
- 10
- 11
- 12

Accepted version input formats include:

- `10`
- `"10.0"`
- `"v10.0R2"`
- `(10, 0, 2)`

```python
# Equivalent examples
db_base.create_schema(version="v11.0R4")
db_base.create_schema(version=(12, 0, 3))
```

If `version` is omitted, only the SQL schema is created (no master template is
imported).

Call `create_schema(...)` once per database instance. If the schema already
exists, re-running schema SQL is skipped.

## Create empty schema (tables only)

`create_schema()` by itself creates the table structure and config rows, but it
Expand All @@ -30,18 +91,6 @@ db = PlexosDB()
db.create_schema()
```

## Set schema version explicitly

Use `version="..."` when your schema includes a `t_config` row for `Version` and
you want to update that value during schema creation.

```python
from plexosdb import PlexosDB

db = PlexosDB()
db.create_schema(version="11.0")
```

## Create empty schema and seed defaults

Use `seed_defaults=True` when you want a fresh DB that can immediately add
Expand Down Expand Up @@ -102,7 +151,4 @@ VALUES (1, 'System', 1, 1, '00000000-0000-0000-0000-000000000001', 'Default syst

db = PlexosDB()
db.create_schema(schema=schema)

# If your custom schema does not provide Version, you can still set it here:
# db.create_schema(schema=schema, version="10.0")
```
30 changes: 30 additions & 0 deletions docs/source/howtos/import_export.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,36 @@ version = db.version
print(f"Imported PLEXOS model version: {version}")
```

## Importing a PLEXOS solution ZIP and reading report tables

Use `PLEXOS2SQLite` when you want to analyze PLEXOS solution ZIP outputs. With
`materialize_on_enter=False`, you can materialize only the table you need.

```python
from plexosdb import PLEXOS2SQLite
import pandas as pd

PLEXOS_SOLUTION = "/path/to/solution.zip"

client = PLEXOS2SQLite(PLEXOS_SOLUTION, force=True, materialize_on_enter=False)
client.convert()

table = "ST__Interval__Regions__Fixed_Load"
with client as db:
db.materialize_table(table, schema="report")
df_table = pd.read_sql_query(f'SELECT * FROM report."{table}"', db.connection)
```

How to use this flow:

1. Create a `PLEXOS2SQLite` client with `materialize_on_enter=False` to avoid
materializing every derived solution table.
2. Call `convert()` once to create the SQLite database from the solution ZIP.
3. Inside `with client as db:`, call `materialize_table(..., schema="report")`
for the table you want.
4. Read that table with pandas using `pd.read_sql_query(...)` and
`db.connection`.

## Exporting to XML

Export your database to a PLEXOS-compatible XML file:
Expand Down
Loading
Loading