Playground quickstart: 4 issues found getting sandbox/start_playground.sh running (Podman, sfw, stale dataset checksum, node_id dedup bug)

[mikesmullin]

Summary

Followed the Playground guide on a fresh clone (sandbox/start_playground.sh, default csr_readonly mode, panama dataset) and hit four distinct blockers along the way. Filing this as one report since they were found together, but happy to split into separate issues if preferred. Two are environment-specific (Podman, sfw), two are real bugs reproducible on any machine.

Environment: Arch Linux (rolling), kernel 6.18.38-3-lts, x86_64, Podman 6.0.1 (aliased as docker), Python 3.12.12 managed via uv.

---

1. Dockerfile images aren't resolvable under Podman without a registries.conf

sandbox/start_playground.sh → docker build fails on the second stage:

Error: creating build container: short-name "postgres@sha256:4f736ae..." did not resolve to an alias and no containers-registries.conf(5) was found

Both FROM lines in the Dockerfile use short image names pinned by digest (rust:1.96.0-bookworm@sha256:..., postgres:17-bookworm@sha256:...). Docker resolves unqualified names against docker.io implicitly; Podman requires a registries.conf (unqualified-search-registries) to do the same, and machines with no such file (nothing in /etc/containers/ or ~/.config/containers/) fail outright.

Workaround: add ~/.config/containers/registries.conf with unqualified-search-registries = ["docker.io"].

Possible fix: fully-qualify both ARGs in the Dockerfile (docker.io/library/rust:..., docker.io/library/postgres:...). No-op for Docker (docker.io is already its default registry), and removes the Podman dependency on host-level registry config.

---

2. sandbox/start_playground.sh / run_benchmarks.sh hard-require sfw with no usable fallback

Per AGENTS.md, sfw is a host-provided package-manager safety wrapper, not something the repo ships. Both scripts exit immediately if it's not on PATH:

Error: sfw is required before the playground can install Python dependencies.
Install sfw, or provision sandbox/playground/.venv from requirements.txt ahead of time.

The suggested fallback ("provision the venv ahead of time") isn't actually reachable — the command -v sfw check happens unconditionally before the script ever inspects whether sandbox/playground/.venv already exists and is fully populated. So there's currently no way to run the playground at all on a workstation without sfw installed, even if you're willing to provision the venv by hand first.

Suggested fix: only require sfw when the script is actually about to invoke pip to install/change something — i.e., skip the sfw requirement (and the run_venv_pip install calls) if the venv already exists and its installed packages already satisfy requirements.txt.

---

3. Pinned checksum for the panama dataset is stale

sandbox/common/run_benchmarks.py pins:

expected_sha256="a2e37e8b878c12fb8f946d4e85026a4ae9026dc866b1aa925730bc1b50e52914"

for https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip. As the URL implies, ICIJ regenerates this "latest" export periodically (ours came with GENERATED_ON_20260729.txt inside, i.e. regenerated 2026-07-29), so the pin goes stale over time and every fresh playground run fails with:

RuntimeError: Panama Papers / ICIJ Offshore Leaks archive checksum mismatch: expected a2e37e8b..., got 34475194b6a8c2d683fddc55cca02f88f08f0a538521fb13a324975221624380.

Verified the new download is a well-formed, uncorrupted zip (not a MITM/corruption issue) before treating this as a legitimate upstream update. Current sha256 as of 2026-07-31: 34475194b6a8c2d683fddc55cca02f88f08f0a538521fb13a324975221624380.

Given LATEST in the filename, this will keep recurring. Might be worth a periodic re-pin job, or documenting the expected cadence/how to re-pin in the playground guide.

---

4. transform_panama doesn't dedupe node_id across ICIJ's category CSVs, breaking COPY into panama.nodes

With the current dataset snapshot, loading fails:

ERROR:  duplicate key value violates unique constraint "nodes_pkey"
DETAIL:  Key (node_id)=(51122) already exists.
CONTEXT:  COPY nodes, line 785427

Root cause: ICIJ's export legitimately reuses the same node_id across category files when a person/entity plays more than one role (e.g. 51122 = "Peng, Wan-Hsiung" appears in both nodes-officers.csv and nodes-intermediaries.csv, same person). We counted 1,150 such cross-file collisions in the current snapshot. transform_panama (in sandbox/common/run_benchmarks.py) iterates all nodes-*.csv files and writes every row into a combined nodes.csv keyed by node_id text PRIMARY KEY, without checking for IDs already emitted by an earlier file — so the second occurrence always breaks the load.

Fix applied locally (happy to send as a PR): skip a node_id if it's already been seen from an earlier file, so the first occurrence (in whatever file ordering node_files sorts to) wins:

for row in csv.DictReader(handle):
    node_id = first_value(row, "node_id", "id", "_id")
    if not node_id or node_id in node_ids:
        continue
    node_ids.add(node_id)
    ...

---

5. (Environment-specific, informational) venv creation breaks under uv-managed Python shims

Not a pgGraph bug, but noting it in case others hit it: both scripts resolve the interpreter via command -v python3.NN, which on a machine with Python managed by uv (~/.local/bin/python3.12 → shim → real uv-installed interpreter) returns the shim path. Creating a venv through the shim (rather than the interpreter's real, fully-resolved path) writes a pyvenv.cfg with home pointing at the shim's directory (~/.local/bin), which has no lib/python3.12 beside it — so the new venv's own interpreter can't find its standard library and crashes before ensurepip even runs:

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
ModuleNotFoundError: No module named 'encodings'

Fix applied locally in both start_playground.sh and run_benchmarks.sh: resolve the interpreter with readlink -f before using it to create the venv. Harmless for normal (non-shimmed) system Pythons, and fixes it for uv/pyenv/asdf-style setups. Also happy to include in a PR.

---

Happy to open PRs for #1, #4, and #5 (all low-risk, mechanical fixes) if that's useful — let me know which you'd want, or if you'd rather handle them yourselves.

[mikesmullin]

Update: 3 more issues found once the playground was actually running

After getting past the original 4 blockers, we hit three more problems while using the playground itself (clicking around the Streamlit UI). Filing here rather than a new issue since it's the same investigation thread.

6. main() re-verifies/rebuilds the graph on every single script rerun

sandbox/playground/app.py's initialize_graph() (which calls bootstrap.ensure_graph_loaded()) was called unconditionally at the top of main(), with no caching. Since Streamlit reruns the entire script on every widget interaction, this meant the "Preparing Panama graph..." verification (registration check + potential graph.build()) ran again on every click, and the metric strip visibly flashed back to 0/0 each time before repopulating.

Fix: decorate initialize_graph() with @st.cache_resource(show_spinner=False) (same pattern already used for runtime()), so it only runs once per server process.

7. Clicking "Run SQL" could re-trigger graph.build() and wipe the projection

Separately, the "Run SQL" button handler called ensure_graph_loaded(connection, config) again directly (app.py, in the run_clicked branch) — a second, uncached call on top of #6. If _ensure_active_graph()'s status check ever decided a rebuild was needed at that point, it would invoke graph.build(), which takes ~52 seconds for the Panama dataset (2,016,523 nodes / 6,678,534 directed edges, confirmed via graph.build()'s own build_time_ms output). This surfaced as:

psycopg.errors.QueryCanceled: canceling statement due to statement timeout

thrown from inside bootstrap._ensure_active_graph → graph.build(), and afterward graph.status() showed node_count: 0, edge_count: 0 — the cancellation wiped the projection instead of preserving the previously-good build.

Root cause of why the build got canceled: sandbox/playground/execution.py's run_statements() sets statement_timeout via SELECT set_config('statement_timeout', %s, false) — the false makes it session-scoped, not transaction-scoped. Since client.py deliberately reuses one long-lived autocommit connection for the whole server process, that 30-second cap (PlaygroundConfig.statement_timeout_ms, default 30000) set by the first query a user ever ran leaked forward into every later statement on that connection, including any subsequent graph.build() call triggered by #7 — which reliably needs more than 30s and so was reliably killed.

Fixes:

• Removed the redundant ensure_graph_loaded(connection, config) call from the Run SQL handler — the graph is already guaranteed loaded once via nix dev setup opt-in #6's cached initialize_graph() before the SQL editor is even rendered.
• Added a try/finally in run_statements() that resets statement_timeout back to 0 after every execution, so it can no longer leak into later statements on the reused connection.

Verified end-to-end against the real bootstrap.ensure_graph_loaded / execution.run_statements code and a live Postgres connection: one ensure_graph_loaded() call takes ~52s and builds the graph; three subsequent simulated "Run SQL" executions on the same connection each completed in 44-47ms with the projection staying intact throughout (previously it would have been destroyed by the second or third click).

8. Unpinned pandas/pyarrow/numpy/narwhals cause a silent segfault when rendering results

sandbox/playground/requirements.txt pins streamlit and psycopg directly, but not their transitive deps needed for dataframe rendering. pip install resolved those to whatever was newest at install time — pandas==3.0.5, pyarrow==25.0.0, narwhals==2.24.0 (Streamlit's dataframe-compatibility shim) — and that combination segfaults the Python process (no traceback, just process death) the instant st.dataframe(...) tries to render a real query result. Reproduced both via the live server (crashed after clicking a sidebar example + Run SQL) and via Streamlit's own headless AppTest harness against the unmodified app.py.

Fix: pin the dataframe stack to a known-stable combination in requirements.txt:

pandas==2.2.3
pyarrow==17.0.0
numpy==1.26.4
narwhals==1.9.4

Verified via AppTest replaying the exact failing sequence (click a sidebar query example, then Run SQL) — clean run, 3 result tables rendered, no crash, metrics stable.

---

Happy to send all of this (findings #1-8) as one or more PRs if useful — let us know how you'd like it split up.

[evokoa-admin]

Thanks for letting us know, we will investigate and update again!