fix(privacy): merge token vault writes across instances; thread-safe SQLite vault

[kevincostner17]

Summary

Fixes two problems with sharing the persistent token vaults.

• JsonTokenVault merges and serialises writes. The vault used to load its file once and rewrite it from that in-memory copy on every put, so two instances on the same path overwrote each other's entries.
  • Each put/save now opens the file and takes an exclusive lock: fcntl.flock on POSIX, msvcrt.locking on byte 0 on Windows, or only the per-instance thread lock when neither is available.
  • Under that lock it always re-reads the file, merges it with the in-memory map, rewrites the file in place, and calls flush() + os.fsync() before unlocking.
  • A put whose entry is already on disk skips the rewrite.
  • __init__ reads under a shared lock, get reloads on a miss when the file has changed, and an empty file loads as an empty vault.
  • The docstring notes that a crash mid-write can leave the file incomplete.
• SqliteTokenVault is thread-safe. It connects with check_same_thread=False, timeout=30.0, and an RLock serialises get, put, __len__ and close. A vault created in the main thread can therefore be passed to apply_privacy_policy running in a worker pool.

Only the two vault classes (plus a small lock helper next to them) change; tokenization, masking and key handling are untouched. The on-disk format is unchanged.

Behaviour changes:

• An empty JSON vault file loads as an empty vault instead of raising JSONDecodeError.
• save() keeps entries that other instances wrote.

Limitation: CI has no Windows job, so the msvcrt path is covered only by a stub-module test.

Rebased onto #399 (merged); the only overlap was the privacy.py import block.

Tests

New tests/test_token_vault_concurrency.py (17 tests, well under a second, tmp_path only):

• JSON vault sharing:
  • Issue repro: two instances on one path keep both mappings.
  • Interleaved puts from two instances.
  • Instance B's get sees A's entry.
  • 8 threads, each with its own JsonTokenVault, and 8 threads sharing one instance: all entries present.
• JSON vault behaviour:
  • An empty file loads.
  • Nothing is written before put.
  • A repeated put skips the rewrite.
  • save() keeps other instances' entries.
  • Tokenize round trip through two instances.
• Locking:
  • Exclusive lock for writes and shared lock for reads (flock).
  • Windows byte-0 locking via a stubbed msvcrt.
  • Fallback with no lock module.
• SQLite vault:
  • Issue repro: used from a ThreadPoolExecutor worker.
  • get/put/len from another thread.
  • 8 threads sharing one vault.
  • Close twice.

12 of the new tests fail on main, including both issue repros. The new file passed 5 runs in a row on each interpreter.

Verification

• ruff check .: passed.
• mypy src/freshdata: no issues.
• pytest -m "not online and not large" on main 78b7790 (after fix(privacy): missing values stay missing, categorical k-anonymity, duplicate labels, fpe audit metadata, NER status #399) + this commit:
  • Python 3.12: 5549 passed, 13 skipped.
  • Python 3.9 / pandas 1.5: 5545 passed, 17 skipped.

Closes #279

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: cd4a222c-900a-45f1-844e-b90fdc258fb2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)