feat: rclone.conf-based NAS sync (named remotes, lsjson reconcile, cleanup hash-gate) - #17
Merged
Conversation
Replace the equipment-written completeness signal (sentinel file / manifest) with a background quiescence poller that syncs each file independently once it has been observed unchanged for a settle window (default 10 min) -- no equipment cooperation, no operator marking. - config: drop per-equipment completeness_signal/sentinel/manifest; add sync.quiescence_minutes / ignore_globs / poll_interval_seconds. The Add-Equipment wizard goes 5 -> 4 steps. - new per-run sync_state.json (per-file synced_signature / verified_at / keep_local, plus cleared_at); ingest.json / IngestWriter / IngestState removed. - QuiescenceSyncPoller replaces StagingWatcher; discovers both staging-area and nas-mode runs. - per-file sync through the SyncQueue (new files column), transports gain --files-from, verifier gains a subset filter, per-file verify reconciliation credits each verified file into sync_state.json. - cleanup honors per-file keep-local and is symlink-safe; the run rollup (syncing/synced/cleared) is derived from sync_state.json. - GUI: per-file sync status including "On NAS" tombstones for cleared runs, plus a keep-local toggle and its API endpoint. - wire the sync pipeline that was never actually started in production -- the poller and the NASSyncClient worker now run via the app lifespan. Also includes earlier same-session fixes: the Add-Equipment wizard's reactive Next button, persistent multi-step state, and working confirm; the encrypted-at-rest keyring fallback (EXLAB_WIZARD_SECRET_PASSPHRASE); and a repo-wide lims_password_present accessor. Design: docs/superpowers/specs/2026-05-21-operator-free-per-file-nas-sync-design.md Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add EXLAB_WIZARD_TEST_MODE env (and unify it with the existing --test tray flag): when set, every loaded equipment.id gains a TEST_ prefix at config-load time. Run dirs and NAS paths inherit it, so test runs land under <nas_root>/TEST_<id>/... -- trivially identifiable and bulk-deletable. Idempotent; re-validates the rewritten config. - TEST_MODE_ENV / TEST_MODE_PREFIX constants in constants/app.py, shared with paths.py. - Rename the test app's seeded equipment to TEST_EQ1 / TEST_RELAY_EQX to match the convention; cascade through 9 e2e flow files. - File list: add NAME / SIZE / MODIFIED / STATUS column headers in proper <thead>/<tbody>. - Test app: keep-local context action now mutates the seeded feed and re-renders so the badge visibly toggles (production routes the same action through SyncStateWriter.set_keep_local). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate the NAS sync subsystem off rsync-over-SSH (key-only) and the rclone.conf-based remote to a single rclone-binary path with inline- configured SFTP and SMB backends, password auth via the existing KeyringStore, and a hybrid integrity model (full SHA verify on freshly synced files via `rclone check --download --files-from`, size+mtime trust in steady state). Adds Slot A: every per-file sync_state.json record gains `verified_sha256` so the offline-audit affordance lost by dropping `.exlab-wizard/checksums.sha256` is recovered. Slot B (scheduled drift audit) deferred to its own spec. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Foundation of the rclone-only NAS sync migration (see docs/superpowers/specs/2026-05-26-rclone-only-nas-sync-design.md): - Config: replace RcloneTransport + RsyncSshTransport with RcloneSftpTransport + RcloneSmbTransport (password-based) and add transport_requires_keyring_password() predicate. - Transports: delete rsync_ssh.py; rename RcloneTransport->RcloneDriver; add check/about/obscure methods + build_rclone_env helper; keep legacy hashsum() alive at the phase boundary (Phase 2 deletes it). - _run.py: extend run_subprocess with env / stdin / mask_for_log forwarding so RCLONE_CONFIG_<remote>_PASS injection works and obscured passwords are redacted from debug logs. - nas_client: thread keyring_store through NASSyncClient ctor and the push/hashsum factories; each closure now fetches the per-equipment password fresh, runs obscure, builds env, and dispatches. - Wizard + Settings: new SFTP/SMB transport radio with conditional fields; review-step banner directs operator to Settings to set the NAS password. - Test fixtures: stub_rclone gains obscure/about/check behaviors plus env-dump probe; stub_rsync deleted; YAML fixtures updated. - 2162 unit+integration tests pass; mypy + ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Address the Phase 1 code-review findings: - nas_client._resolve_env_for_equipment: surface keyring-backend exceptions as AUTH-class TransportError with __cause__ attached, instead of silently swallowing them and misreporting as "password not set". Also explicitly treat empty-string passwords as missing. - enums.TransportType: replace legacy RCLONE / RSYNC_SSH members with RCLONE_SFTP / RCLONE_SMB so the enum reflects the post-migration transport set. - settings.py: refresh the equipment-section docstring to name the new transport radio (rclone_sftp / rclone_smb) and the post- registration credential flow. - tests: new tests/unit/sync/transports/test_run.py covering env forwarding, stdin piping, and mask_for_log redaction; new test_rclone_env.py covering the exact env-key set per backend, the conditional SMB _DOMAIN key, and the four AUTH paths (None keyring_store, None password, empty password, broken keyring). 2176 unit+integration tests pass; mypy + ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the Python SHA pipeline with `rclone check --download --combined` as the integrity authority, and add Slot A: a durable per-file SHA captured at sync time and recorded in sync_state.json. - Verifier (verifier.py): rewritten as a thin adapter that translates a CheckResult (from RcloneDriver.check) into the queue-worker's VerifyResult shape. Deleted compute_local_manifest, verify_against_local, verify_against_remote, format_manifest, parse_manifest, plus the durable <run>/.exlab-wizard/checksums.sha256 artefact + CHECKSUMS_RELATIVE constant. - FileSyncRecord (sync_state_schema.py): adds verified_sha256: str | None. upsert_file accepts the new kwarg; the writer preserves the prior digest when called without it so re-verify passes don't blank out the audit trail (synced_signature / verified_at keep None-overwrites-existing for the rollup-back-to-syncing path). - nas_client._drive_job: re-ordered as (existence checks) -> (Slot A local SHA) -> (push) -> AWAITING_VERIFY -> (rclone check) -> (reconcile from VerifyResult.verified, crediting verified_sha256 from Slot A). _build_transport_driver now returns (driver, push, check); _build_hashsum_callable / _verify_pass / _build_hashsum deleted. force_verify reports only -- never updates verified_sha256. - _discover_run_files: helper that synthesises a whole-run files-from list for enqueues that didn't supply one (force-sync, first-time poller sweep), so the verify path always has a concrete subset. - Transport driver: RcloneDriver.hashsum deleted (legacy probe path). - Stub rclone: drops hashsum behaviors; keeps check + obscure + about. - Tests: deleted test_verifier.py (18 tests for the deleted SHA pipeline); new test_rclone_env.py for build_rclone_env / AUTH paths; test_run.py for env / stdin / mask_for_log; refactored test_nas_client(_extra).py to use check_callable_factory; _helpers.py gains local_check_factory and corrupt_one_check_factory. - 2155 unit+integration tests pass; ruff + mypy clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…A assertions Address the Phase 2 code-review findings: - sync_state_writer.upsert_file docstring: rewrite to spell out the asymmetric None handling -- synced_signature / verified_at always overwrite (poller re-modified flow), verified_sha256 preserves the prior digest when None (audit-trail survival). - nas_client._drive_job docstring + retry-routing comment: drop stale references to deleted _verify_pass / verify_against_remote / remote hashsum probe; name the new rclone check path. - stub_rclone.py: delete the legacy hashsum verb dispatch, the _emit_hashsum helper, the hashsum_success behavior token, the STUB_RCLONE_HASHSUM_PATH env probe, and the orphan hashlib import. The new world has no hashsum invocations to stub. - test_nas_client_extra batch tests: add verified_sha256 assertions on credited records so the Slot A capture path is pinned at assertion level (was implicitly trusted via shape checks before). 2155 unit+integration tests pass; ruff + mypy clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add INCOMPLETE_NO_NAS_CREDENTIAL setup state between NO_EQUIPMENT and NO_LIMS: every nas-mode equipment whose transport requires a keyring password must have its entry before the app is READY. The state maps to a SET_NAS_CREDENTIALS next-action and per-equipment missing-field rows. Wire deps.nas_password_present (hydrated at tray boot from the keyring) and a real deps.equipment_probe that obscures the keyring password, builds the inline rclone env, and runs `rclone about`. POST /setup/test-equipment now requires equipment_id (the pre-save body-equipment path is impossible without a stored password) and 404s on unknown ids. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Hoist `import time` to module top in tray/dependencies. - Add `pass_env_keys_for` to rclone.py __all__ (it is part of the log-redaction security surface, imported by name by the probe). - Extend the setup-state-gate test to cover the NAS-credential and LIMS hard-block states (previously stopped at no_equipment). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a per-equipment NAS-credentials section to Settings, shown only when password-requiring nas-mode equipment exists (inserted after Equipment List). Each row pairs a keyring-backed credential field (writing through per-equipment Save/Clear handlers that also flip the live deps.nas_password_present set) with a Test connection button that runs the rclone probe and renders the result inline. The setup-incomplete banner subline is now next-action aware so the INCOMPLETE_NO_NAS_CREDENTIAL state points the operator at the new section, and _missing_setup_sections surfaces it so opening Settings auto-selects it. Adds unit coverage for section visibility/rendering and the mount handlers, plus e2e flow 27 driving the full set -> ready -> test -> clear round-trip against the production app. Also refreshes the stale EquipmentWizardState seed in the e2e test app. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- first_incomplete_section now folds the dynamic NAS-credentials section into the canonical order (after equipment), so an INCOMPLETE_NO_NAS_CREDENTIAL install auto-selects it instead of landing on the default section. - _is_setup_ready mirrors the NAS-credential gate (via shared _nas_credential_missing helper) so a registered-but-uncredentialed NAS equipment keeps the setup-incomplete banner up and the index redirect correct, instead of reading as ready on LIMS alone. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…se 5) Sweep the legacy transport out of the test surface: - stub_rclone gains STUB_RCLONE_REQUIRE_ENV to assert the production path injects the full inline backend env; the integration happy-path asserts TYPE/HOST/USER/PASS were injected. - e2e flow_00 lifecycle + flow_26 wizard now register rclone_sftp / rclone_smb equipment (was rsync_ssh / rclone_remote), and flow_00 sets NAS passwords post-restart so creation clears the new gate. - wizard_equipment page object exposes the SFTP/SMB fields; ux_catalog swaps the rsync entries for the SFTP/SMB + NAS-credentials affordances (UX_INTERACTIONS.md regenerated). - ProdServer accepts extra_env so the lifecycle fixture pins the keyring to the encrypted fallback. Convert tests/docker/ from SSH-key auth to password auth: SFTP (PasswordAuthentication) + a new Samba SMB service, password set at boot from NAS_PASSWORD, keys/ removed, README/compose/.env/rclone.conf rewritten for the rclone-only model. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- entrypoint.smb.sh: smbpasswd -a crash-looped on restart because the Samba TDB persists in the writable layer; fall back to a plain password change when the user already exists. - Add a :445 healthcheck to the nas-smb service so `docker compose ps` reports readiness. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extract VerifyResult.from_check_result so the three identical CheckResult->VerifyResult translations (Verifier.verify, force_verify, the _drive_job worker) share one source of truth, and drop the never-read `verifier` constructor param + self._verifier field NASSyncClient carried since the Phase 2 collapse. Other simplify-pass targets (shared password+obscure helper, build_rclone_env per-backend DRY, push-vs-verify error classification) were deliberately left: their apparent duplication guards genuinely divergent error contracts and would trade readability for a few saved lines. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Settings "Test connection" probe passed the bare remote name to `rclone about`, which real rclone reads as a local path and fails with "directory not found". Pass `<remote>:` so it targets the SFTP/SMB backend root. Found during live-NAS verification against the Docker fixture; the stub had masked it by ignoring the remote argument. The stub's `about` verb now rejects a colon-less remote, and a new test drives the probe end-to-end through the stub so this can't regress. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The SFTP/SMB compose healthchecks call `ss`, which debian-slim does not ship; both containers reported "unhealthy" despite working. Add iproute2 to both images. Verified: `docker compose ps` now shows healthy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`_is_setup_ready` (drives the `/` index route and `MainPageState.setup_incomplete`) was a hand-rolled mirror of `compute_setup_state` that only recognized the LIMS *keyring* branch. For a disconnected-workstation install whose LIMS slot is satisfied by `offline_catalogue_path` (no stored password), it returned False even though `GET /setup/status` reported `ready` -- so `/main` kept the setup-incomplete banner up perpetually. The banner subline already delegated to `compute_setup_state` via `_setup_next_action`, so the two disagreed: banner shown, but with the generic subline (next_action None). Delegate `_is_setup_ready` to `compute_setup_state` -- the single source of truth the `/setup/status` endpoint and route gate use -- so the main page agrees with the API across every §4.9 gate (paths, orchestrator, equipment, NAS credentials, and both LIMS branches). Best-effort: any evaluation failure degrades to "not ready" so a half-wired backend keeps the operator on the onboarding path. Verified live against the production app seeded with the exact offline-catalogue config: `/main` now renders the file explorer with no banner, matching `/setup/status: ready`. Tests reworked to exercise the real evaluator (the old block leaned on the narrow mirror ignoring equipment/plugin_dir) and add an offline-catalogue readiness regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make orchestrator.staging_root optional. Blank = the device cannot act as a staging PC; nothing is created. label stays required. No /staging directory is ever auto-created without the operator specifying a path; a specified path is created on Settings save. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live config reload — apply config changes in-process instead of forcing a tray relaunch. Settings save now pushes the validated config into the running components via tray.dependencies.apply_live_config, which fans out to apply_config()/reconfigure() on the quiescence poller, NAS sync client, controller/creation, and validator engine. The restart-required gate and relaunch screen are removed from ui.mount. Opt-in staging_root (design: docs/superpowers/specs/2026-05-28-staging-root-opt-in-design.md): - staging_root no longer gates setup; only orchestrator.label is required (it identifies the workstation in every run's creation.json). A device with a blank staging_root simply is not a staging PC. - Replace the dead /staging default with paths.suggested_staging_root(): a pure, side-effect-free OS-appropriate suggestion shown only as the Settings placeholder, nested under exlab-wizard/ (never a bare /staging). - The staging directory is created only when the operator saves a non-empty path (ui.mount._ensure_staging_root); a creation failure is non-fatal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds docs/REMAINING_WORK.md (verified audit of unimplemented / unwired features, categories A-D) and docs/REMAINING_WORK_TASKS.md (13 tracked tasks T1-T13 with status + implementation-note slots to mark complete as integrated). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Production built the creation controller without a readme_generator, so GUI-created projects/runs ran on NoOpReadmeGenerator and shipped a 3-line stub README with no YAML front matter and no readme_fields.json cache. - Inject ReadmeGenerator() in tray.dependencies._build_controller (the sole production constructor; covers the lifespan build and the apply_live_config fresh-build branch). - Replace the controller's flat placeholder ReadmeContext with the canonical layered type from exlab_wizard.readme; the controller's ReadmeGeneratorProtocol and NoOpReadmeGenerator now use the tuple[Path, Path] contract. - Add CreationController._build_readme_context: partitions readme_extra across the template/config/custom layers by id, maps template + config field declarations, and fills the §10.6 system block (created_by = OS user, project = folder name, run = run dir / null). - Presence is already gated by _validate_inputs and the GUI submits no typed extra fields yet, so the generator's strict validation adds no new failures for current creations. - Add end-to-end test asserting four-layer front matter + readme_fields.json. https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
Code review flagged that the §10.6 system block's `project` field must be the machine-safe LIMS short id (e.g. PROJ-0042) recorded in README metadata (§3.1), not the human-readable <project>/ folder segment. Use _short_id_for (matching the original flat ReadmeContext) and correct the acceptance test. https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
The Confirm & Create progress bar was static (active_phase=None): nothing consumed CreationController.subscribe(), and the session_progress component keyed two of its six phases on the wrong strings. - Fix phase-name mismatch: session_progress PHASES/PHASE_LABELS used 'post_validation'/'queueing_sync' while the controller emits the wire-format 'validating_post_creation'/'queueing_nas_sync' (state_machine.Phase). Align the component to the wire-format so a live phase frame maps onto a row without translation. - Add SessionProgressState + apply_frame(state, frame): folds phase/ progress/done frames into render args; ignores input_required (T5) and unknown phases. - Render the confirm-step bar through a @ui.refreshable bound to state.progress, exposing state.progress_refresh. - mount._run_creation consumes controller.subscribe() via _consume_session_progress while the pipeline runs; race-free because _launch creates the event queue before the pipeline starts and the queue buffers early phases. The controller emits no 'progress' frame yet, so the §9.3 per-plugin sub-row stays dormant until the plugin host emits one (handled defensively). https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
The operations_modal component was built and exported but rendered nowhere; there was no way to inspect in-flight sessions from the GUI. - Add SessionStore.iter_sorted() (public, sorted accessor) and a shared project_identifier(request) helper; refactor the /operations route onto them so neither the route nor the UI reaches into store._sessions, and rows are labelled identically. - Add OperationRow.from_session(): maps the §4.7 state machine onto the panel's running/suspended/completed buckets. - main.py: [Operations…] toolbar button (visible when >0 in flight, warning-colored when any need input) and a footer Sync segment that flips to 'N operations need input' and opens the same modal (§3.5.5); MainPageState gains operations_count / operations_input_required. - mount.py: _operation_counts, _build_operation_rows, _open_operations_modal (fresh snapshot per open), and an _open_operation_details 'view log' dialog. Resume/cancel are baseline here (cancel keeps files); the §9.4 dialog and §9.6 disable rule land in T4, the input dialog in T5. - Fix a latent circular import: operations.py now imports SessionState / project_identifier from the controller submodules, not the package. https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
- _cancel_operation opens a Discard/Keep confirm dialog mapped to controller.cancel(id, discard_files=...): Discard removes the partial directory (shutil.rmtree), Keep leaves it as an orphan. Errors are toasted; cancel is a no-op on an already-terminal session. - §9.6 single-equipment concurrency: _operation_counts now also reports an 'active' (strictly non-terminal) count; _build_main_state sets MainPageState.creation_in_flight, and the New Project / Run / Test Run buttons disable (with a tooltip) while any creation is in flight. - Tests for _operation_counts (panel vs active vs input_required). Resume row action still routes through the INPUT_REQUIRED dialog landing in T5. https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
When a plugin suspended mid-creation the wizard hung indefinitely with no way to answer; only the HTTP /resume route could. - New component ui/components/input_required_dialog.py: the §9.1 'Additional input required' dialog -- plugin pill, reason line, one widget per pending_input field (string/text/choice/boolean) two-way bound to a values dict; persistent so it must resolve before the next frame (§9.2). - The T2 progress consumer opens the dialog on an input_required frame and force-closes it on a terminal done/failed frame (plugin timeout). Submit -> controller.resume(sid, values) (errors surfaced as a toast; plugin re-rejection re-emits input_required and re-opens the dialog), Cancel -> the §9.4 cancel dialog. - Split _cancel_operation into a controller-driven _cancel_session core (reused by the dialog) plus a deps resolver; the Operations modal's Resume reads the parked pending_input and re-opens the same dialog. https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
Settings had no editor for OperatorsConfig.allowlist (the controller's allowlist gate was a permanent no-op for GUI users) and rendered content_scan_extensions read-only. - Add a reusable _render_chip_editor(values, ...) in settings.py (add / per-chip delete / optional reset; mutates the draft list in place so persistence rides the existing draft -> finalize -> Save path). - T7: add 'operators' to SETTINGS_SECTIONS (between nas_cleanup and validator) + SECTION_TITLES and an operators section with §7.9 helper text + a chip editor bound to draft.operators.allowlist. Stored verbatim (case-sensitive; trimmed on add). Non-gating. - T10: replace the read-only extensions label with the same chip editor bound to draft.validator.content_scan_extensions, with Reset to defaults and an on-add '.'-prefix validator. - Update the settings section-count test (8 -> 9, operators present). https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
- Consumer leak (Major): the abandoned-session GC closed INPUT_REQUIRED
sessions directly on the store, bypassing the controller's _publish, so
the in-process subscribe() consumer parked forever on queue.get(). The
GC now pushes a terminal {kind: failed} frame onto the session's event
queue so the wizard progress loop wakes and exits.
- FAILED mislabeled (Major): OperationRow.from_session mapped FAILED to
the running bucket (play glyph, no action). Add a STATE_FAILED bucket
(error glyph) so a failed op is labelled distinctly; sort places it
after running, before completed.
- Footer Sync segment is only clickable when operations_count > 0, so a
click never opens an empty Operations panel.
- Fix an inaccurate _open_input_required_dialog docstring (resume does
not reject an empty payload).
Tests for the FAILED bucket and the GC terminal-frame wake-up.
https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
The Application section's three controls were inert: the autostart checkbox was unbound, the tray-status label was a static literal, and the Quit button had no handler. - T8: tray/dependencies seeds deps.autostart_is_registered; _apply_autostart returns the toggle's real is_registered(); the checkbox seeds from it and applies immediately on change (exempt from the draft, §7.13), reverting to the actual post-op state on mismatch. - T9: the tray builder attaches deps.request_quit = tray_app.request_quit; the Quit button is gated behind a confirm and scheduled non-blocking via ui.timer so the HTTP response flushes before shutdown. - T11: the tray builder sets deps.tray_available from a pystray-import probe; the label reflects available / unavailable (window-only) and the window-on-close behavior copy is added. - AppDependencies gains typed autostart_is_registered / request_quit / tray_available fields. Controls disable cleanly when their hook is absent (headless / tests). https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
The Problems tab badge and right-pane summary were hardcoded to 0 and the /problems footer showed 'Last audit: --'; nothing read the audit. - The 30s background _audit_loop now caches tier counts on deps.last_audit_hard / last_audit_soft (single source; avoids a per-render O(tree) re-audit). _build_main_state reads them into MainPageState.problems_count_hard/soft, so the tab badge and the right-pane summary are real (right-pane no longer hardcodes 'Showing 0'). - render_problems_page takes last_audit_at and renders 'Last audit: HH:MM:SS · Next refresh in Ns' with a 1s ui.timer countdown. start_audit_task=True confirmed in the tray build. - AppDependencies gains typed last_audit_hard/soft fields. Deferred (documented in the tracker): wiring the §11.5 override action and the full live WS-delta stream -- render_problems_page expects a view-model shape the raw Validator Finding doesn't provide (a pre-existing, e2e-only mismatch); reconciling that + the in-process override-write path is a follow-up. The counts/last-audit core lands here. https://claude.ai/code/session_01AeEGo2KMn5xq7UNDuap1Wn
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 4 of the rclone.conf NAS-sync migration. The routine post-push verify no longer downloads and rehashes every file via ``rclone check --download``; it now lists the remote subtree with ``rclone lsjson`` and credits each file whose remote size + modtime match local (within ``nas.mtime_tolerance_s``). The expensive download-and-rehash moves to a single integrity gate run immediately before any irreversible local deletion in ``_maybe_cleanup``. - Build the rclone target from the ``nas:`` block (named remote + base_root) via ``_build_target_for_run`` / ``_target_for_equipment``; the driver is constructed from ``nas.rclone_config_path`` + perf dials (``_build_driver``). Credentials now live entirely in the operator's rclone.conf; the keyring/env threading is gone. - ``_drive_job``: after a successful push -> AWAITING_VERIFY, fetch the lsjson manifest and reconcile per file. A complete reconcile promotes to VERIFIED; an incomplete one re-queues (``remote_reconcile_incomplete``) with no backoff so the next sweep re-pushes the laggards. A new ``_handle_verify_transport_error`` preserves the spec §7.1.5 routing for lsjson transport failures (AUTH terminal, NETWORK/UNKNOWN backoff). - ``_maybe_cleanup``: after the §7.1.6 interlocks pass, run a two-stage gate over the tracked files -- an lsjson existence probe then ``rclone check --download``; any failure defers in CLEANUP_ELIGIBLE rather than deleting. - Drop the now-dead ``_resolve_env_for_equipment`` / ``_build_transport_driver`` / ``_build_target_for`` / ``_reconcile_synced_files``. Keep ``_remote_name_for`` and the (now unused) ``keyring_store`` ctor arg for additive compatibility with the not-yet-migrated tray surface. - ``Verifier.verify`` drops the ``env`` / ``mask_for_log`` params. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-gate Update the sync unit + integration suites for the named-remote migration. - ``_helpers.py``: add ``local_lsjson_factory`` / ``missing_one_lsjson_factory`` returning a ``RemoteManifest`` that mirrors (or omits one of) the local files' size + modtime, for injecting via the new ``lsjson_callable_factory``. - ``test_nas_client.py``: drive the routine path through the lsjson reconcile; add ``test_routine_reconcile_marks_synced_from_lsjson`` (asserts the hash-verify ``check`` is NOT called on the routine path) and ``test_routine_reconcile_requeues_when_remote_file_missing``. - ``test_nas_client_extra.py``: re-point the cleanup tests at the lsjson factory + hash-gate; replace the old verify-retry-via-check tests with ``test_cleanup_runs_hash_gate_before_delete`` / ``test_cleanup_aborts_delete_on_hash_mismatch``; recast the partial batch test as a reconcile-incomplete (re-queue) case. Drop the deleted ``_build_transport_driver`` tests. - ``test_rclone_env.py``: drop the deleted ``_resolve_env_for_equipment`` AUTH-path tests; keep the ``build_rclone_env`` / ``pass_env_keys_for`` coverage (those helpers still back the tray probe path). - ``test_nas_sync.py`` + ``stub_rclone.py``: add a ``nas:`` block to the config and ``lsjson`` support to the stub; migrate the integration flows to the reconcile + cleanup-gate model. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tring Add ``test_cleanup_aborts_delete_when_remote_file_vanished_at_gate``: a run reconciles to VERIFIED normally, but the cleanup-gate lsjson existence probe omits a tracked file (present at reconcile, gone at cleanup). A stateful lsjson factory returns the full manifest on the first (reconcile) call and a missing-one manifest on subsequent (cleanup-probe) calls. The injected hash-gate check would pass, so a CLEANUP_ELIGIBLE result can only come from the existence probe -- asserting the run defers and the local data file is not deleted. Also fix the stale docstring in ``test_default_push_factory_uses_real_driver`` which still referenced the removed ``_build_transport_driver``; it now describes the default ``_build_push`` -> ``_build_driver`` path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-equipment NAS-keyring setup gate with one that depends on the configured nas: remote being present in rclone.conf. evaluate_setup_state now takes nas_remote_available; setup_state_missing reports nas.remote. Thread the new predicate through api/setup, api/routers/config, and add the nas_remote_available helper + AppDependencies fields the callers read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The "Test connection" probe now runs `rclone about <nas.remote>:` through the driver (no keyring), and boot hydrates deps.nas_remotes via RcloneDriver.listremotes() so deps.nas_remote_available drives the §4.9 setup gate. Keyring-presence plumbing (_check_nas_passwords_present, _nas_keyring_password) is retained for the not-yet-migrated Settings UI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The api deps now expose nas_remotes + nas_remote_available and the _dependencies.nas_remote_available reader; setup/status and the config router thread the predicate into the §4.9 evaluator so GET /setup/status returns next_action == "configure_rclone_remote" when a nas-mode device's remote is absent from rclone.conf. (Deps fields + helper + api threading landed with the paths-gate commit; this finalizes the tray docstring that no longer feeds the gate.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Phase 9) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ransport rclone.conf NAS-sync migration (Phase 6, Task A). The nas: block now carries the single rclone-remote connection, so nas-mode equipment no longer requires (nor forbids) a per-equipment transport block. A still-declared transport is allowed for backward compatibility; stage-mode rules are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 6 (Tasks B–D). Replace the per-equipment NAS-credential UI with the single rclone-remote model: - Settings: rename the "NAS Credentials" section to "NAS Remote" (NAS_REMOTE_SECTION). The new read-only section shows the configured nas.remote + base_root, a found/not-found badge from nas_remote_available, and a single Test-connection button. No password input, no per-equipment rows. render_settings_page drops nas_password_present_for / nas_credential_handlers / on_test_equipment in favour of nas_remote_available + on_test_connection. - Equipment add-form (Settings + wizard): drop the SFTP/SMB transport radio/fields for nas-mode; the nas-mode build path now sets transport=None. - mount.py: delete _nas_credential_handlers; replace _nas_credential_missing with _nas_remote_missing (gates on nas.remote availability); the readiness missing-list appends "nas_remote". _nas_test_connection now probes the single nas: remote. - Remove the keyring plumbing parked in Phase 5: nas_password_present reader (api/_dependencies), the nas_password_present deps field (api/app), and _check_nas_passwords_present / _nas_keyring_password / the hydration block (tray/dependencies). keyring_nas_username stays defined (Phase 7). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Rename test_settings_nas_credentials.py -> test_settings_nas_remote.py and rewrite for the read-only NAS Remote section: renders the configured remote + base_root, found/not-found badge from nas_remote_available, a single Test-connection control, and asserts NO password input (scoped to the section body). - test_mount.py: drop _nas_credential_handlers tests; replace the nas_credentials missing-section tests with _nas_remote_missing cases (remote unavailable / unset / available). - test_wizard_equipment.py + test_dynamic_form.py: drop the removed SFTP/SMB transport inputs; nas-mode now builds transport=None; stage-mode staging transport path retained. - test_pages.py: first_incomplete now resolves "nas_remote". - Strip the now-dead nas_password_present kwarg from api/tray/integration test AppDependencies constructions; remove _check_nas_passwords_present tray tests + unused keyring_nas_username import. - Add config test_nas_mode_equipment_needs_no_transport_block (Task A) plus a back-compat test that a still-declared nas-mode transport validates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rclone.conf NAS-sync migration Phase 7.1. NAS sync is now purely rclone.conf-driven, so the per-equipment transport machinery is gone: - Delete RcloneSftpTransport, RcloneSmbTransport, the EquipmentTransport discriminated union, and transport_requires_keyring_password from config.models; drop EquipmentConfig.transport. - Simplify the EquipmentConfig model_validator: nas-mode imposes no transport requirement; stage-mode still requires orchestrator_staging_transport. - Remove the now-unused TransportType enum and its constants exports. - Fix SRC callers: nas_client sources bandwidth from the nas: block; the UI equipment-form builder and tray sample-config no longer set transport; api/setup drops the dead EquipmentTransport holdover. - Strip stray transport: blocks from fixtures/tests and move the complete.yaml bandwidth policy onto the nas: block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rclone.conf NAS-sync migration Phase 7.2. NAS credentials live entirely in the operator's rclone.conf, so the per-call credential plumbing is gone: - rclone driver: delete build_rclone_env, pass_env_keys_for, and the obscure helper; drop the env / mask_for_log params from push, check, and about. - _run.run_subprocess: reduce to a plain spawn helper (no env override, stdin piping, or log redaction). - constants: delete keyring_nas_username and KEYRING_USERNAME_NAS_TEMPLATE (LIMS keyring helpers untouched); drop the dead _remote_name_for in nas_client. - tests: git rm test_rclone_env; rewrite test_run + drop the obscure / env-injection tests in test_transports; strip the obscure verb and env-probe handling from the rclone stub. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-equipment mount-based orchestrator_staging_transport block with a global orchestrator.staging_remote / staging_base_root / staging_perf. stage-mode equipment now carry no per-equipment connection block; the staging hop is a second named rclone remote in the same rclone.conf as the nas: remote. - OrchestratorConfig gains staging_remote, staging_base_root, staging_perf - Delete OrchestratorStagingTransport model + OrchestratorTransportType enum - Drop EquipmentConfig.orchestrator_staging_transport and its now-trivial sync-mode validator (extra="forbid" rejects any stray block) - UI: wizard / equipment_form no longer collect stage-mode mount fields - Adapt config / constants / ui / orchestrator tests to the new model Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Branch the per-equipment target/driver by sync_mode, reusing the same RcloneDriver ops and _build_target_for_run helper as the NAS leg (no new rclone operation): - _target_for_equipment: stage-mode -> orchestrator.staging_remote + staging_base_root; otherwise the nas: block (unchanged) - _driver_for_equipment: stage-mode uses orchestrator.staging_perf, sharing nas.rclone_config_path (staging + nas remotes live in the same rclone.conf); nas-mode uses nas.perf - _build_lsjson strip_prefix uses the per-equipment base root - _build_driver refactored to (config_path, perf) Tests: stage-mode target -> stagepc:/staging/<eq.id>/<run.name>, nas-mode target unchanged, stage-mode driver picks staging_perf. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e new gate Add NasConfig(remote="nas01", base_root="/srv/nas") to the ready_config fixture so all nas-mode equipment in the integration tests satisfy the INCOMPLETE_NO_NAS_REMOTE setup gate introduced by the rclone.conf migration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a listremotes verb that prints remotes from STUB_RCLONE_LISTREMOTES env (default "nas01:"), one per line, mirroring the real rclone output. The --config flag was already tolerated (it appears in flags_with_value). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace stale settings-nav-nas_credentials / incomplete_no_nas_credential / settings-nas-test-EQ1 references with the new settings-nav-nas_remote / incomplete_no_nas_remote / settings-nas-test-connection equivalents. - test_flow_00_full_lifecycle.py: Phase 6 updated to navigate to settings-nav-nas_remote section (read-only) instead of setting per- equipment passwords. - test_flow_27: docstring + module skip reason refreshed; test renamed to test_nas_remote_gate_configure_and_test; body rewritten to drive the read-only NAS Remote section; Phase 9B TODO left for the full wiring. - ux_catalog.py: NAS Credentials entries replaced by NAS Remote entries (settings-nav-nas_remote, settings-nas-remote-name, settings-nas-test-connection). - docs/UX_INTERACTIONS.md regenerated from the updated catalog. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Provides operator-ready [nas01-sftp] and [nas01-smb] stanzas pointing at the compose service hostnames (localhost:2222 / localhost:1445) for manual and dockerised integration runs. Password lines are commented out so the file is safe to commit; operators fill in `rclone obscure` output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add test_drive_job_bandwidth_comes_from_nas_block to verify that _drive_job derives its bwlimit_kibps from config.nas.bandwidth (the global nas: block), not from any per-equipment source. Uses a recording push factory that captures bwlimit_kibps on each push call and asserts it equals effective_bandwidth_limit_kibps(nas.bandwidth). BandwidthConfig(upload_mbps=8.0) is schedule-free so the cap applies deterministically at any time of day. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sort the bandwidth import into the correct alphabetical position within the exlab_wizard.sync.* import block and drop the unused handle variable in test_drive_job_bandwidth_comes_from_nas_block. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ansport UI Phase 8 removed the per-equipment SFTP/SMB transport fields from both the Add-Equipment wizard and the Settings equipment form; the NAS connection is now the single nas: remote. These e2e tests still referenced the deleted UI, causing TypeErrors and missing-testid failures. - _test_app.py: drop transport_type/sftp_* kwargs from the /wizard/equipment route's seeded EquipmentWizardState; seed sync_mode="nas" instead. This was the source of the EquipmentWizardState.__init__() TypeError that broke test_flow_16/22. - page_objects/wizard_equipment_page.py: remove the transport_type/sftp_*/smb_* locator properties for deleted UI; add nas_note/stage_note for the new sync-mode step. - test_flow_26_equipment_wizard_persist.py: step 3 now waits on the nas-mode note instead of filling removed SFTP fields. - test_flow_00_full_lifecycle.py: Phase 4 creates two nas-mode devices via the no-transport Settings form; drop the rclone_smb radio assertion and the now unused _pick_radio helper. - ux_catalog.py: remove the 7 removed-UI entries (settings-equipment-transport + 3 sftp + 3 smb); docs/UX_INTERACTIONS.md regenerated. tests/e2e: 64 passed, 3 skipped, 0 failures/errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…NAS sync Add docs/setup/rclone-remote-setup.md: operator walkthrough for creating SFTP and SMB rclone remotes, wiring nas.remote/base_root into config.yaml, performance and modtime-tolerance tuning, encrypted-config and tray-user caveats, and stage-mode staging_remote setup. Update design spec sections to reflect the rclone.conf named-remote model: - §09: replace per-equipment transport: blocks with top-level nas: block (remote, base_root, rclone_config_path, mtime_tolerance_s, perf, bandwidth); update orchestrator: to staging_remote/staging_base_root; drop nas keyring refs; add missing sync: fields (quiescence_minutes, poll_interval_seconds, ignore_globs). - §07 §7.1.1: single RcloneDriver replaces rclone+rsync dual-driver model. §7.1.3: document the four RcloneDriver ops and named-remote target composition. §7.1.4: replace stale sha256-walk verify description with two-tier model (lsjson size+mtime reconcile post-push; rclone check --download hash gate at cleanup only). §7.1.7: bandwidth is global on nas: block, not per-equipment. §7.1.8: NASSync manages no credentials. §7.4.1: remove stale nas keyring row. - §04 §4.9.1: add INCOMPLETE_NO_NAS_REMOTE state + configure_rclone_remote next_action. §4.9.3: add configure_rclone_remote to next_action list. §4.9.5 step 2: update equipment setup description. Fix two stale test-equipment API endpoint rows. - Aggregated ExLab-Wizard_Design_Spec.md: update §7 and §9 blurbs. - ExLab-Wizard_Frontend_Spec.md: remove per-equipment NAS HTTP-basic password references; update equipment sub-dialog Transport group to named-remote model. Update README.md: add NAS sync setup section pointing at the new guide. Update tests/docker: rewrite README.md to describe rclone.conf-based approach (remove build_rclone_env/RCLONE_CONFIG_* env-injection prose); update rclone.conf and rclone.conf.example comments; fix docker-compose.yml header comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
§07 §7.1.1: the component diagram listed "- rclone / - rsync-over-ssh" as Transport Drivers and the prose said "the rclone or rsync transport drivers" — contradicting the corrected "sole transport driver is RcloneDriver" line and the code (sync/transports/rclone.py is the only driver). Update the diagram to show RcloneDriver's actual ops (copy/lsjson/check/about/listremotes) and reword the subprocess-lifetime line to reference only the rclone driver. §07 §7.1.4: "No temporary files are written to disk" was literally false — RcloneDriver.check writes the --combined output and the --files-from list as small tempfiles (immediately unlinked). Reword to clarify no *data* is staged; only two small, immediately-cleaned-up tempfiles touch disk. §04 §4.9.1: add the pre-existing-omitted INCOMPLETE_NO_ORCHESTRATOR setup state (enums.py:151), in evaluator order between MISSING_PATHS and NO_EQUIPMENT. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Quality-only cleanup of the rclone.conf NAS-sync migration. No behavior change: same rclone targets, same driver perf dials, same setup-state output; public method signatures the tests call are preserved. - nas_client: collapse the three duplicated ``sync_mode`` branches (_target_for_equipment / _base_root_for_equipment / _driver_for_equipment + the lsjson strip_prefix) behind one ``_resolve_remote`` returning a ``_RemoteResolution(remote, base_root, perf)``. Drop the now-redundant _base_root_for_equipment. Rename _build_target_for_run to _remote_subpath so the run-relative path composition is shared by both the target string and the lsjson strip_prefix instead of being open-coded twice. - rclone: extract _checkers_flags() so push/check/lsjson forward the --checkers dial through one helper (argv order unchanged). - paths: convert setup_state_missing's exhaustive if-chain to a match, dropping the unreachable trailing ``return []`` Pyright flagged. - tray/dependencies: correct the stale _build_nas_sync docstring that still claimed keyring resolves per-equipment NAS passwords (migration removed credential injection; the param is accepted for compat only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The simplify pass converted setup_state_missing to an exhaustive match over SetupState but dropped the trailing `return []`, so an unrecognized state fell through to an implicit `None` -- breaking test_setup_state_missing_unrecognized_state_returns_empty and violating the function's `list[dict[str, str]]` contract. Restore the defensive fallback (mypy warn_unreachable is off, so the type checker stays clean). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI was red on three counts after the rclone.conf NAS-sync migration:
- `ruff format --check` flagged 16 unformatted test files -> reformatted
(ruff 0.15.15, matching CI).
- coverage fell to 89.11% (gate is 91%) because the new code was
under-tested. Add real unit tests covering:
* sync/manifest.py edge cases (unparseable modtime, non-list JSON,
empty path, non-numeric Size) -> 100%
* sync/transports/rclone.py error paths (missing binary, auth failure
with no combined output, unreadable combined file, malformed about
JSON, UNKNOWN classification) -> 100%
* ui/mount.py module-level helpers (NAS test-connection, operations /
session dialogs, file-context + keep-local, run wizard, staging
state) -> ~82%
* ui/pages/settings.py NAS-remote section + save/quit/equipment/chip
handlers -> ~97%
Total coverage 89.11% -> 91.68%; 2258 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the NAS sync subsystem from inline per-equipment credentials (OS keyring +
RCLONE_CONFIG_*env injection) to operator-managedrclone.confnamed remotes.nas:block —remote,base_root,rclone_config_path,mtime_tolerance_s,perf{transfers,checkers},bandwidth. Per-equipmenttransport:blocks and the NAS keyring are gone; equipment entries carry just id/label/local_root/nas_root/sync_mode.RcloneDrivermethod (copy/check/about/lsjson/listremotes); nothing else shells out to rclone.rclone lsjson(credited when present AND size-equal AND modtime withinmtime_tolerance_s); the expensiverclone check --downloadhash-verify runs only as the integrity gate immediately before local deletion at cleanup (streams + hashes in memory, no data staged to disk) and on manual force-verify.incomplete_no_nas_remote/configure_rclone_remote); Settings shows a read-only "NAS Remote" status + "Test connection" (rclone about <remote>:). No NAS password entry.orchestrator.staging_remote/staging_base_root/staging_perf), reusing the same driver ops + target helper.--transfers/--checkers(also the memory dial for space-constrained acquisition machines).obscure/build_rclone_env/env-injection plumbing.docs/setup/rclone-remote-setup.md+ updated design-spec sections (04/07/09) + README.Design & plan:
docs/superpowers/specs/2026-05-28-rclone-conf-nas-sync-design.md,docs/superpowers/plans/2026-05-28-rclone-conf-nas-sync.md.Test plan
uv run pytest tests/unit/config tests/unit/sync tests/unit/api tests/unit/tray tests/unit/ui tests/unit/orchestrator tests/integration -q→ 1230 passed, 3 skippeduv run ruff check src/exlab_wizard tests+uv run mypy src/exlab_wizard→ cleangrep -rnE "run_subprocess|['\"]rclone['\"]" src/exlab_wizard | grep -v sync/transports/→ zerokeyring_nas_username,build_rclone_env,RCLONE_CONFIG_, transport models, etc.) → zero insrctest_flow_27parked (skip)Known follow-ups (non-blocking)
test_flow_27_nas_credential_settings.pyparked behind a skip (behavior covered at unit/integration).keyring_storeconstructor param onNASSyncClient; two stale docstrings.🤖 Generated with Claude Code