Dbock vacc compat pr - #89
Open
mzouink wants to merge 20 commits into
Open
Conversation
Re-introduces the instance-review workflow on top of upstream/finetuning_refactor: - cellmap_flow/review.py — shared SQL helpers (CLI + dashboard backend) - cellmap_flow/dashboard/routes/review_routes.py — Flask review_bp - cellmap_flow/dashboard/templates/_review_tab.html — review tab UI - cellmap_flow/dashboard/app.py — register review_bp - cellmap_flow/dashboard/templates/_dashboard.html — add Review nav tab + pane Pure-ours port from vacc-compat-pre-refactor-merge (zero collision with upstream's new file layout). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Phase 1b) threading.Semaphore around the full forward block in predict(). Bounds simultaneous GPU forwards regardless of client behavior, so orphaned forwards (e.g. XHR.abort'd by NG after a pan) keep their slot until predict() returns and new arrivals queue at acquire() rather than stacking activations into the OOM cliff. Cap defaults to 24 (margin below the measured 32-64 OOM cliff) and is tunable via CELLMAP_FLOW_MAX_INFLIGHT. Curated re-port of 350e27a (semaphore only). The original commit on vacc-compat was layered on top of 5 diag-only commits (1d6fbe3, 11dcbc5, 0a3bb59, 2d1b753, 5e049cd) so a literal cherry-pick would drag in fwd_diag scaffolding + deferred-raise pattern that the unified integration plan §4.12 says to drop. This is the surviving cap-only subset. Validated on vacc-compat (xargs -P sweep on hgnode02/H100, 0 OOMs at N=48, N=64; reserved pool plateaued at 36.38 GiB). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-introduces the instance-correction workflow on top of upstream's
post-refactor routes/finetune/ layout. Three `*_response(data)` handlers
in a new file, plus the helpers and the spine URL-rewriter they depend
on:
- routes/finetune/instance_correction.py (NEW) — 3 handlers, re-authored
from vacc-compat finetune_routes.py:661/979/1036 to take pre-parsed
data dict instead of inline request.get_json():
* create_instance_correction_response (was 47d5b1f + 946a0ea + c15b87f)
* sync_instance_correction_response (was eef8c8d + b18895c)
* cc3d_relabel_annotation_response (was c5cf4d1)
- routes/finetune/routes.py — register the 3 routes at
/api/viewer/{create-instance-correction, sync-instance-correction,
cc3d-relabel-annotation}.
- dashboard/finetune_utils.py — append 4 helpers required by the
handlers (~451 LOC) plus a backward-compatible mc_target_name=None
kwarg on ensure_minio_serving (Patch 44, multi-ROI stable bucket name):
* create_instance_annotation_volume_from_seg
* minio_backing_store_populated
* sync_instance_correction_from_minio
* cc3d_relabel_instance_correction
- routes/finetune/common.py — add rewrite_minio_url_for_proxy at its
Phase-4 final destination (instead of finetune_utils.py). Phase 4 is
now purely "wire into 3 more callers in annotation_core.py +
annotation_sessions.py" with no file move.
Scope expansion vs the plan's §4.4 spec ("3 handlers + register routes"):
the plan didn't enumerate the 5 helpers the handlers depend on, the
mc_target_name kwarg patch upstream lacks, or the Phase 4 forward port
required for clean import smoke. All surfaced via Phase 1's read-first
discipline.
Imports cleanly: dashboard blueprints register, all 3 new
/api/viewer/* routes attach, instance_correction handler trio
importable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-introduces the 4 static-layer CRUD handlers from vacc-compat
finetune_routes:1775+ (originally commit 9e534f8), plus a surgical
forward-port from Phase 5 of the kwargs the handlers depend on:
- routes/finetune/viewer.py — append 4 `*_response(data)` handlers
re-authored from vacc-compat's `def …():` forms (data is now a
pre-parsed dict per the routes/finetune/ convention):
* add_segmentation_layer_to_viewer_response
* add_image_layer_to_viewer_response
* remove_layer_from_viewer_response
* rename_layer_in_viewer_response
- routes/finetune/routes.py — register 4 routes at
/api/viewer/{add-image-layer, add-segmentation-layer, remove-layer,
rename-layer}, all POST.
- utils/scale_pyramid.py — `get_raw_layer` gains 3 backward-compatible
kwargs the handlers require:
* segmentation=False (wrap as SegmentationLayer not ImageLayer)
* min_scale=0 (skip pyramid levels below sN)
* disable_meshes=False (turn OFF on-the-fly mesh generation —
required for whole-cell-scale label
volumes on memory-tight nodes)
All 7 existing upstream callers use only positional/default args
matching the original (dataset_path, normalize, wrap_raw) signature,
so the additions are non-breaking. Surgical, NOT a wholesale port
from vacc-compat — preserved upstream's remote-zarr code paths
(_is_remote_path, _open_zarr, _join_path) introduced post-divergence.
Scope expansion vs the plan's §4.5 spec ("append 4 functions to
viewer.py from 9e534f8"): the plan didn't enumerate the
`get_raw_layer` kwarg dependency. Without these kwargs the handlers
would TypeError on first call. Same lesson as Phase 2: re-author from
vacc-compat HEAD, map all deps before writing.
Phase 5 sequencing implication: the `get_raw_layer` portion of
Phase 5's dc06561+9b2cae7 disposition (segmentation/min_scale/
disable_meshes) is now done. Phase 5 remains responsible for the
YAML CLI side (cli/yaml_cli.py) that surfaces these kwargs in
extra_layers: config.
Imports cleanly: all 8 /api/viewer/* routes attach; 4 new handlers
importable; signature reflects the 3 new kwargs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply rewrite_minio_url_for_proxy at every server-side MinIO URL
emission point, so the dashboard accessed through spine's nginx
reverse-proxy hands NG a browser-reachable URL instead of an
unreachable http://localhost:<port>/... that mixed-content-blocks.
The helper was forward-ported to routes/finetune/common.py in Phase 2
(when instance_correction.py required it). Phase 4 wires it into the
three remaining server-side emit sites the plan named, plus a fourth
the plan missed:
- annotation_core.py:174 — create_annotation_crop_response (plan site)
- annotation_core.py:266 — create_annotation_volume_response (plan site)
- annotation_sessions.py:315 — load_existing_volume_response (plan site)
- yaml_crops.py:163 — _create_session_annotation_volume (NOT in plan —
upstream-only file post-refactor; same minio_url emission pattern as
the other three, called from load_crops_from_yaml_response in
request-thread context, so the flask.request global proxy is bound)
Each site is identical: insert `minio_url = rewrite_minio_url_for_proxy(
minio_url, request)` immediately after `ensure_minio_serving(...)` and
before the URL is stored in g.annotation_volumes or returned to the
client.
Imports per file: add `request` to existing flask import; add
`rewrite_minio_url_for_proxy` to existing routes.finetune.common
import.
Smoke (unit-style, via app.test_request_context):
in : http://localhost:9002/annotations/vol-X.zarr
- direct-ssh (no X-Forwarded-Host) -> NO-OP (returns unchanged) — PASS
- spine (X-Forwarded-Host: spine.med.uvm.edu, X-Forwarded-Proto: https)
-> https://spine.med.uvm.edu/minio/annotations/vol-X.zarr — PASS,
matches plan §4.6's expected curl response form.
Phase 5 sequencing: still independent. Other Phase 5 finetune_utils.py
patches (mc retry, localhost URL) are orthogonal to URL rewriting.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five vacc-compat commits ported into the new tree, all touching how the
finetuned/inference layer renders and how the viewer is bootstrapped:
- add_finetuned_layer_to_viewer_response (cab5308 Patches 25+26):
accept optional `shader` field in POST body; default shader updated
from logit-space range=[0,255] to range=[0,0.5] with the preferred
non-transparent-during-loading form (commented emitTransparent line
per feedback_shader_format). Logit-space adapter outputs now render
visibly without manual shader editing.
- generate_neuroglancer_url body changes (in neuroglancer_utils.py):
* Patch 36 (1c5d4cb): `_read_data_reference_scale` helper +
`s.dimensions = _ref_dims` pin BEFORE adding any layers, so the
JS NG client's global coordinate space stays locked to the data
layer's smallest scale regardless of layer add order. Session 11
retrospective: paste-coords silently landed out-of-bounds because
a later precomputed extra_layer at a different voxel size
captured the global space.
* e3a3e5f: bake `s.concurrent_downloads = 32` into the viewer-state
JSON, so the cap survives tab reloads. Maps to chunkQueueManager
.capacities.download.itemLimit = total queue cap (NOT in-flight),
cap=8 starves FOV; cap=32 fills cleanly; tunable per-alloc.
* d534d5b: default `blend: "additive"` on inference ImageLayers.
* Preferred shader form (per-color cycle, non-transparent default,
commented emitTransparent toggle).
* Hostname→localhost replacement in viewer_url (from 8580634
Session 4 fixes — separate concern from this commit's main scope
but folded in because the new bake/pin pieces inside the txn would
print an un-reachable hostname without it; same Phase 0.5 finding
family as the pipe-deadlock fix slated for Phase 7).
* port=5000 explicit in create_and_run_app call so the SSH tunnel
can be set up at a stable port.
- dashboard/app.py: `create_and_run_app` gains backward-compatible
`port=0` kwarg to receive the explicit port=5000 above. Default
preserves the random-port behavior all 6 existing callers expect.
Bidirectional divergence (Phase 3 lesson): upstream has
`build_prediction_source` + `override_scales` + `model_configs_by_name`
that vacc-compat lacks (override per-model voxel size for raw overlay
alignment). Preserved upstream's logic; the new bakes/pins are added
inside the same txn without touching `build_prediction_source`.
Decomposition note (deviation from plan §4.7 "batch into 1-2 commits"):
Phase 5 is split into 5a/5b/5c by functional area. 5a is the smallest
self-contained piece (finetuned-layer presentation + viewer
bootstrap perf). 5b will do the extra_layers YAML feature, 5c the
server-side correctness (Cache-Control, mc retry, etc.). Cleaner
bisect surface than a single mega-commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five vacc-compat commits ported as one functional unit — load static
zarr layers (image or segmentation) at dashboard startup from a YAML
config's `extra_layers:` block, with per-layer shader/blend/min_scale/
disable_meshes/layer_type overrides:
- ab0165b (Patch 31) — extra_layers YAML config for startup-loaded zarrs
- d534d5b (Patch 33) — per-layer shader/blend honor + additive default
- 3404ae8 — per-layer min_scale (default 0 = full pyramid incl. s0)
- dc06561 — per-layer disable_meshes opt-in for segmentation
- 9b2cae7 (Patch 38) — segmentation layer_type in YAML
Producer side (cli/yaml_cli.py): after model-config build, iterate
`config["extra_layers"]` (default empty), call get_raw_layer with the
forward-ported segmentation/min_scale/disable_meshes kwargs from
Phase 3, and stage each result as a (layer, shader, blend) tuple in
g._extra_startup_layers keyed by layer name.
Consumer side (utils/neuroglancer_utils.py::generate_neuroglancer_url):
inside the viewer.txn(), after the for-job loop, iterate
g._extra_startup_layers and add each to s.layers. Apply non-None
shader/blend overrides to the layer object first (skip shader override
for SegmentationLayer — NG uses its built-in categorical palette).
Backward-compat: extra_layers absent from YAML = no-op (the existing
config files keep working without modification).
Yaml schema (example):
extra_layers:
- name: base_mito_whole_cell
path: /gpfs1/.../mito_aff.zarr
shader: |
#uicontrol invlerp ...
blend: additive
min_scale: 2
- name: instance_segs
path: /gpfs1/.../instance.zarr
layer_type: segmentation
disable_meshes: true
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final Phase 5 batch — correctness/perf for the request-handling side
of the dashboard. Three orthogonal vacc-compat patches + the
compressor portion of 8580634 (Phase 0.5 carryover family):
- server.py (37ae237 Patch 35): add `Cache-Control: public,
max-age=31536000, immutable` header to CellMapFlowServer chunk
responses. Chunk URLs are content-addressed by viewer/volume token
so the body for a given URL can't change within a session; immutable
lets the browser serve repeats from disk cache with no revalidation.
- cli/yaml_cli.py (d446769 Patch 34): monkey-patch
neuroglancer.server.SubvolumeHandler.get at yaml_cli import time to
set the same Cache-Control header on every NG-server chunk
response. Idempotent (guarded by _cmf_cache_control_patched flag).
- dashboard/finetune_utils.py (815344e perf): replace single
fire-and-forget `mc alias set` with a 5-attempt retry loop +
time.sleep(1). MinIO may not accept connections immediately after
port-bind, so the original `check=True` form was raising
intermittent CalledProcessError on dashboard boot.
- dashboard/finetune_utils.py (8580634 Session 4, two pieces):
* 4× `compressor=zarr.Blosc(...)` -> `compressor=None` in
create_correction_zarr (raw/s0, annotation/s0, mask/s0) and
create_annotation_volume_zarr (annotation/s0). NG's browser-side
reader cannot decode blosc; without this all paint operations
fail at the first chunk fetch.
* minio_url localhost-substitution: ensure_minio_serving's returned
URL now uses `http://localhost:<port>/...` instead of
`http://<compute-node-ip>:<port>/...` so the SSH-tunneled
browser can reach it. Spine routing in Phase 4 (X-Forwarded-Host
rewrite) still works on top — the localhost form is the
pre-rewrite source.
Phase 0.5 carryover note: 8580634 also contains the bsub_utils.py
pipe-deadlock fix that's slated for Phase 7. That piece is independent
of the URL/compressor changes here and stays scheduled for Phase 7.
Phase 5 complete after this commit: 5a (presentation+perf) + 5b
(extra_layers feature) + 5c (server caching+MinIO robustness).
3 commits instead of the plan's 1-2; the decomposition gives a clean
bisect surface and per-area import smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ogger (Phase 6)
Four vacc-compat commits ported, mostly correctness fixes that gate the
end-to-end paint workflow:
- b480d7e KD-fiber sync correctness (Patches A+B+D+E, load-bearing for
"Save Annotations to Disk"). Four surgical patches in
dashboard/finetune_utils.py:
* _copy_chunks_parallel now returns the set of src paths that
successfully copied; caller filters chunk_sync_state by it, so
failed chunks aren't silently locked out of subsequent non-force
diff syncs.
* _make_s3_filesystem adds `skip_instance_cache=True` (THE FIX from
the 2026-05-10 KD-fiber debug session): defeats fsspec's default
instance-caching, which had shared a single poisoned dircache
across every supposedly-fresh verify call.
* _diff_and_sync_chunks computes removed_keys from
known_chunk_state\\remote_chunk_state, then verifies each via a
FRESH s3fs instance before unlinking — guards against partial s3.ls
listings (suspected under heavy spine-routed NG fetch load) that
would otherwise silently unlink chunks still on MinIO. Spurious
keys are re-added to remote_chunk_state so they don't re-trigger
as "new" on next sync.
* extract_correction_from_chunk gains an idempotency short-circuit:
if an existing extract's annotation/s0 byte-matches the source
labels, skip the expensive raw-EM read + scipy resample + zarr
write. Triggered on dashboard restart (wipes chunk_sync_state →
every existing extract marked "changed" against empty known-state).
- 0197483 list-valued postprocess params (Submit-All round-trip).
ChannelSelection.channels=[5] used to render as `<input value="[5]">`
(Jinja str(list)), which gatherPostProcessData echoed back and
/api/process 500'd on int("[5]"). New helper
stringify_list_values_for_template in utils/web_utils.py converts
lists/tuples to comma-joined strings at the template-render
boundary; index_page.py applies it to default_post_process and
default_input_norm. Mirrors list_cls_to_dict's URL convention.
Round-trip smoke confirmed: list -> "5"; scalar unchanged.
- 0adf67e iframe X-Forwarded-Host rewrite. Plan said "may already
exist upstream"; verified-then-port: upstream has zero X-Forwarded-Host
handling in index_page.py, so this DOES need to be ported. When the
dashboard is accessed via spine.med.uvm.edu, rewrite the embedded NG
iframe URL to stay on the same external host — otherwise the HTTPS
page can't load the http://localhost:NGPORT/v/<hash>/ iframe
(mixed-content + cross-origin block). No-op for direct (localhost)
access. Mirrors rewrite_minio_url_for_proxy's X-Forwarded-Host shape
from Phase 4.
- 442db4e logger fix. Configure the cellmap_flow.dashboard namespace
logger at INFO with a stream handler (otherwise INFO calls in
submodules were filtered out — only app's module-local logger had
INFO set), AND demote three high-cardinality periodic-sync log
lines from logger.info to logger.debug so the dashboard log isn't
spammed during normal operation: "Syncing all annotations from
MinIO", "Synced N/M annotations", "No metadata for volume X".
Smoke (import + behavior): all four function patches in finetune_utils
verified by inspection; stringify_list_values_for_template round-trip
test passes ([5]→"5", "x * 2" unchanged); index_page contains both
stringify import and forwarded_host rewrite; cellmap_flow.dashboard
namespace logger is INFO with propagate=False; three sync logger.debug
demotions confirmed. Paint-routed smoke (chunk lands on MinIO via
spine HTTPS + persists on GPFS mirror after autosave) deferred to
Phase 8 per the integration plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…se 7)
Two clusters of work, both load-bearing for sessions that train via the
legacy CorrectionDataset path on VACC (Track C session 1 will do so).
(1) correction_dataset.py — four vacc-compat patches:
- e58f7d0 skip-on-KeyError: wrap the corr_group / has_* probe loop in
CorrectionDataset._load_corrections in try/except KeyError and
`continue` past chunks whose zarr children list disagrees with their
__getitem__ resolution. Triggered by dashboard's periodic_sync thread
racing the trainer's startup scan — create_correction_zarr builds
.zgroup files non-atomically, so the listing transiently contains
"raw" before the subgroup's full .zgroup is in place.
- 1192dc3 skip-and-fall-forward in __getitem__: replace single-shot
try/except with a bounded retry loop (max_attempts =
max(8, len(corrections)//100)) that walks forward through the
corpus on transient read errors. Without this, a single race with
the dashboard's live extract_correction_from_chunk thread kills the
whole training job mid-epoch.
- a4b9b0f oversample_weight kwarg on create_dataloader: when
oversample_weight != 1.0, build a torch.utils.data.WeightedRandomSampler
that upweights chunks whose attrs carry a non-empty
contains_false_merge_labels list. Required for Run 13+ verdict-
calibration workflow (reference_false_merges_schema, feedback_verdict_
calibration). Defaults to 1.0 = no upweighting, so the change is
backward-compatible with existing dataloader callers.
- 78437db symmetric gamma + noise-no-clip in _augment_3d (verify-then-
port: upstream's _augment_3d clipped to [0, 1], destroying COSEM's
negative half — KD fibers at COSEM≈-0.6, basal bodies, mitochondrial
cristae). Replace `np.clip(raw * scale, 0, 1)` and
`np.clip(raw + noise, 0, 1)` with `sign(raw) * |raw|^gamma` followed
by un-clipped Gaussian noise. Equivalent to upstream's behavior for
inputs already in [0, 1] (sign=+1, no clip needed).
(2) bsub_utils.py + finetune_job_manager.py — Phase 0.5 carryover from
8580634 "Session 4 fixes". Dashboard-launched training takes the
`run_locally()` fallback path on VACC (Phase 0.5 verified `bsub` is
not on PATH and `is_bsub_available()` returns False). Upstream's
run_locally + finetune_job_manager pipe each subprocess through
PIPE/`tee` which deadlocks at the 64 KB pipe-buffer once DataLoader
workers start emitting traceback output.
- run_locally(log_file=None) kwarg: when supplied, opens the file
in the parent and feeds child stdout/stderr to it directly (no
pipe). PYTHONUNBUFFERED=1 in the child env so log lines flush
promptly. wait_for_host timeout 180s -> 120s; output-on-premature-
exit log line; LocalJob.job_id = "local-{pid}" (81b88b4 QoL).
- finetune_job_manager._build_command: drop `stdbuf -oL ... 2>&1 |
tee {log_file}` wrapper — same pipe-buffer hazard one layer up
(tee's stdout feeds the Popen pipe). Log redirection is now done
inside run_locally itself when log_file is passed.
- finetune_job_manager: pass log_file=log_file to run_locally in
the local-fallback branch.
Verify-then-decide outcomes (per plan §4.9):
- 78437db: PORT (upstream clips to [0, 1] — confirmed above).
- Patch 24e KL-on-sigmoid distillation: DROP. Plan worried about
MSE-on-logits-while-teacher-sigmoided (fly_run07 case). Upstream's
distillation path uses a single model with disable_adapter_layers
on/off for teacher/student; `pred` and `teacher_pred` always come
from the same architecture, so they're always on the same scale —
no scale-mismatch correctness bug. Patch 24e's training-dynamics
argument (unbounded MSE on logits) is a valid optimization but not
a correctness fix; defer to a separate effort if/when the user
wants the KL option. Decision documented for future re-evaluation.
Smoke (import + sanity): all 4 dataloader patches present in
correct methods (e58f7d0 in _load_corrections, not __init__);
oversample_weight kwarg + WeightedRandomSampler branch wired;
symmetric gamma replaces intensity scaling; bsub_utils.run_locally
accepts log_file kwarg with file-redirect branch; finetune_job_manager
drops tee wrapper and passes log_file. Phase 7 smoke (2-epoch
training via spawn_instrumented_lora.sh) deferred to Phase 8
end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restores the s0-opt-out portion of vacc-compat 815344e perf: when the
YAML has top-level `min_scale: N`, the EM data layer skips s0..s(N-1)
in get_raw_layer (default N=1 skips 6nm s0). Without this, configs
like phase2_review_wholecell_minimal.yaml that set `min_scale: 1`
silently no-op'd on the data layer — the browser would load 6nm chunks
unnecessarily.
Two pieces:
- cli/yaml_cli.py: `g.min_scale = config.get("min_scale", 1)` after
config-load. Default 1 matches existing YAML conventions.
- utils/neuroglancer_utils.py: data-layer get_raw_layer call now
passes `min_scale=getattr(g, "min_scale", 1)`.
Independent of per-layer `min_scale` in extra_layers entries (which
defaults to 0 = full pyramid incl. s0; see Phase 5b commit).
Surfaced during Phase 8 pre-flight when picking
phase2_review_wholecell_minimal.yaml for the smoke test. Caught
because the YAML field was set but had no functional effect — same
"gap the plan didn't enumerate" pattern as the Phase 4 yaml_crops.py
finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaced during Phase 8 pre-flight when bootstrap_dashboard.sh hung
waiting for `cellmap_flow_server fly .* -p \K[0-9]+` in the dashboard
log. Upstream's run_multiple() didn't include `-p N` in the launch
command; the subprocess auto-picked a port (verified: ss -tlnp on
hgnode01 showed cellmap_flow_server bound on 37949), but the
bootstrap script's port-parse rule couldn't find it.
The plan §4.7 Phase 5 disposition listed 10 named commits but did
NOT enumerate this pair. b8c22bf is from April 8 ("Pre-assign
inference server port to eliminate startup wait"); a3d1cd9 from the
same day refines `get_public_ip()` → `localhost` for SSH-tunnel
browser reachability. Same Phase-0.5-style "carryover the plan
didn't list" pattern as the 8580634 pieces folded into Phases 5c/7.
Changes:
- Import `get_free_port` from `cellmap_flow.utils.web_utils`
(already present at line 12)
- Pre-assign port via `get_free_port()` BEFORE constructing the
launch command
- Append `-p {server_port}` to the SERVER_COMMAND string
- Capture `job = start_hosts(..., wait_for_host=False)` (upstream
already supports this signature at bsub_utils.py:511-512)
- Set `job.host = f"http://localhost:{server_port}"` directly
(combines b8c22bf + a3d1cd9 final form)
NOT YET RUNTIME-VERIFIED: this commit was authored on the basis of
verify-then-decide reading of b8c22bf + a3d1cd9 + upstream
prerequisites. The actual end-to-end smoke (bootstrap_dashboard.sh
finds `-p N` in the log, parses INFPORT, completes the spine update,
prints the URL) hasn't been run yet — session paused for context
hygiene before plunging further. Next session's first action: launch
the dashboard on this branch and confirm the smoke is green; if any
further Phase-N-amendment gaps surface, repeat the careful-debug
loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When CMFLOW_PROXY_MODE=spine is exported by bootstrap_dashboard.sh, emit
the spine HTTPS URL (https://<spine>/inf-{port}/) instead of localhost.
Falls back to localhost (current direct-ssh behavior) when unset or
=direct-ssh.
Refines the b8c22bf+a3d1cd9+ee06a94 chain: ee06a94 hard-coded localhost,
which works in direct-ssh (user has SSH -L tunnel forwarding the inference
port to their Mac) but breaks in --proxy spine (browser fetches from the
Mac without that tunnel). Verified end-to-end this session: bootstrap
with --proxy spine emits the spine URL, NG state's mito_aff_trichocyst
source is the spine URL, browser fetches /inf-{port}/.../s0/.zarray etc
through spine and the inference layer renders.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…audit) Two vacc-compat patches the plan §4.7 didn't enumerate; surfaced by the 2026-05-19 bidirectional-divergence audit (β) after Phase 8 visual validation passed for the green-path features. (1) scale_pyramid.py — Patch 30 (vacc-compat 60cc600). get_raw_layer constructed neuroglancer.LocalVolume(voxel_offset=image.offset), but ImageDataInterface.offset is in nm while neuroglancer expects voxel units. Latent bug: any LocalVolume layer with nonzero zarr offset renders displaced by offset_nm/voxel_size voxels in space. Not triggered in Phase 8 α.3 because run15_wholecell_instances had n_segments=0 (segmentation visible but nothing selected) and inference uses zarr:// remote URL, not LocalVolume. Would trigger as soon as the user selects a segment to view in the run15 wholecell catalog (offset_nm=[118352, 976, 976]). (2) web_utils.py — 81b88b4 part 2. list_cls_to_dict stringified all attrs via str(v), turning ChannelSelection.channels=[5] into "[5]" which then failed re-deserialization at int(channel) on URL round-trip via the add-finetuned-layer path. Different code path from Phase 6's stringify_list_values_for_template (0197483, in 92dc515) which fixes Submit-All template render — both are needed. Part 1 of 81b88b4 (LocalJob.job_id = "local-{pid}") was already in 89544c5 Phase 7. β audit summary: 81 vacc-compat-only commits enumerated; ~50 covered by the 13 prior ports; ~17 explicitly dropped per plan §4.12 (OOM diag, dataloader speedups, Patches 24a-f); ~12 in heavily-refactored upstream code paths (server.py, inferencer.py, lora_trainer.py — likely subsumed by the refactor or made obsolete by structural changes; defer to runtime-surfaced exercise rather than pre-port). The 2 gaps above are the only items requiring port to functionality-complete vs vacc-compat. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 0.5 of the zarr-sharding pilot (notes/260519_cellmap_flow_v3_read_support_plan.md). The dashboard /api/viewer/add-image-layer endpoint was returning 500 on v3 sharded paths because get_ds_info uses zarr-python 2 for metadata introspection, and zarr-python 2 can't read zarr v3 stores (zarr.json vs .zarray/.zgroup). funlib.persistence pins zarr<3, so swapping to zarr-python 3 isn't an option for now. Approach: add a thin read-only proxy (ZarrV3Node) that parses zarr.json directly. Data reads still flow through tensorstore (already handles v3 + sharding via its zarr3 driver). No new dependencies. In cellmap_flow/utils/ds.py: - New _zarr_format(path) returns 2/3/None; prefers an explicit zarr.json zarr_format=3 over stray .zgroup markers that zarr-python 2's default mode='a' silently writes against a v3 store. - New ZarrV3Node class duck-types zarr-python 2's Group/Array surface (attrs, shape, chunks, dtype, keys, __getitem__). For sharded arrays, .chunks returns the *inner* chunk shape (compression/IO unit); the outer shard shape is exposed via .shard_shape. - New _is_zarr_group / _is_zarr_root structural-typing helpers replace isinstance(node, zarr.hierarchy.Group) and node.path == "" checks. - _open_zarr dispatches local v3 paths to ZarrV3Node; v2 + remote unchanged. - _is_zarr_container now also recognizes zarr.json. - _detect_filetype now returns 'zarr3' for v3 paths so open_ds_tensorstore picks the right driver. - access_parent + check_for_multiscale accept ZarrV3Node. - get_ds_info gets a v3 branch handling group-level and scale-level array paths; reads OME-NGFF multiscales from the parent group's zarr.json attributes. In image_data_interface.py: - isinstance(ds, zarr.hierarchy.Group) -> _is_zarr_group(ds). In utils/scale_pyramid.py: - get_raw_layer unifies local + remote multiscale-discovery to use _open_zarr + group.attrs (was os.listdir locally, _open_zarr remotely). Works for v2 and v3 uniformly; also handles non-s0/s1 naming (v3 multiscale typically uses 0/1/2). Tests: - New tests/utils/test_zarr_v3_support.py (35 cases) covers format detection, the proxy class, dispatch helpers, access_parent + check_for_multiscale traversal, get_ds_info on single + multiscale v3 stores, and end-to-end ImageDataInterface with the zarr3 tensorstore driver. Fixtures are built in tmp_path so the suite is self-contained. Regression: tests/finetune/ continues to pass 23/23. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When input_voxel_size doesn't exist natively in the source pyramid (e.g. requested 8nm on a 6/12/24nm zarr), ImageDataInterface silently falls back to the closest available scale but leaves self.output_voxel_size at the requested value. Emitting output_voxel_size as the NGFF scale then mis-declares each served voxel's extent (e.g. 8nm declared while the underlying source is actually being read at 6nm), causing LIVE LoRA layers to render 4/3 LARGER than the EM they were read from. Compute the effective scale from roi.shape / vol_shape — this recovers the actual nm-per-output-voxel regardless of the input/output_voxel_size mismatch. Empirically verified on alloc 4271707 (H200, talk-figures dashboard): - kd_fiber LoRA: was emitting scale=8, now emits scale=6 - mito LoRA: was emitting scale=16, now emits scale=12 - LIVE LoRA bbox matches EM bbox in NG cross-section + 3D views - Position bar invariant holds (no NG renormalization drift, unlike the previously-tried per-layer transform override workaround) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.