Skip to content

feat: add wait helpers to run all cells #9545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pjwerneck merged 6 commits into from
Oct 7, 2026
Merged

feat: add wait helpers to run all cells #9545

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
39e4fa9
feat: add wait helpers so demo notebooks can run all cells
pjwerneck Sep 30, 2026
be665f3
docs: clarify job status waiting conditions in SyftRDSClient
pjwerneck Oct 1, 2026
8669836
refactor: rename _attest_options to _given_shorthands for clarity and…
pjwerneck Oct 1, 2026
30bcee7
refactor: remove redundant tests for job and peer waiting conditions
pjwerneck Oct 1, 2026
585e07a
Merge branch 'dev' into pjwerneck/wait-for-helpers
pjwerneck Oct 7, 2026
c69a467
chore: retrigger checks
pjwerneck Oct 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 30 additions & 37 deletions notebooks/enclave/double-blind-eval-w-receipts/1. DO-benchmark-owner-dbe.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,8 @@
"source": [
"### Step 1.1 — Wait until both accept\n",
"\n",
"The enclave accepts automatically; the model owner approves us from their notebook. Re-run the cell\n",
"below until both show as accepted."
"The enclave accepts automatically; the model owner approves us from their notebook. The cell below\n",
"shows both peers, and the cell after it waits until the model owner has approved."
]
},
{
Expand All @@ -169,11 +169,11 @@
"metadata": {},
"outputs": [],
"source": [
"# Fail on a wrong email, or if the peer has not approved yet. Warn if the peer\n",
"# was already there before this notebook logged in.\n",
"# attest_peer() below checks the enclave.\n",
"benchmark_owner.validate_peer(MODEL_OWNER_EMAIL)\n",
"print(f\" Connected to {MODEL_OWNER_EMAIL}\")"
"# Wait up to 5 minutes for each peer to approve. Fail at once on a wrong email.\n",
"# Warn if the peer was already there before this notebook logged in.\n",
"for peer_email in [MODEL_OWNER_EMAIL, ENCLAVE_EMAIL]:\n",
" benchmark_owner.wait_until_peered(peer_email)\n",
" print(f\" Connected to {peer_email}\")"
]
},
{
Expand Down Expand Up @@ -216,14 +216,11 @@
" expected_syft_version=None,\n",
")\n",
"\n",
"result = benchmark_owner.attest_peer(ENCLAVE_EMAIL, policy=policy)\n",
"\n",
"if result is None:\n",
" print(\" 🟠 The enclave published no attestation yet. Wait a moment and re-run this cell.\")\n",
"else:\n",
" result.print_checklist()\n",
" print()\n",
" print(f\" Bound to the report: {result.verified_key_bundle is not None}\")"
"# Wait until the enclave publishes its attestation, then verify it.\n",
"result = benchmark_owner.wait_until_attested(ENCLAVE_EMAIL, policy=policy)\n",
"result.print_checklist()\n",
"print()\n",
"print(f\" Bound to the report: {result.verified_key_bundle is not None}\")"
]
},
{
Expand Down Expand Up @@ -430,11 +427,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Unlike `jobs`, `datasets` does not sync when you read it, so pull first —\n",
"# the model owner may have uploaded since our last sync.\n",
"benchmark_owner.sync()\n",
"\n",
"model_dataset = benchmark_owner.datasets.get(\"dbe_adapter\", datasite=MODEL_OWNER_EMAIL)\n",
"# Wait until the model owner's card is here.\n",
"model_dataset = benchmark_owner.wait_until_has_dataset(\"dbe_adapter\", datasite=MODEL_OWNER_EMAIL)\n",
"print(model_dataset.mock_files[0].read_text())"
]
},
Expand Down Expand Up @@ -954,14 +948,11 @@
"metadata": {},
"outputs": [],
"source": [
"job = next((j for j in benchmark_owner.jobs if j.name == \"dbe_eval_job\"), None)\n",
"\n",
"if job is not None and not job.can_approve:\n",
" print(\" 🟠 Job 'dbe_eval_job' is here, but the enclave has not sent our approval file yet. Wait a moment and re-run this cell.\")\n",
"elif job is None:\n",
" print(\" 🟠 Job 'dbe_eval_job' not visible yet — the enclave is still distributing it. Wait a moment and re-run this cell.\")\n",
"else:\n",
" print(f\" ✅ Benchmark owner sees 'dbe_eval_job' status={job.status}\")"
"# Wait until the enclave has distributed the job and sent us our approval file.\n",
"job = benchmark_owner.wait_until_has_job(\n",
" \"dbe_eval_job\", where=lambda j: j.can_approve, timeout=900\n",
")\n",
"print(f\" ✅ Benchmark owner sees 'dbe_eval_job' status={job.status}\")"
]
},
{
Expand Down Expand Up @@ -998,7 +989,8 @@
"\n",
"Once the model owner approves, the enclave runs the evaluation and sends the output back to us. It\n",
"installs PyTorch, downloads the base model and generates on two CPUs, which took about three\n",
"minutes on the release this was tested against. Re-run until the status is `done`."
"minutes on the release this was tested against. The cell below waits until the status is `done`.\n",
"If the job fails, the cell stops with an error: see 'If the job failed' below."
]
},
{
Expand All @@ -1007,15 +999,16 @@
"metadata": {},
"outputs": [],
"source": [
"job = next(j for j in benchmark_owner.jobs if j.name == \"dbe_eval_job\")\n",
"print(f\" Job status : {job.status}\")\n",
"from syft_rds.waiting import JobEndedError\n",
"\n",
"if job.status == \"failed\":\n",
" print(\" ❌ The job failed inside the enclave. See 'If the job failed' below to read its logs.\")\n",
"elif job.status != \"done\" or not job.output_paths:\n",
" print(\" 🟠 Not finished yet — wait until the status is 'done', then re-run this cell.\")\n",
"else:\n",
" print(f\" ✅ Output files : {[p.name for p in job.output_paths]}\")"
"try:\n",
" job = benchmark_owner.wait_until_has_job(\n",
" \"dbe_eval_job\", status=\"done\", where=lambda j: bool(j.output_paths), timeout=900\n",
" )\n",
"except JobEndedError:\n",
" print(\" ❌ The job did not finish in the enclave. See 'If the job failed' below to read its logs.\")\n",
" raise\n",
"print(f\" ✅ Output files : {[p.name for p in job.output_paths]}\")"
]
},
{
Expand Down
42 changes: 18 additions & 24 deletions notebooks/enclave/double-blind-eval-w-receipts/2. DO-model-owner-dbe.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@
"source": [
"### Step 1.1 — Approve the benchmark owner\n",
"\n",
"Re-run the cell below until their request appears, then approve it. `peer_must_exist=False` lets us\n",
"approve a request that has arrived before our own view of them has caught up."
"The cell below shows our peers. `peer_must_exist=False` lets us approve the benchmark owner before\n",
"their request reaches us, and the cell after it waits until the connection is live."
]
},
{
Expand Down Expand Up @@ -176,11 +176,11 @@
"metadata": {},
"outputs": [],
"source": [
"# Fail on a wrong email, or if the peer has not approved yet. Warn if the peer\n",
"# was already there before this notebook logged in.\n",
"# attest_peer() below checks the enclave.\n",
"model_owner.validate_peer(BENCHMARK_OWNER_EMAIL)\n",
"print(f\" Connected to {BENCHMARK_OWNER_EMAIL}\")"
"# Wait up to 5 minutes for each peer to approve. Fail at once on a wrong email.\n",
"# Warn if the peer was already there before this notebook logged in.\n",
"for peer_email in [BENCHMARK_OWNER_EMAIL, ENCLAVE_EMAIL]:\n",
" model_owner.wait_until_peered(peer_email)\n",
" print(f\" Connected to {peer_email}\")"
]
},
{
Expand Down Expand Up @@ -216,14 +216,11 @@
" expected_syft_version=None,\n",
")\n",
"\n",
"result = model_owner.attest_peer(ENCLAVE_EMAIL, policy=policy)\n",
"\n",
"if result is None:\n",
" print(\" 🟠 The enclave published no attestation yet. Wait a moment and re-run this cell.\")\n",
"else:\n",
" result.print_checklist()\n",
" print()\n",
" print(f\" Bound to the report: {result.verified_key_bundle is not None}\")"
"# Wait until the enclave publishes its attestation, then verify it.\n",
"result = model_owner.wait_until_attested(ENCLAVE_EMAIL, policy=policy)\n",
"result.print_checklist()\n",
"print()\n",
"print(f\" Bound to the report: {result.verified_key_bundle is not None}\")"
]
},
{
Expand Down Expand Up @@ -362,7 +359,7 @@
"## Step 5 — Wait for the evaluation job\n",
"\n",
"The benchmark owner submits the job that runs our model on their prompts. The enclave distributes it\n",
"to both of us for approval. Re-run the cell below until it appears."
"to both of us for approval. The cell below waits until it is here and we can approve it."
]
},
{
Expand All @@ -371,14 +368,11 @@
"metadata": {},
"outputs": [],
"source": [
"job = next((j for j in model_owner.jobs if j.name == \"dbe_eval_job\"), None)\n",
"\n",
"if job is not None and not job.can_approve:\n",
" print(\" 🟠 Job 'dbe_eval_job' is here, but the enclave has not sent our approval file yet. Wait a moment and re-run this cell.\")\n",
"elif job is None:\n",
" print(\" 🟠 Job 'dbe_eval_job' not visible yet — the benchmark owner may not have submitted it. Wait a moment and re-run this cell.\")\n",
"else:\n",
" print(f\" ✅ Model owner sees 'dbe_eval_job' status={job.status}\")"
"# Wait until the enclave has distributed the job and sent us our approval file.\n",
"job = model_owner.wait_until_has_job(\n",
" \"dbe_eval_job\", where=lambda j: j.can_approve, timeout=900\n",
")\n",
"print(f\" ✅ Model owner sees 'dbe_eval_job' status={job.status}\")"
]
},
{
Expand Down
22 changes: 10 additions & 12 deletions notebooks/enclave/gemma/colab-3-persona/1. DO-model-owner-gemma-restrict.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@
"source": [
"### Step 1.1 — Wait for the Researcher peer request, then approve\n",
"\n",
"The Researcher notebook adds you as a peer. Re-run the cell below until you see their request appear, then approve."
"The Researcher notebook adds you as a peer. The cell below shows your peers. `peer_must_exist=False` lets you approve before their request arrives, and the next cell waits until the connection is live."
]
},
{
Expand Down Expand Up @@ -198,11 +198,11 @@
"metadata": {},
"outputs": [],
"source": [
"# Fail on a wrong email, or if the peer has not approved yet. Warn if the peer\n",
"# was already there before this notebook logged in.\n",
"# attest_peer() below checks the enclave.\n",
"model_owner.validate_peer(RESEARCHER_EMAIL)\n",
"print(f\" Connected to {RESEARCHER_EMAIL}\")"
"# Wait up to 5 minutes for each peer to approve. Fail at once on a wrong email.\n",
"# Warn if the peer was already there before this notebook logged in.\n",
"for peer_email in [RESEARCHER_EMAIL, ENCLAVE_EMAIL]:\n",
" model_owner.wait_until_peered(peer_email)\n",
" print(f\" Connected to {peer_email}\")"
]
},
{
Expand All @@ -222,8 +222,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Wait for enclave to accept peer request\n",
"model_owner.attest_peer(ENCLAVE_EMAIL)"
"# Wait until the enclave publishes its attestation, then verify it.\n",
"model_owner.wait_until_attested(ENCLAVE_EMAIL)"
]
},
{
Expand Down Expand Up @@ -492,8 +492,7 @@
"metadata": {},
"outputs": [],
"source": [
"model_owner.sync()\n",
"restrict_job = next(j for j in model_owner.jobs if j.name == RESTRICT_JOB_NAME)\n",
"restrict_job = model_owner.wait_until_has_job(RESTRICT_JOB_NAME, where=lambda j: j.can_approve)\n",
"print(f\" Model owner sees '{RESTRICT_JOB_NAME}' status={restrict_job.status}\")"
]
},
Expand Down Expand Up @@ -537,8 +536,7 @@
"outputs": [],
"source": [
"JOB_NAME = \"safety_eval_job\"\n",
"model_owner.sync()\n",
"model_owner_job = next(j for j in model_owner.jobs if j.name == JOB_NAME)\n",
"model_owner_job = model_owner.wait_until_has_job(JOB_NAME, where=lambda j: j.can_approve, timeout=900)\n",
"print(f\" Model owner sees '{JOB_NAME}' status={model_owner_job.status}\")"
]
},
Expand Down
27 changes: 13 additions & 14 deletions notebooks/enclave/gemma/colab-3-persona/2. DO-benchmark-owner-gemma-restrict.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@
"source": [
"### Step 1.1 — Wait for the Researcher peer request, then approve\n",
"\n",
"The Researcher notebook adds you as a peer. Re-run the cell below until you see their request appear, then approve."
"The Researcher notebook adds you as a peer. The cell below shows your peers. `peer_must_exist=False` lets you approve before their request arrives, and the next cell waits until the connection is live."
]
},
{
Expand Down Expand Up @@ -171,11 +171,11 @@
"metadata": {},
"outputs": [],
"source": [
"# Fail on a wrong email, or if the peer has not approved yet. Warn if the peer\n",
"# was already there before this notebook logged in.\n",
"# attest_peer() below checks the enclave.\n",
"benchmark_owner.validate_peer(RESEARCHER_EMAIL)\n",
"print(f\" Connected to {RESEARCHER_EMAIL}\")"
"# Wait up to 5 minutes for each peer to approve. Fail at once on a wrong email.\n",
"# Warn if the peer was already there before this notebook logged in.\n",
"for peer_email in [RESEARCHER_EMAIL, ENCLAVE_EMAIL]:\n",
" benchmark_owner.wait_until_peered(peer_email)\n",
" print(f\" Connected to {peer_email}\")"
]
},
{
Expand All @@ -195,8 +195,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Wait for enclave to accept peer request\n",
"benchmark_owner.attest_peer(ENCLAVE_EMAIL)"
"# Wait until the enclave publishes its attestation, then verify it.\n",
"benchmark_owner.wait_until_attested(ENCLAVE_EMAIL)"
]
},
{
Expand Down Expand Up @@ -310,8 +310,7 @@
"outputs": [],
"source": [
"RESTRICT_JOB_NAME = \"restrict_engine_review\"\n",
"benchmark_owner.sync()\n",
"restrict_job = next(j for j in benchmark_owner.jobs if j.name == RESTRICT_JOB_NAME)\n",
"restrict_job = benchmark_owner.wait_until_has_job(RESTRICT_JOB_NAME, where=lambda j: j.can_approve)\n",
"print(f\" Benchmark owner sees '{RESTRICT_JOB_NAME}' status={restrict_job.status}\")"
]
},
Expand Down Expand Up @@ -354,8 +353,9 @@
"metadata": {},
"outputs": [],
"source": [
"benchmark_owner.sync()\n",
"restrict_job = next(j for j in benchmark_owner.jobs if j.name == RESTRICT_JOB_NAME)\n",
"restrict_job = benchmark_owner.wait_until_has_job(\n",
" RESTRICT_JOB_NAME, status=\"done\", where=lambda j: bool(j.output_paths), timeout=900\n",
")\n",
"outputs = {p.name: p for p in restrict_job.output_paths}\n",
"print(f\" Output files : {list(outputs)}\")\n",
"\n",
Expand Down Expand Up @@ -383,8 +383,7 @@
"outputs": [],
"source": [
"JOB_NAME = \"safety_eval_job\"\n",
"benchmark_owner.sync()\n",
"benchmark_owner_job = next(j for j in benchmark_owner.jobs if j.name == JOB_NAME)\n",
"benchmark_owner_job = benchmark_owner.wait_until_has_job(JOB_NAME, where=lambda j: j.can_approve, timeout=900)\n",
"print(f\" Benchmark owner sees '{JOB_NAME}' status={benchmark_owner_job.status}\")"
]
},
Expand Down
30 changes: 14 additions & 16 deletions notebooks/enclave/gemma/colab-3-persona/3. DS-researcher-gemma-restrict.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
"source": [
"### Step 1.1 — Wait until both Data Owners approve, then verify peers\n",
"\n",
"Each DO needs to run their `approve_peer_request` cell. Re-run the cell below until both appear as approved peers."
"Each DO needs to run their `approve_peer_request` cell. The cell below shows your peers, and the next cell waits until both have approved."
]
},
{
Expand All @@ -158,11 +158,10 @@
"metadata": {},
"outputs": [],
"source": [
"# Fail on a wrong email, or if the peer has not approved yet. Warn if the peer\n",
"# was already there before this notebook logged in.\n",
"# attest_peer() below checks the enclave.\n",
"for peer_email in [MODEL_OWNER_EMAIL, BENCHMARK_OWNER_EMAIL]:\n",
" researcher.validate_peer(peer_email)\n",
"# Wait up to 5 minutes for each peer to approve. Fail at once on a wrong email.\n",
"# Warn if the peer was already there before this notebook logged in.\n",
"for peer_email in [MODEL_OWNER_EMAIL, BENCHMARK_OWNER_EMAIL, ENCLAVE_EMAIL]:\n",
" researcher.wait_until_peered(peer_email)\n",
" print(f\" Connected to {peer_email}\")"
]
},
Expand All @@ -181,8 +180,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Wait for enclave to accept peer request\n",
"researcher.attest_peer(ENCLAVE_EMAIL)"
"# Wait until the enclave publishes its attestation, then verify it.\n",
"researcher.wait_until_attested(ENCLAVE_EMAIL)"
]
},
{
Expand All @@ -200,7 +199,8 @@
"metadata": {},
"outputs": [],
"source": [
"researcher.sync()\n",
"researcher.wait_until_has_dataset(\"gemma3_model\", datasite=MODEL_OWNER_EMAIL)\n",
"researcher.wait_until_has_dataset(\"safety_prompts\", datasite=BENCHMARK_OWNER_EMAIL)\n",
"researcher.datasets"
]
},
Expand Down Expand Up @@ -341,7 +341,7 @@
"source": [
"## Step 6 — Wait for both DOs to approve and the enclave to run\n",
"\n",
"Each DO must approve from their notebook; once both approve, the enclave runs the job and pushes results back. Re-sync until status is `\"done\"`."
"Each DO must approve from their notebook; once both approve, the enclave runs the job and pushes results back. The cell below waits until the status is `\"done\"`."
]
},
{
Expand All @@ -351,13 +351,11 @@
"metadata": {},
"outputs": [],
"source": [
"researcher.sync()\n",
"researcher_job = next(j for j in researcher.jobs if j.name == JOB_NAME)\n",
"researcher_job = researcher.wait_until_has_job(\n",
" JOB_NAME, status=\"done\", where=lambda j: bool(j.output_paths), timeout=900\n",
")\n",
"print(f\" Researcher job status : {researcher_job.status}\")\n",
"print(f\" Output files : {researcher_job.output_paths}\")\n",
"\n",
"assert researcher_job.status == \"done\", researcher_job.status\n",
"assert len(researcher_job.output_paths) > 0"
"print(f\" Output files : {researcher_job.output_paths}\")"
]
},
{
Expand Down
Loading
Loading