feat(mcp): return from index_codebase without waiting for the first index

README.md

@@ -130,6 +130,12 @@ most 5 seconds for a first index before it reports that the index is still
 running. The edit actions have no deadline: a limit that fired after the files
 changed would report a failed edit that was applied.
 
+The `index_codebase` tool does not wait for the first index. It registers the
+codebase, starts the scan and upload in the background, and returns at once with
+the codebase id and the path. The `sync_status` tool then reports the first-index
+phase: registering, syncing, embedding, ready, or failed. `semctl daemon status`
+reports the same phase for each checkout.
+
 ### Coming from the old `semctx` CLI
 
 The first `semctl install` automatically retires a previous `semctx` install: it
@@ -175,7 +181,7 @@ A daemon started by hand skips the job check, and clients attach to it.
 Two commands inspect and end a running daemon:
 
 ```sh
-semctl daemon status        # version, pid, uptime, sessions, permits, checkouts
+semctl daemon status        # version, pid, uptime, sessions, permits, checkouts and their first-index phase
 semctl daemon status --json # the same facts as one JSON object
 semctl daemon stop          # end every session and exit
 ```

plugins/semctx/skills/codebase-retrieval/SKILL.md

@@ -45,11 +45,14 @@ for the rest of the MCP session. A previously indexed repo already has consent
 for retrieval and watcher activation, so use it without asking again; that consent
 does not authorize edits outside the user's requested repository set. Only when a
 repo has never been indexed should you tell the user and call `index_codebase`
-after explicit opt-in. That first-ever index is gated:
-retrieval/catalog/graph tools never serve a partial index. They wait at most 5
-seconds, then fail with a "still running" error. Use `sync_status` to follow
-progress and local Read/Grep until the index completes. Later re-syncs do not
-block use of the last complete index.
+after explicit opt-in. That first-ever index runs in the background:
+`index_codebase` returns at once and never waits for embedding. Retrieval,
+catalog, and graph tools never serve a partial index. They wait at most 5
+seconds, then fail with a "still running" error. Call `sync_status` every 10 to
+15 seconds to follow the first-index phase (registering, syncing, embedding,
+ready, or failed), and use local Read/Grep meanwhile. If the phase is failed,
+call `index_codebase` for the same path to retry. Later re-syncs do not block
+use of the last complete index.
 
 ## Rules
 
@@ -85,9 +88,13 @@ block use of the last complete index.
 
 ## Degraded states — read the signals, don't guess
 
-- Empty results right after attach → `sync_status`: `queued` / `running`
-  means the index is still building; retry shortly instead of concluding the
-  repo isn't indexed.
+- Empty results right after attach → `sync_status`: a first-index phase of
+  registering, syncing, or embedding, or a job that is `queued` / `running`,
+  means the index is still building; check again in 10 to 15 seconds instead of
+  concluding the repo isn't indexed.
+- A tool fails → the error ends with a `next:` line. Follow it. Retry once only
+  when it says the call is retryable. Otherwise use local Read/Grep for that
+  request. Do not repeat the same failing call in a loop.
 - A hit flagged `⚠ stale (edited since indexed)`, or a freshness footer →
   `Read` that file for current bytes. The MCP server auto-syncs edits
   (filesystem watcher + periodic re-sync); do not tell the user to run

plugins/semctx/skills/codebase-retrieval/evals.json

@@ -37,7 +37,7 @@
       "expected_behavior": [
         "Uses search_codebase (prefer: docs) or semctx grep — does not skip semctx on the false premise that markdown is not indexed",
         "Falls back to host tools only on a reported degraded state",
-        "Identifies that retrieval/catalog/graph tools wait for the initial embed job while sync_status remains available"
+        "Identifies that index_codebase returns without waiting for embedding, that retrieval/catalog/graph tools wait at most 5 seconds and then report a still-running error, and that sync_status remains available to follow the first-index phase"
       ]
     },
     {
@@ -62,8 +62,9 @@
       "skills": ["codebase-retrieval"],
       "query": "I just opted a brand-new repository into indexing. Can I search it while its first embed job is still running?",
       "expected_behavior": [
-        "Explains that first-time retrieval/catalog/graph calls wait for successful embedding completion and cannot see a partial initial index",
-        "Uses sync_status to monitor first-index progress because status bypasses the readiness gate",
+        "Explains that index_codebase returned at once, and that first-time retrieval/catalog/graph calls cannot see a partial initial index: they wait at most 5 seconds, then fail with a still-running error",
+        "Calls sync_status about every 10 to 15 seconds to follow the first-index phase (registering, syncing, embedding, ready, or failed) because status bypasses the readiness gate",
+        "Uses local Read/Grep for the repository until the phase is ready instead of looping on a retrieval call that reports it is still running",
         "Distinguishes later background re-syncs, which do not block retrieval from the last complete snapshot"
       ]
     },
@@ -108,6 +109,15 @@
         "Uses an immutable codebase id or indexed local directory path for another checkout",
         "Does not present or reuse a friendly slug as a selector"
       ]
+    },
+    {
+      "skills": ["codebase-retrieval"],
+      "query": "A semctx retrieval tool returned an error that ends with the line 'next: use local Read/Grep for this request.' What do you do next?",
+      "expected_behavior": [
+        "Follows the next: line and answers this request with local Read/Grep instead of calling the same semctx tool again",
+        "Does not loop on the failing call, and retries once only when the next: line says the call is retryable",
+        "Does not claim the repository is unindexed or tell the user to reindex because of one failed call"
+      ]
     }
   ],
   "triggering": {

src/daemon/status.rs

@@ -102,6 +102,7 @@ mod tests {
     use crate::engine::CoordinatorStatus;
     use crate::engine::coordinator::WatcherState;
     use crate::engine::scheduler::{PermitUsage, SchedulerUsage};
+    use crate::mcp::readiness::FirstIndexPhase;
 
     fn status() -> DaemonStatus {
         DaemonStatus {
@@ -134,6 +135,9 @@ mod tests {
                 trigger_overflow: false,
                 last_outcome: Some("uploaded 3 files".to_string()),
                 last_error: None,
+                first_index: Some(FirstIndexPhase::Embedding {
+                    job_id: Some("job-1".to_string()),
+                }),
             }],
         }
     }
@@ -164,6 +168,12 @@ mod tests {
             Some("uploaded 3 files")
         );
         assert_eq!(coordinator.last_error, None);
+        assert_eq!(
+            coordinator.first_index,
+            Some(FirstIndexPhase::Embedding {
+                job_id: Some("job-1".to_string())
+            })
+        );
     }
 
     /// A reader that is not this build must find the documented field names.
@@ -184,6 +194,10 @@ mod tests {
         assert_eq!(value["permits"]["scan"]["total"], 8);
         assert_eq!(value["coordinators"][0]["leases"], 2);
         assert_eq!(value["coordinators"][0]["watcher"], "active");
+        assert_eq!(
+            value["coordinators"][0]["first_index"],
+            serde_json::json!({"embedding": {"job_id": "job-1"}})
+        );
     }
 
     #[test]
@@ -205,6 +219,10 @@ mod tests {
             ]
         );
         assert!(lines[8].starts_with("  root /work/checkout"), "{rendered}");
+        assert!(
+            lines[8].contains("first index embedding (job job-1)"),
+            "{rendered}"
+        );
     }
 
     #[test]

src/engine/coordinator.rs

@@ -48,7 +48,7 @@ use super::registry::CheckoutKey;
 use super::scheduler;
 use super::watch_hub::{WatchBatch, WatchRegistration, can_change_tree};
 use crate::client::{self, Client};
-use crate::mcp::readiness::InitialIndexGate;
+use crate::mcp::readiness::{FirstIndexPhase, InitialIndexGate};
 use crate::sync::policy::{SourcePolicy, event_may_affect_policy};
 use crate::sync::{self, LastJob, SyncCache, SyncLimits, SyncOutcome, blocking};
 
@@ -199,13 +199,17 @@ pub(crate) struct CoordinatorStatus {
     pub(crate) trigger_overflow: bool,
     pub(crate) last_outcome: Option<String>,
     pub(crate) last_error: Option<String>,
+    /// The phase of this checkout's first index. `None` when the checkout has
+    /// no first-index gate, and for a daemon that predates the field.
+    #[serde(default)]
+    pub(crate) first_index: Option<FirstIndexPhase>,
 }
 
 impl std::fmt::Display for CoordinatorStatus {
     fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         write!(
             formatter,
-            "root {} codebase {} leases {} watcher {} job {} running {} pending {}{} outcome {} error {}",
+            "root {} codebase {} leases {} watcher {} job {} running {} first index {} pending {}{} outcome {} error {}",
             self.root.display(),
             self.codebase_id.as_deref().unwrap_or("(unbound)"),
             self.leases,
@@ -215,6 +219,9 @@ impl std::fmt::Display for CoordinatorStatus {
             },
             self.last_job_id.as_deref().unwrap_or("(none)"),
             self.running,
+            self.first_index
+                .as_ref()
+                .map_or_else(|| "(none)".to_string(), ToString::to_string),
             self.pending_triggers,
             if self.trigger_overflow {
                 " (overflowed)"
@@ -394,6 +401,11 @@ impl CheckoutCoordinator {
             .await
             .as_ref()
             .map(|job| job.job_id.clone());
+        let first_index = if let Some(gate) = self.gate().await {
+            Some(gate.phase(last_job_id.as_deref()).await)
+        } else {
+            None
+        };
         let run = self.run.lock().await;
         CoordinatorStatus {
             root: self.root.clone(),
@@ -402,6 +414,7 @@ impl CheckoutCoordinator {
             watcher: self.watcher_state(),
             last_job_id,
             running: run.running,
+            first_index,
             pending_triggers: self.triggers.max_capacity() - self.triggers.capacity(),
             trigger_overflow: self.overflow.load(Ordering::Acquire),
             last_outcome: run.last_outcome.clone(),

src/engine/coordinator/tests.rs

@@ -18,7 +18,7 @@ use super::{
 };
 use crate::client::{Client, api};
 use crate::engine::registry::CheckoutKey;
-use crate::mcp::readiness::InitialIndexGate;
+use crate::mcp::readiness::{FirstIndexPhase, InitialIndexGate};
 use crate::sync::policy;
 
 /// Somewhere no test writes, so a coordinator that starts reading the
@@ -289,6 +289,82 @@ async fn status_reports_the_last_job_and_the_watcher_state() {
     assert!(status.to_string().contains("job job"));
 }
 
+/// Poll the coordinator's status until its first index reaches `expected`.
+async fn wait_for_phase(coordinator: &CheckoutCoordinator, expected: FirstIndexPhase) {
+    tokio::time::timeout(Duration::from_secs(5), async {
+        while coordinator.status().await.first_index.as_ref() != Some(&expected) {
+            tokio::task::yield_now().await;
+        }
+    })
+    .await
+    .unwrap_or_else(|_| panic!("the first index must reach {expected}"));
+}
+
+/// Nothing waits for a first index any more, so status is the only way to
+/// follow it. Each phase comes from the gate and from the last recorded job.
+#[tokio::test]
+async fn status_follows_a_first_index_through_its_phases() {
+    let (reconciler, mut runs) = CountingReconciler::new();
+    let gate = Arc::new(InitialIndexGate::pending());
+    let (coordinator, task) = build(reconciler, Some(gate.clone()), Some(0)).await;
+    assert_eq!(
+        coordinator.status().await.first_index,
+        Some(FirstIndexPhase::Registering)
+    );
+    task.spawn();
+    coordinator.trigger(Trigger::Startup);
+
+    gate.register_codebase("codebase".to_string()).await;
+    runs.next_start().await;
+    wait_for_phase(&coordinator, FirstIndexPhase::Syncing).await;
+
+    runs.release(1);
+    let embedding = FirstIndexPhase::Embedding {
+        job_id: Some("job".to_string()),
+    };
+    wait_for_phase(&coordinator, embedding.clone()).await;
+    assert!(
+        coordinator
+            .status()
+            .await
+            .to_string()
+            .contains("first index embedding (job job)")
+    );
+
+    runs.release_polls(1);
+    wait_for_phase(&coordinator, FirstIndexPhase::Ready).await;
+}
+
+/// A checkout that no `index_codebase` call claimed has no first index to
+/// report.
+#[tokio::test]
+async fn status_has_no_first_index_phase_without_a_gate() {
+    let (reconciler, _runs) = CountingReconciler::new();
+    let (coordinator, _task) = coordinator(reconciler).await;
+
+    let status = coordinator.status().await;
+
+    assert_eq!(status.first_index, None);
+    assert!(status.to_string().contains("first index (none)"));
+}
+
+/// The daemon status crosses the process boundary. A daemon that predates the
+/// first-index field sends none, and the reader must still decode its status.
+#[test]
+fn a_status_without_a_first_index_field_decodes_as_none() {
+    let old = serde_json::json!({
+        "root": "/work/checkout", "codebase_id": "codebase", "leases": 1,
+        "watcher": "active", "last_job_id": null, "running": false,
+        "pending_triggers": 0, "trigger_overflow": false,
+        "last_outcome": null, "last_error": null,
+    });
+
+    let status: super::CoordinatorStatus =
+        serde_json::from_value(old).expect("decode a status from an older daemon");
+
+    assert_eq!(status.first_index, None);
+}
+
 /// A second reconcile during embedding sees the same pending gate. It must not
 /// start a second poll: the poll that finished last would otherwise decide the
 /// result for every session, and a succeeded first index could turn failed.

src/mcp/docs/instructions/server.md

@@ -25,8 +25,8 @@ Code retrieval and graph navigation over semctx-indexed codebases. Tools default
 - **external_links** — cross-repo "jump to definition": imports that leave this codebase, resolved into the public API of other codebases you can see.
 - **list_codebases / current_context** — discover visible codebases and diagnose the effective server/tenant/codebase/checkout/index/graph binding.
 - **list_domains** — discover what's searchable and which tags are available for filtering.
-- **sync_status** — check the background index job; if `list_files` / `search` are empty right after attaching, this tells you whether indexing is still in progress.
-- **index_codebase** — explicit opt-in indexing. If startup or a tool says the current repository is not indexed, tell the user and ask permission; call this only after they agree.
+- **sync_status** — check the background index job and the first-index phase; if `list_files` / `search` are empty right after attaching, this tells you whether indexing is still in progress.
+- **index_codebase** — explicit opt-in indexing. If startup or a tool says the current repository is not indexed, tell the user and ask permission; call this only after they agree. It returns as soon as the first index starts; follow the first index with `sync_status`.
 - **rename_symbol / safe_delete_symbol / replace_symbol_body / insert_before_symbol / insert_after_symbol** — immediate local edit actions. Each internally consumes the server's grammar-validated plan, verifies source identity, graph generation, paths, and hashes, and goes through the client's normal approval flow.
 - **undo_edit** — reverse a completed symbolic edit by its returned edit id while retained preimages and current postimage hashes still match.
 
@@ -38,12 +38,15 @@ without asking again; that consent does not authorize edits beyond the user's
 requested repository set. Ask only when a directory has never been indexed and
 would need `index_codebase`.
 
-The first-ever `index_codebase` call is a readiness boundary: retrieval, catalog,
-and graph tools never serve a partial first index. They wait at most 5 seconds
-for it to complete. If it still runs, they fail with a "still running" error:
-call `sync_status` to follow progress, and use local Read/Grep until it
-completes. `sync_status` stays callable. Later re-syncs do not block retrieval
-and continue to expose the last complete snapshot with freshness warnings.
+The first-ever `index_codebase` call starts the first index and returns at once.
+It never waits for embedding. Retrieval, catalog, and graph tools never serve a
+partial first index. They wait at most 5 seconds for it to complete. If it still
+runs, they fail with a "still running" error. `sync_status` stays callable. It
+reports the first-index phase: registering, syncing, embedding, ready, or failed.
+Call it every 10 to 15 seconds, and use local Read/Grep until the phase is ready.
+If the phase is failed, call `index_codebase` for the same path to retry. Later
+re-syncs do not block retrieval and continue to expose the last complete
+snapshot with freshness warnings.
 
 ## Workflow
 
@@ -68,3 +71,9 @@ Prefer the symbol-graph tools over search whenever you have an exact name — th
 Expanded search bodies share the server's total result-content budget. `top_k` is
 an upper bound, not a guarantee that every full body fits. Use expanded bodies
 directly instead of fetching the same files again.
+
+## When a tool fails
+
+A tool failure arrives as an error result with a `next:` line. Follow that line.
+Retry once only when it says the call is retryable. Otherwise use local
+Read/Grep for that request. Do not repeat the same failing call in a loop.

src/mcp/docs/tools/index_codebase.md

@@ -2,6 +2,10 @@ Explicitly opt a local directory into semctl indexing and keep it watched for th
 
 Call this only after the user has agreed to index a genuinely unindexed directory. Never infer first-time consent from a search request, an unindexed-repository notice, or a failed code tool. A repository that resolves to an existing index already has prior consent and should be synced/watched automatically through normal codebase-scoped tools without asking again. Omit `path` to index the MCP launch directory, or pass an absolute directory path.
 
-For a first-ever index, the tool registers the codebase and starts the scan/upload in a detached task, then waits for the server to finish embedding every file. All codebase-scoped retrieval/catalog/graph tools targeting that repository check the same readiness gate, so they cannot observe a partial first index: each waits at most 5 seconds, then fails with a "still running" error. `sync_status` deliberately remains available for progress checks. If the initiating tool call times out, is cut off at the call deadline, or is cancelled, indexing and the readiness gate continue in the MCP server.
+For a first-ever index, the tool registers the codebase, starts the scan and upload in a background task, and returns at once. It does not wait for the server to embed the files. The answer names the codebase id and the path. Call `sync_status` every 10 to 15 seconds to follow the first index, and use local Read/Grep until it completes. If the call times out or is cancelled, the first index continues in the MCP server.
+
+All codebase-scoped retrieval/catalog/graph tools targeting that repository check the same readiness gate, so they cannot observe a partial first index: each waits at most 5 seconds, then fails with a "still running" error. `sync_status` deliberately remains available and reports the first-index phase: registering, syncing, embedding, ready, or failed.
+
+A second call for the same path while the first index runs does not start another one. It answers "already in progress" with the same instruction. After the first index completes, a call answers that it is complete. If the first index failed, call this tool again with the same path to retry it.
 
 For a previously indexed repository, this starts the normal background re-sync/watcher immediately without a first-index gate.