Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/web/PRODUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The console runs beside the administrator's own Core, with execution, files and
- **Monitor**: Overview (service status, running Sessions, sandbox slots, Sessions needing attention, 24-hour Session activity, a compact inventory of Core and up to four nodes, prioritizing offline and degraded nodes when the list is full, with a popover glance at each, the attention table, usage by project), Core metrics (the Core process's CPU and memory, execution slots and the Turn queue, connected daemons, the database and background jobs), Agent metrics (requests, errors, duration, tokens, models, tools, Agents and API keys for 1 h / 6 h / 24 h / 7 d), Sandbox metrics (node capacity and hosted Runtimes across projects; a node or a sandbox opens in a dialog with its figures and CPU and memory charts), Session log (every Session, read-only, with a failed Session's reason under its status, opening one Session's history, which jumps to its failed Turns; a self-hosted Session's page also has its environment's executor credentials). Agent metrics' By Agent table opens an Agent's page and, from its failed Turns, its Sessions in the Session log.
- **Resources**: Agents, Environment templates, Skills, Files, Vaults. Each list shows one project or all projects, with a Project column when all are shown and a Creator column naming the creating key. Detail pages show the resource's facts and offer Delete.
- **Platform**: Projects and keys (projects, their assets and usage, named keys, write history), Nodes (the node list, capacity, host figures, allocations and individual node operations). Add node asks for limits before issuing its one-time command; installers use Core's public URL and require supported node artifacts. Removal offers the host's uninstall command. System owns installation facts, the Domain and HTTPS secondary page, each harness's default model configuration, startup settings, and a link to the Sandbox configuration secondary page. That page owns setup, resource edits, rollout details and reset. Setup selects a backend, size and Runtime, then asks for a deliberate save; own-machine setup continues to Add node.
- A node whose provider is not ready names the reason (Docker unreachable, no Docker limits, missing Runtime image, no KVM, missing microsandbox components, a host too small) and its fix in the help tip beside its status, wherever that status shows.
- A node whose provider is not ready names the reason as one Provider-neutral readiness class (provider unavailable, host unsupported, provider files or Runtime image missing, Runtime download failed, a host too small) and its fix in the help tip beside its status, wherever that status shows.
- A node enrolled with an earlier Core address gets no new sandboxes, so on the Nodes list and its page its status is Old address, with "Remove and add again", never Available.
- **Sandbox reset** is an explicit administrator operation in System → Sandbox configuration. Auto clear is the default, with a one-hour deadline (5 minutes–24 hours); Force clear requires destructive confirmation. Reset stops new hosted Session admission, clears idle, suspended and pending hosted work, and waits for busy Turns and file writes until Core forces the remaining work. It does not affect self-hosted execution. Histories and persisted Files/Artifacts remain; archived Sessions cannot resume, and unpersisted workspace contents may be lost. Cancel stops further clearing without undoing archives. Core alone reports progress and completion, including resources blocked on named offline nodes; force does not bypass their cleanup. Completion clears the backend configuration and retires old nodes/enrollment credentials. A new configuration is then a separate deliberate save.
- **Online sandbox configuration** changes the same backend's resources, Runtime or E2B template without retiring existing nodes or changing existing Sessions' resource ownership. New placement follows Core's qualified capacity; saving a target does not promise immediate placement on it. Configuration rollout shows Core's target preparation and retained previous-generation sandbox count. A settled rollout can still have failed, update-required or unknown nodes and old resources. An offline node stays offline even when it has a recorded serving generation. Node and allocation detail distinguish the serving pin, target preparation and each resource's configuration generation.
Expand Down
2 changes: 1 addition & 1 deletion apps/web/e2e/data/routes.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ export function buildDemo(now = Math.floor(Date.now() / 1000), publicUrl = "http
sessions.sort((a, b) => b.created_at - a.created_at);
const nodes = [
{ rollout: { state: "ready", ready_generation: 1 }, id: "node-local", name: "core-01", provider: "docker", online: true, provider_ready: true, cpu_count: 16, available_memory_bytes: 38 * 2 ** 30, available_disk_bytes: 410 * 2 ** 30, running: 5, snapshots: 2, last_seen_at: new Date((now - 8) * 1000).toISOString(), max_active: 8, max_retained: 16, active: 5, reserved: 1, retained: 2, cleanup_pending: 0, created_at: new Date((now - 86400 * 30) * 1000).toISOString() },
{ rollout: { state: "failed", ready_generation: null, diagnostic: "docker_limits_unsupported" }, id: "node-gpu", name: "gpu-worker-02", provider: "docker", online: true, provider_ready: false, diagnostic: "docker_limits_unsupported", cpu_count: 32, available_memory_bytes: 12 * 2 ** 30, available_disk_bytes: 96 * 2 ** 30, running: 7, snapshots: 5, last_seen_at: new Date((now - 12) * 1000).toISOString(), max_active: 8, max_retained: 16, active: 7, reserved: 0, retained: 5, cleanup_pending: 1, created_at: new Date((now - 86400 * 12) * 1000).toISOString() },
{ rollout: { state: "failed", ready_generation: null, diagnostic: "host_unsupported" }, id: "node-gpu", name: "gpu-worker-02", provider: "docker", online: true, provider_ready: false, diagnostic: "host_unsupported", cpu_count: 32, available_memory_bytes: 12 * 2 ** 30, available_disk_bytes: 96 * 2 ** 30, running: 7, snapshots: 5, last_seen_at: new Date((now - 12) * 1000).toISOString(), max_active: 8, max_retained: 16, active: 7, reserved: 0, retained: 5, cleanup_pending: 1, created_at: new Date((now - 86400 * 12) * 1000).toISOString() },
{ rollout: { state: "unknown", ready_generation: 1 }, id: "node-edge", name: "edge-03", provider: "docker", online: false, provider_ready: false, cpu_count: 8, available_memory_bytes: null, available_disk_bytes: null, running: 0, snapshots: 0, last_seen_at: new Date((now - 5400) * 1000).toISOString(), max_active: 4, max_retained: 8, active: 0, reserved: 0, retained: 0, cleanup_pending: 0, created_at: new Date((now - 86400 * 3) * 1000).toISOString() },
];
const hosted = sessions.filter((session) => session.environment.type === "openai_hosted");
Expand Down
2 changes: 1 addition & 1 deletion apps/web/e2e/monitoring.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ test("shows the deployment's health on Overview and each monitor page", async ({
await page.getByRole("button", { name: "Sandbox metrics" }).click();
await expect(page.getByRole("table").first()).toContainText("core-01");
// A degraded node names why its provider is not ready.
await expect(page.locator(".status-with-help").filter({ hasText: /^Provider not ready/ }).getByRole("button", { name: "Docker limits unsupported", exact: true })).toBeVisible();
await expect(page.locator(".status-with-help").filter({ hasText: /^Provider not ready/ }).getByRole("button", { name: "Host unsupported", exact: true })).toBeVisible();
});

test("opens a Session's conversation from the Session log, read-only", async ({ page, request }) => {
Expand Down
6 changes: 3 additions & 3 deletions apps/web/e2e/nodes.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,9 @@ test("adds a node: host requirements, a root/sudo command, a countdown, the same
await expect(problem).toContainText("Not connected yet");
await expect(problem).toContainText("sudo journalctl -u oac-node-7f3c2a90-5b1e-4c2d-9e3f-0a1b2c3d4e5f.service");
await expect(add.getByText("No sudo on this host?")).toHaveCount(0);
// Connected, it reports why Docker isn't ready; once ready, the node is connected.
await setNode(request, { id: "node-new", online: true, diagnostic: "docker_limits_unsupported" });
await expect(problem).toContainText("Docker limits unsupported");
// Connected, it reports why its provider isn't ready; once ready, the node is connected.
await setNode(request, { id: "node-new", online: true, diagnostic: "host_unsupported" });
await expect(problem).toContainText("Host unsupported");
await expect(progress).toContainText("Waiting for Docker");
await setNode(request, { id: "node-new", provider_ready: true, diagnostic: "" });
await expect(progress).toHaveText("edge-04 · Connected");
Expand Down
2 changes: 1 addition & 1 deletion apps/web/src/features/sandbox/node-enrollment.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ describe("node enrollment", () => {
const connected = { ...fresh, online: true };
expect(enrollmentProgress(connected, 0, 1_000)).toEqual({ stage: "connected", problem: "" });
expect(enrollmentProgress(connected, 0, NODE_READY_WAIT_MS)).toEqual({ stage: "connected", problem: "provider_unavailable" });
expect(enrollmentProgress({ ...connected, diagnostic: "kvm_unavailable" }, 0, 1_000)).toEqual({ stage: "connected", problem: "kvm_unavailable" });
expect(enrollmentProgress({ ...connected, diagnostic: "host_unsupported" }, 0, 1_000)).toEqual({ stage: "connected", problem: "host_unsupported" });
expect(enrollmentProgress({ ...connected, provider_ready: true }, 0, NODE_READY_WAIT_MS * 2)).toEqual({ stage: "ready", problem: "" });
});

Expand Down
14 changes: 5 additions & 9 deletions apps/web/src/lib/locale-strings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -285,21 +285,17 @@ export const chinese = {
"Sandbox ownership mismatch": "沙箱归属不一致",
"Reconcile the assigned resource and its ownership record before resuming execution.": "请核对已分配资源及其归属记录,再恢复执行。",
"Sandbox provider unavailable": "沙箱后端不可用",
"Restore the provider on the assigned node, then refresh. A connected node alone does not confirm that its sandbox provider is ready.": "请恢复已分配节点上的运行后端,然后刷新。节点在线并不代表其沙箱后端已就绪。",
"Restore the provider on the assigned node, then refresh. Running the install command again on the host checks its requirements and names the fix.": "请恢复已分配节点上的运行后端,然后刷新。在该主机上重新运行安装命令,会检查主机要求并指出修复方法。",
"Sandbox state needs attention": "沙箱状态需要检查",
"Inspect the assigned node and resource, then refresh.": "请检查已分配节点和资源,然后刷新。",
"Docker unavailable": "Docker 不可用",
"The node can't reach the Docker daemon. Check that Docker is running and the node can use its socket.": "节点连不上 Docker 守护进程。请确认 Docker 正在运行,且节点能访问它的 socket。",
"Docker limits unsupported": "Docker 无法限制资源",
"Docker on this host doesn't enforce CPU and memory limits. Enable cgroup limits.": "这台主机上的 Docker 不能限制 CPU 和内存。请启用 cgroup 限制。",
"Host unsupported": "主机不满足要求",
"The host lacks a capability its sandbox provider requires. Running the install command again on the host checks its requirements and names the fix.": "主机缺少沙箱后端所需的能力。在该主机上重新运行安装命令,会检查主机要求并指出修复方法。",
"Provider files missing": "后端文件缺失",
"Pinned provider files are missing or fail their checksum. Run the install command again.": "指定版本的后端文件缺失或校验不通过。请重新运行安装命令。",
"Runtime download failed": "Runtime 下载失败",
"Runtime files could not be downloaded or verified. Check the node's network access and the configured Runtime release.": "Runtime 文件下载或验证失败。请检查节点网络连接及配置的 Runtime 发布版本。",
"Runtime image missing": "缺少 Runtime 镜像",
"The pinned Runtime image isn't on the host. Run the install command again.": "主机上没有指定版本的 Runtime 镜像。请重新运行安装命令。",
"KVM unavailable": "KVM 不可用",
"/dev/kvm isn't available to the node. Enable virtualization or use a KVM-capable host.": "节点无法使用 /dev/kvm。请开启虚拟化,或换一台支持 KVM 的主机。",
"microsandbox components missing": "microsandbox 组件缺失",
"microsandbox components are missing or fail their checksum. Run the install command again.": "microsandbox 组件缺失或校验不通过。请重新运行安装命令。",
"Host too small": "主机资源不足",
"The host has less CPU or memory than one sandbox needs. Use a bigger host or a smaller sandbox size.": "主机的 CPU 或内存不够运行一个沙箱。请换一台更大的主机,或调小沙箱规格。",
"Connecting to this console's Core…": "正在连接此控制台的 Core…",
Expand Down
4 changes: 2 additions & 2 deletions apps/web/src/lib/locale.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ describe("sandbox localization", () => {
expect(sandboxStateLabel("internal-value", "zh")).toBe("未知状态");
});
it("localizes all diagnostic labels and advice", () => {
for (const code of ["node_unavailable", "resource_missing", "compute_unconfirmed", "ownership_mismatch", "provider_unavailable", "docker_unavailable",
"docker_limits_unsupported", "runtime_download_failed", "runtime_image_unavailable", "kvm_unavailable", "microsandbox_artifacts_unavailable", "capacity_insufficient", "unknown"]) {
for (const code of ["node_unavailable", "resource_missing", "compute_unconfirmed", "ownership_mismatch", "provider_unavailable", "host_unsupported",
"artifacts_unavailable", "runtime_download_failed", "runtime_image_unavailable", "capacity_insufficient", "unknown"]) {
const message = sandboxDiagnosticMessage(code, "zh");
expect(message?.label).toMatch(/[\u4e00-\u9fff]/);
expect(message?.advice).toMatch(/[\u4e00-\u9fff]/);
Expand Down
4 changes: 2 additions & 2 deletions apps/web/src/lib/sandbox-diagnostic.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ describe("sandbox diagnostics", () => {
expect(sandboxDiagnosticMessage("provider_unavailable")?.label).toBe("Sandbox provider unavailable");
});
it("names why a node's provider is not ready, reading an unknown code as provider_unavailable", () => {
expect(sandboxDiagnosticMessage(nodeProviderDiagnostic({ online: true, provider_ready: false, diagnostic: "kvm_unavailable" }))?.label).toBe("KVM unavailable");
expect(sandboxDiagnosticMessage(nodeProviderDiagnostic({ online: true, provider_ready: false, diagnostic: "host_unsupported" }))?.label).toBe("Host unsupported");
expect(nodeProviderDiagnostic({ online: true, provider_ready: false, diagnostic: "future_code" as SandboxNodeDiagnostic })).toBe("provider_unavailable");
expect(nodeProviderDiagnostic({ online: true, provider_ready: true, diagnostic: "" })).toBe("");
// An offline node's last code may no longer apply.
expect(nodeProviderDiagnostic({ online: false, provider_ready: false, diagnostic: "docker_unavailable" })).toBe("");
expect(nodeProviderDiagnostic({ online: false, provider_ready: false, diagnostic: "host_unsupported" })).toBe("");
});
it("does not expose an unknown raw error or turn it into a healthy state", () => {
const message = sandboxDiagnosticMessage("private-provider-error-with-secret");
Expand Down
24 changes: 8 additions & 16 deletions apps/web/src/lib/sandbox-diagnostic.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,16 @@ const diagnostics: Record<string, { label: MessageKey; advice: MessageKey }> = {
},
provider_unavailable: {
label: "Sandbox provider unavailable",
advice: "Restore the provider on the assigned node, then refresh. A connected node alone does not confirm that its sandbox provider is ready.",
advice: "Restore the provider on the assigned node, then refresh. Running the install command again on the host checks its requirements and names the fix.",
},
// Fixed Runtime preparation and provider readiness diagnostics; Core sends only the code.
docker_unavailable: {
label: "Docker unavailable",
advice: "The node can't reach the Docker daemon. Check that Docker is running and the node can use its socket.",
// Provider-neutral readiness classes; Core sends only the code, and the node keeps the local detail.
host_unsupported: {
label: "Host unsupported",
advice: "The host lacks a capability its sandbox provider requires. Running the install command again on the host checks its requirements and names the fix.",
},
docker_limits_unsupported: {
label: "Docker limits unsupported",
advice: "Docker on this host doesn't enforce CPU and memory limits. Enable cgroup limits.",
artifacts_unavailable: {
label: "Provider files missing",
advice: "Pinned provider files are missing or fail their checksum. Run the install command again.",
},
runtime_download_failed: {
label: "Runtime download failed",
Expand All @@ -41,14 +41,6 @@ const diagnostics: Record<string, { label: MessageKey; advice: MessageKey }> = {
label: "Runtime image missing",
advice: "The pinned Runtime image isn't on the host. Run the install command again.",
},
kvm_unavailable: {
label: "KVM unavailable",
advice: "/dev/kvm isn't available to the node. Enable virtualization or use a KVM-capable host.",
},
microsandbox_artifacts_unavailable: {
label: "microsandbox components missing",
advice: "microsandbox components are missing or fail their checksum. Run the install command again.",
},
capacity_insufficient: {
label: "Host too small",
advice: "The host has less CPU or memory than one sandbox needs. Use a bigger host or a smaller sandbox size.",
Expand Down
18 changes: 6 additions & 12 deletions contracts/agents-api/core.openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -946,12 +946,10 @@ definitions:
description: Fixed reason for the last reported unreadiness; absent while the provider is ready. Clients treat an unknown value as provider_unavailable.
enum:
- provider_unavailable
- docker_unavailable
- docker_limits_unsupported
- host_unsupported
- artifacts_unavailable
- runtime_download_failed
- runtime_image_unavailable
- kvm_unavailable
- microsandbox_artifacts_unavailable
- capacity_insufficient
type: string
enrollment_id:
Expand Down Expand Up @@ -1037,12 +1035,10 @@ definitions:
description: Fixed reason for the last reported unreadiness; absent while the provider is ready. Clients treat an unknown value as provider_unavailable.
enum:
- provider_unavailable
- docker_unavailable
- docker_limits_unsupported
- host_unsupported
- artifacts_unavailable
- runtime_download_failed
- runtime_image_unavailable
- kvm_unavailable
- microsandbox_artifacts_unavailable
- capacity_insufficient
type: string
enrollment_id:
Expand Down Expand Up @@ -1106,12 +1102,10 @@ definitions:
diagnostic:
enum:
- provider_unavailable
- docker_unavailable
- docker_limits_unsupported
- host_unsupported
- artifacts_unavailable
- runtime_download_failed
- runtime_image_unavailable
- kvm_unavailable
- microsandbox_artifacts_unavailable
- capacity_insufficient
type: string
ready_generation:
Expand Down
Loading
Loading