jonbaldie · jonbaldie · Sep 26, 2026 · Sep 26, 2026
diff --git a/docs/exploratory-testing/2026-09-26-queue/2026-09-26-report.md b/docs/exploratory-testing/2026-09-26-queue/2026-09-26-report.md
@@ -0,0 +1,96 @@
+# Queue exploratory-testing report — 2026-09-26
+
+## Outcome
+
+Comprehensive automated and exploratory testing across all public HTTP endpoints, payload ingestion validation, protocol invariants, persistence contracts, capacity limits, and Docker deployment completed with **zero defects detected**. The current codebase on `main` (commit `176cddf`, including the newly extracted `src/payload.ts` ingestion module) is robust, correct, and strictly adheres to specifications and RFCs. No new bugs were found, and no issues were filed.
+
+## Setup and evidence
+
+All journeys drove the service strictly through its public HTTP interface against real `deno run --allow-all main.ts` processes and Docker containers using standard HTTP requests. No mocks, internal monkey-patching, or synthetic stubs were used. Isolated temporary directories were used for each persistence test and removed upon completion.
+
+- Environment: Deno 2.7.6 (`aarch64-apple-darwin`), Docker 29.4.0 (macOS 15.x / Apple Silicon).
+- Base test driver utility: [driver.ts](./driver.ts)
+- Comprehensive exploratory test suite: [run_exploratory.ts](./run_exploratory.ts)
+- Full four-journey transcript: [journey-transcript.txt](./journey-transcript.txt)
+
+## Journeys exercised
+
+### 1. Core FIFO, Multi-Queue Isolation, Payload Fidelity & Edge Values
+
+- **Health check probes**: Verified `GET /health`, `HEAD /health`, and trailing slash `/health/` return `200 OK` without authentication. Confirmed that `HEAD` responses return empty bodies while preserving `Content-Type: application/json`.
+- **Initial state**: Confirmed that an empty server returns `[]` on `GET /queues`.
+- **Rich payload types & fidelity**: Enqueued 22 distinct payloads to queue `orders` via the newly refactored `src/payload.ts` ingestion pipeline:
+  - Strings: ASCII (`"order-101"`), Unicode emojis and multi-language characters (`"🔥 rocket 🚀 ñoño 漢字 こんにちは"`), empty string (`""`), multiline strings (`"line1\nline2\r\nline3"`), JSON escaped strings (`"\"quoted\" and \\backslashed\\"`).
+  - Numbers: zero (`0`), negative zero (`-0`), positive integers (`42`), negative integers (`-42`), floating-point values (`3.14159`), positive scientific notation (`1e5`), negative scientific notation (`1e-5`), `Number.MAX_SAFE_INTEGER + 1` (`9007199254740992`), `Number.MIN_SAFE_INTEGER - 1` (`-9007199254740992`).
+  - Booleans: `true` and `false`.
+  - Objects: empty object (`{}`), shallow objects (`{"item":"widget","count":5}`), deeply nested structures (`{"user":{"id":1,"active":true,"tags":["a","b"]}}`), and objects with `__proto__` keys.
+  - Arrays: empty array (`[]`) and mixed arrays (`[1, "two", false, null, {"nested": true}]`).
+- **Multi-queue isolation**: Enqueued an item to a separate queue (`notifications`). Verified that `GET /queues` returned both active queues and operations on `orders` had no side-effects on `notifications`.
+- **Non-mutating inspections**: Verified `GET /peek/orders`, `HEAD /peek/orders`, and `HEAD /dequeue/orders` returned `200 OK` without consuming items or altering queue length.
+- **Strict FIFO ordering**: Dequeued all 22 items sequentially, asserting each item emerged in the exact order enqueued with 100% value and type fidelity.
+- **Empty queue draining & pruning**: Dequeued from empty `orders`, verifying `204 No Content` with no body, and verified that `orders` was automatically pruned from `GET /queues`. Drained `notifications` and confirmed `GET /queues` returned `[]`.
+
+### 2. Protocol, Validation, Authentication, Limits & Error Paths
+
+- **Authentication enforcement**: Confirmed unauthenticated requests to protected endpoints return `401 Unauthorized` with the required `WWW-Authenticate: Bearer` challenge. Verified valid tokens authenticate regardless of scheme casing (`bearer`, `Bearer`, `BEARER`) or whitespace formatting between scheme and token (single space, multiple spaces, tabs).
+- **HTTP method routing (RFC 9110 §15.5.6)**: Verified that sending unsupported HTTP methods returns `405 Method Not Allowed` with the exact `Allow` header matching supported methods (`POST /health` -> `Allow: GET`, `GET /enqueue/q` -> `Allow: POST`, `POST /dequeue/q` -> `Allow: GET`, etc.).
+- **Route 404s**: Verified unrouted paths (`/unknown-path`, `/enqueue`, `/enqueue/`, `/dequeue/`) return `404 Not found.`.
+- **Payload validation**:
+  - Empty body and malformed JSON return `400 Invalid JSON`.
+  - Non-object bodies (`123`, `[]`) or objects missing the required `"payload"` key return `400 Missing payload key`.
+  - Top-level `null` payloads (`{"payload":null}`) return `400 Null payload not allowed`.
+  - Unsafe numbers exceeding IEEE 754 precision (`9007199254740993`, `-9007199254740993`), underflow numbers (`1e-325`), and overflow numbers (`1e309` / `Infinity`) return `400 Payload contains an unsupported number`.
+  - Invalid UTF-8 byte sequences return `400 Invalid JSON`.
+- **Nesting depth boundaries**:
+  - Nesting depth at exactly 3,000 container levels (`MAX_JSON_DEPTH`): accepted with `200 OK`.
+  - Nesting depth exceeding 3,000 levels (3,001 levels): rejected with `400 Invalid JSON`.
+  - Parser stack overflow (> 50,000 levels): cleanly caught and rejected with `400 Invalid JSON` (no uncaught 500 error).
+- **Queue name limits**: 128 Unicode characters / 128 emojis accepted; 129 characters / 129 emojis rejected with `400 Queue name too long`. Malformed percent encodings (`%ZZ`) return `400 Invalid queue name`.
+- **Body size limits**: Verified bodies of exactly 1 MB (1,048,576 bytes) are accepted with `200 OK`, while bodies exceeding 1 MB (1,048,577 bytes) are rejected with `413 Payload too large`.
+- **Queue capacity limits**:
+  - `QUEUE_DEPTH_LIMIT=3`: Enqueued 3 items; 4th item returned `507 Queue full or too many queues`. Dequeueing 1 item freed capacity and allowed immediate re-enqueue.
+  - `QUEUE_COUNT_LIMIT=2`: Created 2 queues; 3rd queue returned `507 Queue full or too many queues`. Fully draining 1 queue freed capacity and allowed creating a new queue.
+- **Rate limiting**: Configured `RATE_LIMIT_REQUESTS=5`; requests 1–5 returned `200 OK`, request 6 returned `429 Too many requests`. Verified `/health` bypassed the rate limiter and remained responsive (`200 OK`).
+
+### 3. Persistence, Clean Restarts & Crash Recovery
+
+- **Clean shutdown and recovery (SIGTERM)**:
+  - Started server with `--persist` and an isolated temporary directory.
+  - Enqueued items across queues `p1` and `p2`, dequeued one item, and initiated a graceful `SIGTERM` shutdown.
+  - Verified `persist.dat` snapshot on disk contained only remaining active items (`beta` on `p1`, `gamma` on `p2`).
+  - Started a second clean server pointing to the same persist directory. Verified that active queues, lengths, and FIFO ordering were restored cleanly, and that draining queues left an empty snapshot.
+- **Crash recovery / unclean shutdown (SIGKILL)**:
+  - Started server with `--persist` and an isolated directory.
+  - Enqueued 3 items to queue `tasks` (`task-1`, `task-2`, `task-3`) and dequeued `task-1`.
+  - Sent `SIGKILL` to simulate an unhandled crash / power failure without graceful `save()`.
+  - Verified `persist.dat` retained the raw 4-line transaction log (3 enqueues + 1 dequeue).
+  - Started a recovery server against the same directory with identical limits. Confirmed that `tasks` was recovered with length 2 and that subsequent dequeues returned `task-2`, `task-3`, and then `204 No Content`.
+- **Corrupted persistence log recovery**:
+  - Injected malformed lines (invalid JSON, missing fields, corrupted event structures) alongside valid events into `persist.dat`.
+  - Started recovery server with `--persist`. Verified server booted without error, recovered the valid items, ignored the corrupted lines, and compacted cleanly into a valid snapshot.
+- **Automatic directory creation**:
+  - Configured `PERSIST` to a non-existent deeply nested path (`.../nested/sub/data/`). Verified server creates the missing path and writes `persist.dat` successfully without startup failure.
+
+### 4. Docker Container Deployment & Volume Persistence
+
+- Built Docker image `queue-explore:20260926` locally from current `Dockerfile`.
+- Confirmed container runs as non-root user `deno` (`uid=1993(deno)`).
+- Started container with `--persist` and a dedicated named volume mounted at default `/data`.
+- Enqueued payload `"persisted-in-docker"` to queue `docker-q` and received `200 OK`.
+- Stopped and restarted the container (`docker stop` followed by `docker start`).
+- Dequeued from `docker-q` and retrieved `"persisted-in-docker"` with `200 OK`, confirming persistent volume storage across container restarts.
+
+## Rejected candidates & usability observations
+
+- **Observation on legacy verification script**: `verify_dockerfile_hardening.sh` checks for scoped `--allow-write=` and `--allow-net=` compile flags in `Dockerfile`. Those flags were intentionally relaxed to unscoped `--allow-write` and `--allow-net` in PRs #65 and #68 to enable configurable runtime persistence directories (`PERSIST`) and configurable listening addresses/ports (`HOST`/`PORT`). The verification script is a legacy helper from earlier development and is not part of CI or documented build steps.
+- **Observation on OpenAPI 400 descriptions**: In `openapi.yaml`, the 400 responses for `/dequeue`, `/peek`, and `/length` mention `Bad Request - Queue name too long`. An invalid percent-encoded sequence (`%ZZ`) also triggers a `400 Bad Request` with body `Invalid queue name`. The behavior is correct and safe, but `openapi.yaml` could document `Invalid queue name` on those endpoints.
+
+## Limitations and unexplored areas
+
+- Testing was performed on macOS APFS (`aarch64-apple-darwin`). Direct physical filesystem-full conditions during `writeSync` were not simulated at the host kernel level.
+- Multi-region proxy topologies and high latency networks were not simulated.
+
+## Cleanup
+
+- All Docker containers, volumes, and temporary images created during the pass (`qx-explore-20260926`, `queue-explore-vol-20260926`, `queue-explore:20260926`) were removed.
+- All temporary persistence directories (`queue-explore-persist-*`, `queue-explore-crash-*`, `queue-explore-corrupt-*`) were removed upon completion.
diff --git a/docs/exploratory-testing/2026-09-26-queue/driver.ts b/docs/exploratory-testing/2026-09-26-queue/driver.ts
@@ -0,0 +1,156 @@
+const REPO_DIR = Deno.cwd();
+const TOKEN = "explore-token-20260926";
+
+export type ResponseSummary = {
+    status: number;
+    body: string;
+    headers: Record<string, string>;
+};
+
+export type RunningServer = {
+    child: Deno.ChildProcess;
+    persistDir: string;
+    port: number;
+    stdout: string;
+    stderr: string;
+    stdoutTask: Promise<void>;
+    stderrTask: Promise<void>;
+};
+
+export function compactBody(body: string): string {
+    if (body.length <= 240) return body;
+    return `${body.slice(0, 240)}… (${body.length} characters)`;
+}
+
+export function selectedHeaders(headers: Headers): Record<string, string> {
+    const selected: Record<string, string> = {};
+    for (const name of ["allow", "content-length", "content-type", "www-authenticate"]) {
+        const value = headers.get(name);
+        if (value !== null) selected[name] = value;
+    }
+    return selected;
+}
+
+export async function request(
+    server: RunningServer,
+    pathname: string,
+    init: RequestInit = {},
+): Promise<ResponseSummary> {
+    const response = await fetch(`http://127.0.0.1:${server.port}${pathname}`, init);
+    const body = await response.text();
+    return {
+        status: response.status,
+        body: compactBody(body),
+        headers: selectedHeaders(response.headers),
+    };
+}
+
+export function authHeaders(contentType = false, customToken = TOKEN): Record<string, string> {
+    return contentType
+        ? { Authorization: `Bearer ${customToken}`, "Content-Type": "application/json" }
+        : { Authorization: `Bearer ${customToken}` };
+}
+
+export function postInit(body: string | Uint8Array, customToken = TOKEN): RequestInit {
+    return {
+        method: "POST",
+        headers: authHeaders(true, customToken),
+        body,
+    };
+}
+
+export function record(journey: string, name: string, result: ResponseSummary, expectation: string): void {
+    console.log(JSON.stringify({ journey, name, result, expectation }));
+}
+
+export async function startServer(
+    extraEnv: Record<string, string> = {},
+    persist = false,
+    persistDir = "",
+): Promise<RunningServer> {
+    const child = new Deno.Command(Deno.execPath(), {
+        args: ["run", "--allow-all", "main.ts", ...(persist ? ["--persist"] : [])],
+        cwd: REPO_DIR,
+        env: {
+            HOST: "127.0.0.1",
+            PORT: "0",
+            QUEUE_API_TOKEN: TOKEN,
+            ...(persistDir ? { PERSIST: persistDir } : {}),
+            ...extraEnv,
+        },
+        stdout: "piped",
+        stderr: "piped",
+    }).spawn();
+
+    let stdout = "";
+    let stderr = "";
+    let started: (port: number) => void = () => {};
+    let failed: (error: Error) => void = () => {};
+    const startedPromise = new Promise<number>((resolve, reject) => {
+        started = resolve;
+        failed = reject;
+    });
+
+    const stdoutTask = (async () => {
+        const reader = child.stdout.getReader();
+        const decoder = new TextDecoder();
+        try {
+            while (true) {
+                const { done, value } = await reader.read();
+                if (done) break;
+                stdout += decoder.decode(value, { stream: true });
+                const match = stdout.match(/Listening on (?:http:\/\/)?(?:127\.0\.0\.1|localhost|0\.0\.0\.0):(\d+)/);
+                if (match) started(Number(match[1]));
+            }
+        } finally {
+            reader.releaseLock();
+        }
+    })();
+
+    const stderrTask = (async () => {
+        const reader = child.stderr.getReader();
+        const decoder = new TextDecoder();
+        try {
+            while (true) {
+                const { done, value } = await reader.read();
+                if (done) break;
+                stderr += decoder.decode(value, { stream: true });
+                if (!stdout.includes("Listening on") && stderr.includes("ConfigError")) {
+                    failed(new Error(stderr));
+                }
+            }
+        } finally {
+            reader.releaseLock();
+        }
+    })();
+
+    const timeout = new Promise<never>((_, reject) => {
+        setTimeout(() => reject(new Error(`startup timeout; stdout=${stdout}; stderr=${stderr}`)), 5000);
+    });
+
+    let port: number;
+    try {
+        port = await Promise.race([startedPromise, timeout]);
+    } catch (error) {
+        try {
+            child.kill("SIGKILL");
+        } catch {
+            // The child may already have exited.
+        }
+        await child.status.catch(() => {});
+        await Promise.allSettled([stdoutTask, stderrTask]);
+        throw error;
+    }
+
+    return { child, persistDir, port, stdout, stderr, stdoutTask, stderrTask };
+}
+
+export async function stopServer(server: RunningServer, signal: Deno.Signal = "SIGTERM"): Promise<void> {
+    try {
+        server.child.kill(signal);
+    } catch {
+        // Already exited.
+    }
+    await server.child.status.catch(() => {});
+    await Promise.allSettled([server.stdoutTask, server.stderrTask]);
+}