Notice the stop — the handler was the right shape and did nothing

CLAUDE.md

@@ -194,10 +194,40 @@ covered by a test, and none of them is enforced by the platform.
    so an inventory local to the work function strands everything already uploaded.
    Record an object *before* its upload starts: a store can accept a body after the client
    is gone, and an object nobody named is never looked at again.
-6. **Handle SIGTERM.** This process is PID 1, and Linux gives process 1 no default signal
-   handling — without a handler the signal is discarded entirely. Set a flag, never do
-   work in the handler, and check the flag *between units of work* so a stop changes what
-   the step does next rather than only how it ends.
+6. **Handle SIGTERM, and notice it without waiting for the network.** This process is
+   PID 1, and Linux gives process 1 no default signal handling — without a handler the
+   signal is discarded entirely. Set a flag, never do work in the handler, and check the
+   flag *between units of work* so a stop changes what the step does next rather than only
+   how it ends. The flag is not enough on its own: a process parked in a socket call
+   cannot read it, so the handler also shuts down the transport in flight. Two measured
+   facts decide the shape of that — closing the *response* does nothing (mid-read it
+   raises `reentrant call inside <_io.BufferedReader>` inside the handler, where it is
+   swallowed, and the read waits out its whole timeout), and a response does not exist at
+   all while the store is still deciding whether to answer. Registering the *connection*
+   and calling `shutdown` on its socket is what works, and it took this file from 12.2
+   seconds to 0.2. Then ask it of every OTHER wait on the path, because that fix shipped
+   with two of them still open: the connection must go on the ledger before the call that
+   blocks, and it must NOT come off when `http.client` closes it after the headers of a
+   `Connection: close` response, or the ledger is empty for the whole body. Getting a
+   connection (DNS, TCP, TLS) cannot be interrupted at all — `ssl` detaches the socket
+   while wrapping it — so bound it with its own ELAPSED deadline and re-check the flag when
+   it returns. A timeout is not a deadline: `create_connection` spends yours once per
+   address, `getaddrinfo` ignores it entirely (so the lookup needs a thread), and a proxy's
+   CONNECT spends it a second time unless you recompute after the tunnel. Size it for a real
+   job — nothing retries an external step automatically.
+7b. **One receipt, or none.** Decide what it says from the flag BEFORE composing it, protect
+   that write from your own handler, give it an elapsed deadline (a socket timeout measures
+   silence, not duration), and never write a second, correcting document to the same name: a
+   write that failed ambiguously may still be accepted and may commit after its own
+   correction. This repository shipped both repairs — abandonment, then correction — and
+   both lost the same way. Give that write an ELAPSED deadline (a socket timeout measures
+   silence), and size it knowing it is best-effort: **nothing tells the container how long
+   it has after a stop** — not the injected variables, not the credentials envelope, not
+   the job description, and the orchestrator's stop object stops at the agent — so no
+   positive number survives a remaining grace of zero. What makes that safe is measured on the platform side: the
+   orchestrator decides an outcome from its own journal, a marker can only veto a success,
+   a stopped attempt's objects are salvaged as diagnostics rather than published, and the
+   operator's sentence quotes `exit_code` and `error` but never `status`.
 7. **Write the marker last**, and write one on the failure and cancellation paths too,
    with the real exit code and an inventory of whatever already landed.
 8. **Classify exits honestly.** 0 succeeded, 1 a later attempt might survive, 10 no retry

README.md

@@ -9,7 +9,7 @@ whatever the container declares it produced.
 
 **`node.py` is safe to copy.** It passes the whole conformance suite: the harness in
 `conformance/` builds this repository's image, runs it as a real container and judges it
-from the outside only, and **all 128 of its tests are green**.
+from the outside only, and **all 141 of its tests are green**.
 
 That was not true until recently. Twenty tests used to be red, and
 [CONFORMANCE-BASELINE.md](CONFORMANCE-BASELINE.md) is the measured record of what each one
@@ -28,10 +28,12 @@ one of which decides whether a real job survives:
 * it re-reads its credentials, so a run longer than fifteen minutes can still upload;
 * it keeps its inventory where the failure path can see it, and records an object before
   the upload starts;
-* it handles a stop request, so a cancelled run stops taking on new work instead of
-  running to completion for nobody;
-* it writes the completion marker last, on every path, with the exit code the process
-  really returns;
+* it handles a stop request — stops taking on new work instead of running to completion
+  for nobody, and *notices* the stop instead of sitting in a socket call until it times
+  out, which is the half a handler usually leaves out;
+* it writes the completion marker last, on every path, with the exit code it is about to
+  return — *about to*, because a kill landing between that document and the process's own
+  exit is a boundary nothing can make atomic;
 * it never prints a presigned URL.
 
 Read the label on any test before treating it as a rule: they do not carry the same

conformance/docker.py

@@ -16,6 +16,8 @@
 
 from __future__ import annotations
 
+import contextlib
+import ipaddress
 import json
 import os
 import re
@@ -306,9 +308,20 @@ def start(
     user: str | None = None,
     entrypoint: str | None = None,
     command: tuple[str, ...] = (),
+    extra_hosts: tuple[tuple[str, str], ...] = (),
+    dns: tuple[str, ...] = (),
+    dns_options: tuple[str, ...] = (),
 ) -> Container:
     """Start one workload container, detached.
 
+    ``extra_hosts``, ``dns`` and ``dns_options`` exist for one question the harness cannot
+    ask any other way: how long does this node spend GETTING a connection? Repeating a name
+    in ``extra_hosts`` puts several addresses in the container's ``/etc/hosts``, which is
+    how a test produces a host whose addresses must each be tried; pointing ``dns`` at an
+    address nobody answers, with the resolver's own patience widened by ``dns_options``,
+    is how a test produces a name lookup that hangs. Both are properties of the container's
+    network, not of the store, so no fake server can express either.
+
     ``unset_env`` names variables to REMOVE from the container's environment even if the
     image baked them in. ``docker run -e NAME`` with no ``=`` and no value on the host
     does exactly that — it drops the image's own ``ENV`` for that name. It is the only
@@ -328,6 +341,12 @@ def start(
         '--log-opt', 'max-size=10m',
         '--log-opt', 'max-file=3',
     ]
+    for host, address in extra_hosts:
+        argv += ['--add-host', f'{host}:{address}']
+    for server in dns:
+        argv += ['--dns', server]
+    for option in dns_options:
+        argv += ['--dns-option', option]
     for key, value in env.items():
         argv += ['--env', f'{key}={value}']
     for key in unset_env:
@@ -348,3 +367,112 @@ def start(
     if out.returncode != 0:
         raise DockerUnavailable(f'docker run failed: {out.stderr.strip()}')
     return Container(name=name, started_at=time.monotonic())
+
+
+#: What the silent resolver prints once it is bound and dropping queries. Waiting for this
+#: line is the difference between a test synchronised on evidence and one synchronised on a
+#: guess: a resolver that has not bound yet answers with an ICMP refusal, which makes a
+#: lookup fail in milliseconds and a test about a HANGING lookup pass for the wrong reason.
+RESOLVER_READY = 'silent-resolver-bound'
+
+_SILENT_RESOLVER = f"""
+import socket, sys
+handle = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+handle.bind(('0.0.0.0', 53))
+print({RESOLVER_READY!r}, flush=True)
+while True:
+    handle.recvfrom(4096)
+"""
+
+
+@contextlib.contextmanager
+def silent_resolver(image: str, *, timeout: float = 30.0):
+    """A container that RECEIVES every DNS query and answers none. Yields its address.
+
+    A name lookup only hangs if the query is delivered and ignored. An unroutable
+    nameserver does not do it — measured, inside a container: an address in TEST-NET-3
+    fails in **0.4 s** with "Temporary failure in name resolution", because nothing is
+    routed and the kernel says so at once. A listener that swallows the packet gives the
+    resolver nothing to conclude, so it waits its configured patience, twice over, and the
+    lookup takes tens of seconds (measured: 20.0 s at ``timeout:5 attempts:2``).
+
+    It has to be a container because port 53 is privileged in the HOST's namespace and this
+    harness must never need root; inside a container of its own it is ordinary. It reuses
+    the image this suite already built, so nothing is pulled.
+    """
+    require_docker()
+    name = f'lspo-conformance-resolver-{uuid.uuid4().hex[:10]}'
+    started = subprocess.run(
+        ['docker', 'run', '--detach', '--rm', '--name', name, '--user', '0:0',
+         '--entrypoint', 'python3', image, '-c', _SILENT_RESOLVER],
+        capture_output=True, text=True,
+    )
+    if started.returncode != 0:
+        raise DockerUnavailable(f'the silent resolver would not start: {started.stderr.strip()}')
+    try:
+        deadline = time.monotonic() + timeout
+        address = ''
+        while time.monotonic() < deadline:
+            logs = subprocess.run(['docker', 'logs', name], capture_output=True, text=True)
+            if RESOLVER_READY in (logs.stdout + logs.stderr):
+                found = subprocess.run(
+                    ['docker', 'inspect', '-f', '{{.NetworkSettings.IPAddress}}', name],
+                    capture_output=True, text=True,
+                )
+                address = found.stdout.strip()
+                if address:
+                    break
+            time.sleep(0.1)
+        if not address:
+            raise DockerUnavailable('the silent resolver never reported itself bound')
+        yield address
+    finally:
+        subprocess.run(['docker', 'kill', name], capture_output=True, text=True)
+
+
+#: Addresses whose packets are DROPPED rather than refused, so a connect to one waits out
+#: the caller's timeout instead of failing. Reaching them goes to the container's default
+#: gateway, which has nowhere to send them and says nothing back.
+#:
+#: Three kinds of "unreachable" were measured from inside a container, and only the third
+#: is any use for asking how long a step is prepared to spend connecting:
+#:
+#: * TEST-NET-3 (``203.0.113.7``) — fails in **0.1 s**: nothing is routed there and the
+#:   kernel says so at once;
+#: * an unassigned address on the container's own bridge subnet (``172.17.255.254``) —
+#:   fails in **~3 s** whatever timeout is asked for, because the ARP for it goes
+#:   unanswered and the kernel gives up on its own schedule;
+#: * these — **4.0 s against a 4-second timeout, and 12.0 s across three of them**, which
+#:   is the behaviour a test about connect budgets needs to see.
+SILENTLY_DROPPED = ('10.255.255.1', '10.255.255.2', '10.255.255.3')
+
+_TIME_A_CONNECT = """
+import socket, sys, time
+began = time.monotonic()
+try:
+    socket.create_connection((sys.argv[1], 9), float(sys.argv[2]))
+except Exception:
+    pass
+print('%.2f' % (time.monotonic() - began), flush=True)
+"""
+
+
+def seconds_spent_connecting(
+    image: str, address: str, *, timeout: float = 2.0, extra_hosts: tuple[tuple[str, str], ...] = ()
+) -> float:
+    """How long a container spends failing to reach ``address``. For checking a premise.
+
+    Whether a packet is dropped in silence or refused is a fact about the machine this
+    suite happens to run on, not about the node — so a test that needs a connect to HANG
+    has to establish that one does here, and say so rather than pass quietly when it does
+    not. This is what it asks with.
+    """
+    require_docker()
+    argv = ['docker', 'run', '--rm']
+    for host, host_address in extra_hosts:
+        argv += ['--add-host', f'{host}:{host_address}']
+    argv += ['--entrypoint', 'python3', image, '-c', _TIME_A_CONNECT, address, str(timeout)]
+    done = subprocess.run(argv, capture_output=True, text=True)
+    if done.returncode != 0:
+        raise DockerUnavailable(f'the connect probe would not run: {done.stderr.strip()}')
+    return float(done.stdout.strip().splitlines()[-1])

conformance/fakes3.py

@@ -145,6 +145,10 @@ class Request:
     #: become unauthorized because its body took a while to arrive.
     arrived_at: float = 0.0
     fields: dict = field(default_factory=dict)  #: the POST form fields, for an upload
+    #: Seconds to spend DRIBBLING the answer out, a byte at a time, instead of sending it.
+    #: A store that goes quiet and a store that answers slowly are different faults, and a
+    #: client can only tell them apart if it measures elapsed time rather than silence.
+    drip_for: float = 0.0
 
 
 @dataclass
@@ -628,10 +632,26 @@ def _store_or_refuse(self, request: Request, key: str, payload: bytes) -> None:
                 self._error(refused)
                 return
             endpoint._record_upload(request, key, payload)
+            if request.drip_for:
+                self._drip(b'HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n', request.drip_for)
+                return
             self.send_response(204)
             self.send_header('Content-Length', '0')
             self.end_headers()
 
+        def _drip(self, answer: bytes, seconds: float) -> None:
+            """Answer one byte at a time, never pausing long enough to look idle.
+
+            This is the shape a socket timeout cannot catch: every gap is short, so nothing
+            is ever "quiet", and yet the exchange takes as long as the store feels like.
+            A client that bounds only silence waits it out in full.
+            """
+            per_byte = seconds / max(len(answer), 1)
+            for index in range(len(answer)):
+                self.wfile.write(answer[index:index + 1])
+                self.wfile.flush()
+                time.sleep(per_byte)
+
         # ----------------------------------------------------------------- errors
 
         def _error(self, refused: Refused) -> None:
@@ -737,3 +757,17 @@ def hook(endpoint: 'Endpoint', request: Request) -> None:
             time.sleep(seconds)
 
     return hook
+
+
+def drip_when(predicate, seconds: float):
+    """Answer the matching request a byte at a time, taking ``seconds`` over it.
+
+    For the one property a socket timeout cannot express: a transfer that is never idle
+    and never ends. Everything else in this module models a store that goes QUIET.
+    """
+
+    def hook(endpoint: 'Endpoint', request: Request) -> None:
+        if predicate(request):
+            request.drip_for = seconds
+
+    return hook