Skip to content

Follow a run while it happens: logs.sh - #7

Merged
makseq merged 1 commit into
mainfrom
docs/follow-the-logs-live
Aug 10, 2026
Merged

Follow a run while it happens: logs.sh#7
makseq merged 1 commit into
mainfrom
docs/follow-the-logs-live

Conversation

@makseq

@makseq makseq commented Aug 10, 2026

Copy link
Copy Markdown
Member

Reading a step's output is harder than it should be, for a reason nobody guesses on their first attempt: the agent removes a job's container when the job ends. So docker logs on the container you wanted finds nothing — and the runs most worth reading are exactly the short ones that went wrong.

logs.sh attaches at the moment a container starts, by watching the daemon's event stream.

./logs.sh agent    # the agent: claims, heartbeats, container lifecycle
./logs.sh node     # every job container it starts, from that container's first line
./logs.sh both     # the two interleaved and tagged (the default)

It filters on the lspo.agent label the agent puts on every job container, so on a machine running several agents it follows only this one's work. Job containers are named lspo-<execution>-g<generation>.

The agent half matters on its own: it is the journal that explains a run which never started at all — a refused image pull, a credentials directory it could not write, an environment variable outside the allowlist. None of those ever reach a container of yours, so none of them appears in a node's log, and somebody looking only at the node sees silence.

Which of the two journals to trust

The same node output is shipped to the orchestrator and appears live in the run view, which is the only option when the agent is on somebody else's machine. They are not the same document. The platform keeps a tail of the last 1000 lines and drops the oldest when a run is chattier than that, and the durable copy stored with the execution is built from that same trimmed buffer (PROTOCOL.md §3.3 — which is also why a step should say the important things once, at the end, in few lines). The copy on the machine is the whole of it.

docs/OPERATIONS.md gains a section saying that, and the troubleshooting table gains the row that sends people to it: "a container's output is gone before you could read it" is the symptom, and its cause is a behaviour to know rather than a bug to hunt.

Scope

It reads and never writes. Nothing in the contract mentions it, and a node that never runs it is in no way worse off — the inventory rows in README.md and CLAUDE.md say exactly that, so nobody mistakes it for a requirement.

Verified

Run against a container carrying the agent's own labels — the watcher caught it at start and streamed its lines through to exit:

[watch] following job containers started by agent 'makseq-creator'
[watch] job container started: lspo-777-g2
[lspo-777-g2] hello-node: execution 777 attempt 1
[lspo-777-g2] hello-node: done — 1 file(s)
[watch] lspo-777-g2 finished

bash -n clean; --help works; 129 tests still collect. No code, image or test behaviour is touched.

🤖 Generated with Claude Code

The output of a step is hard to read for a reason nobody guesses on their first
attempt: the agent removes a job's container when the job ends. `docker logs` on
the container you wanted therefore finds nothing, and the runs most worth reading
are exactly the short ones that went wrong.

logs.sh attaches at the moment a container STARTS, by watching the daemon's event
stream and filtering on the lspo.agent label the agent puts on every job container
— so on a machine running several agents it follows only this one's work. It can
also tail the agent itself, which is the journal that explains a run that never
started at all: a refused image pull, a credentials directory it could not write,
an environment variable outside the allowlist. None of those ever reach a
container, so none of them appears in a node's log.

    ./logs.sh agent | node | both        (both is the default)

Verified against a container carrying the agent's own labels: the watcher caught
it at start and streamed its lines through to exit.

docs/OPERATIONS.md gains a section saying why it is needed and which of the two
journals to trust — the platform keeps a tail of the last 1000 lines and the
durable copy is built from that same trimmed buffer, so the copy on the machine
is the complete one. The troubleshooting table gains the row that sends people
there, because "the output was gone before I could read it" is the symptom, and
its cause is not a bug to hunt.

It reads and never writes, nothing in the contract mentions it, and a node that
never runs it is in no way worse off. The inventory rows say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@makseq
makseq merged commit 0dd0a51 into main Aug 10, 2026
1 check passed
@makseq
makseq deleted the docs/follow-the-logs-live branch August 10, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant