Follow a run while it happens: logs.sh - #7
Merged
Conversation
The output of a step is hard to read for a reason nobody guesses on their first
attempt: the agent removes a job's container when the job ends. `docker logs` on
the container you wanted therefore finds nothing, and the runs most worth reading
are exactly the short ones that went wrong.
logs.sh attaches at the moment a container STARTS, by watching the daemon's event
stream and filtering on the lspo.agent label the agent puts on every job container
— so on a machine running several agents it follows only this one's work. It can
also tail the agent itself, which is the journal that explains a run that never
started at all: a refused image pull, a credentials directory it could not write,
an environment variable outside the allowlist. None of those ever reach a
container, so none of them appears in a node's log.
./logs.sh agent | node | both (both is the default)
Verified against a container carrying the agent's own labels: the watcher caught
it at start and streamed its lines through to exit.
docs/OPERATIONS.md gains a section saying why it is needed and which of the two
journals to trust — the platform keeps a tail of the last 1000 lines and the
durable copy is built from that same trimmed buffer, so the copy on the machine
is the complete one. The troubleshooting table gains the row that sends people
there, because "the output was gone before I could read it" is the symptom, and
its cause is not a bug to hunt.
It reads and never writes, nothing in the contract mentions it, and a node that
never runs it is in no way worse off. The inventory rows say so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reading a step's output is harder than it should be, for a reason nobody guesses on their first attempt: the agent removes a job's container when the job ends. So
docker logson the container you wanted finds nothing — and the runs most worth reading are exactly the short ones that went wrong.logs.shattaches at the moment a container starts, by watching the daemon's event stream.It filters on the
lspo.agentlabel the agent puts on every job container, so on a machine running several agents it follows only this one's work. Job containers are namedlspo-<execution>-g<generation>.The agent half matters on its own: it is the journal that explains a run which never started at all — a refused image pull, a credentials directory it could not write, an environment variable outside the allowlist. None of those ever reach a container of yours, so none of them appears in a node's log, and somebody looking only at the node sees silence.
Which of the two journals to trust
The same node output is shipped to the orchestrator and appears live in the run view, which is the only option when the agent is on somebody else's machine. They are not the same document. The platform keeps a tail of the last 1000 lines and drops the oldest when a run is chattier than that, and the durable copy stored with the execution is built from that same trimmed buffer (PROTOCOL.md §3.3 — which is also why a step should say the important things once, at the end, in few lines). The copy on the machine is the whole of it.
docs/OPERATIONS.mdgains a section saying that, and the troubleshooting table gains the row that sends people to it: "a container's output is gone before you could read it" is the symptom, and its cause is a behaviour to know rather than a bug to hunt.Scope
It reads and never writes. Nothing in the contract mentions it, and a node that never runs it is in no way worse off — the inventory rows in
README.mdandCLAUDE.mdsay exactly that, so nobody mistakes it for a requirement.Verified
Run against a container carrying the agent's own labels — the watcher caught it at start and streamed its lines through to exit:
bash -nclean;--helpworks; 129 tests still collect. No code, image or test behaviour is touched.🤖 Generated with Claude Code