Skip to content

run.sh: seven checks still accuse the engine when no interpreter answers (the other half of #51) #54

Description

@marcobambini

#51 fixed the case where python3 is present and is not an interpreter. It
did not finish the case where nothing answers, and the checks that are
left do the thing #51 was written to stop: they report the engine broken
because an interpreter is missing.

Measured on this machine with both #51 and #52 merged, tests/run.sh /nonexistent, against a PATH where python3, python and py all exist
and all exit 49:

passed failed skipped
before #51 7 14 14
after #51 + #52 7 7 22
target 7 0 29

Halved, not closed. The seven that remain:

peak RSS exceeded the budget on K3          default budget not capped on K3
info describes something else on K3         params_total off the rule on K3
K3 image normalization differs              convert.py GLM config
XTML prompts differ from encoding_k3.py

Every one of these is a real check reporting a real-sounding defect —
"peak RSS exceeded the budget", "image normalization differs from the
release" — when what happened is that no interpreter ran. That is exactly
what line 10 of tests/run.sh says this suite must never do, and it is the
finding #51 opened with.

They fall into three shapes

1. One guard still on the pattern #51 replaced. tests/run.sh:1938:

if ! command -v python3 >/dev/null 2>&1; then
    sk "convert.py GLM config" "python3 not installed"

#51 converted the other three of these — convert.py resume, convert.py chat.json, serve suite — to [ -n "$PY_MISS" ]. This one was not missed
so much as added after the PR was written: it arrived with the GLM work in
0.7.0, three days after #51 was opened, on the branch's blind side. It wants
the same one-line treatment.

2. Three helpers that shell out with no guard at all. Each runs
python3 inline and turns a non-zero exit into a verdict about the model:

  • info_ruletests/run.sh:1728, drives info describes something else on K3
  • params_ruletests/run.sh:1564, drives params_total off the rule on K3
  • vision_normtests/run.sh:1796, drives K3 image normalization differs

vision_norm already has a three-way return — its rc=2 is "no vision.json
or no preprocessor config", a SKIP — so it has the shape for this and simply
does not use it for a missing interpreter. The other two are two-way.

Note these only fire where the K3 container is present, so a bare CI box
skips them for want of weights and never sees the problem. That is why the
board is green everywhere and this is only visible on a real machine.

3. A subprocess with three unguarded calls of its own.
tests/check_budget.sh lines 15, 31 and 79 all call python3 directly, and
its caller reports the failure as peak RSS exceeded the budget on K3 and
default budget not capped on K3. #51's PATH shim covers this script when a
working interpreter exists under another name — which is the common Windows
case and the one that mattered — but not when none does. The guard belongs
in check_budget.sh or at its call site, and either way PY_MISS has to
reach it.

XTML prompts differ from encoding_k3.py (tests/run.sh:2003) is the same
shape as group 2: K3_DIR=... python3 -m unittest ..., unguarded.

Reproducing it

No Windows needed — the failure is "the name on PATH is not an interpreter",
and that is three lines:

d=$(mktemp -d)
for n in python3 python py; do
    printf '#!/bin/sh\nexit 49\n' > "$d/$n"; chmod +x "$d/$n"
done
PATH=$d:$PATH tests/run.sh /nonexistent

Dropping python and py from that loop gives the other half of #51's
finding — a python3 that exists and lies, with a working interpreter under
another name — which is now handled and stays handled: 61 passed, 0 failed,
9 skipped, identical to a clean machine.

Why this is worth closing rather than living with

The suite's rule is that a missing prerequisite is a SKIP, never a silent
pass — and never, which is worse, a loud accusation against something that
was never run. Seven checks currently break that rule in the direction that
costs the most: they name the engine, the budget, the converter and the
release's own preprocessing config, and every one of those is a plausible
place to start looking. docs/LEARNED.md §72 is about the same failure in a
different register — a check whose verdict does not say what it saw sends
the next person to re-derive the evidence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions