Skip to content

Harden physical hardware assistant validation #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tsotchke merged 1 commit into from
May 21, 2026
Merged

Harden physical hardware assistant validation #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -460,9 +460,13 @@ until the same quality, performance, and assistant probes are captured from a
physical 486-class DOS machine. Pentium timing is useful scaling evidence, but
it is not a blocker for the solid 486-focused release. The hardware ladder is
tracked in [`docs/hardware-validation.md`](docs/hardware-validation.md), with a
DOS capture batch under `hardware/HWVALID.BAT`, strict host verification
through `scripts/verify_hardware_capture.py --require-filled-notes`, and
release evidence staging through `scripts/stage_hardware_capture_evidence.py`.
DOS capture batch under `hardware/HWVALID.BAT` that writes `QUAL.LOG`,
`PERF.LOG`, `ASSIST.LOG`, `ASTRESS.LOG`, and `ASSISTC.LOG`, strict host
verification through `scripts/verify_hardware_capture.py --require-filled-notes`,
and release evidence staging through
`scripts/stage_hardware_capture_evidence.py`. The physical assistant gate now
requires all five packs plus a 50-reply stress probe from the returned DOS
logs.
The physical-only performance table is generated by
`scripts/hardware_performance_matrix.py` from staged `hardware_<machine>_perf.log`
files only.
Expand Down
11 changes: 8 additions & 3 deletions docs/hardware-validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ the core release gate.
| Tier | Status | Hardware | Release Role | Required Logs |
|---|---|---|---|---|
| 0 | Complete | QEMU 486 profiles | Preview release gate | compile, quality, perf, assistant, vectors |
| 1 | Next gate | Any working 486-class DOS PC with 32-64 MB RAM | Solid release baseline | `QUAL.LOG`, `PERF.LOG`, `ASSIST.LOG`, `ASSISTC.LOG` |
| 1 | Next gate | Any working 486-class DOS PC with 32-64 MB RAM | Solid release baseline | `QUAL.LOG`, `PERF.LOG`, `ASSIST.LOG`, `ASTRESS.LOG`, `ASSISTC.LOG` |
| 2 | Useful | Faster 486DX2/DX4 or comparable late 486 board | Performance confidence | repeated `PERF.LOG`, optional kernel perf |
| 3 | Optional | Pentium 60/90/133+ | Scaling comparison only | `PERF.LOG`, optional quality confirmation |
| 4 | Optional | 386 or 486SX no-FPU class system | Compatibility stress test | quality and perf if memory allows |
Expand Down Expand Up @@ -56,6 +56,7 @@ That batch file runs the minimum test set:
GPT2.EXE --quality-all > QUAL.LOG
GPT2.EXE --perf > PERF.LOG
ASSIST.EXE --scripted > ASSIST.LOG
ASSIST.EXE --stress-probe > ASTRESS.LOG
```

Also keep the assistant compile log when building on the target:
Expand Down Expand Up @@ -106,8 +107,11 @@ verifies the paired `hardware_<machine>_manifest.md` checksum table.

- `QUAL.LOG` shows the same prompt suite completing without runtime failure.
- `PERF.LOG` contains all `PERF_*` rows and reports runtime memory.
- `ASSIST.LOG` includes CHAT, DOSHELP, and OFFICE pack records plus per-pack
usage instructions.
- `ASSIST.LOG` includes CHAT, DOSHELP, OFFICE, DEV, and PORTABLE pack records
plus per-pack usage instructions.
- `ASTRESS.LOG` includes `ASSIST_END|suite=stress-probe|packs=5`, exactly 50
`ASSIST_REPLY|` rows, no `status=model_unavailable` rows, and records for
CHAT, DOSHELP, OFFICE, DEV, and PORTABLE.
- `ASSISTC.LOG` includes `ASSIST_COMPILE_OK` when target-side compilation is
attempted.
- The hardware notes identify machine key, CPU, clock, RAM, DOS version,
Expand All @@ -122,6 +126,7 @@ qemu/evidence/hardware_<machine>_capture.log
qemu/evidence/hardware_<machine>_quality.log
qemu/evidence/hardware_<machine>_perf.log
qemu/evidence/hardware_<machine>_assistant.log
qemu/evidence/hardware_<machine>_assistant_stress.log
qemu/evidence/hardware_<machine>_assistant_compile.log
qemu/evidence/hardware_<machine>_notes.md
qemu/evidence/hardware_<machine>_manifest.md
Expand Down
5 changes: 4 additions & 1 deletion docs/releases/v0.1.0-preview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ https://github.com/tsotchke/gpt2-basic. The next release gate is physical
- `docs/hardware-validation.md`, which defines the non-emulator validation path.
- `hardware/HWVALID.BAT`, `hardware/HWNOTES.TXT`, generated
`C:\GPT2\RETURN.TXT`, and host verification for future physical-machine
captures.
captures. The returned hardware log set now includes `ASTRESS.LOG`, the
five-pack assistant stress proof.
- A minimal `C:\GPT2` hardware-transfer bundle builder for copying the release
payload to a physical DOS machine.
- `SHA256SUMS.txt` inside the package and a zip-level `.sha256` sidecar.
Expand Down Expand Up @@ -85,6 +86,8 @@ https://github.com/tsotchke/gpt2-basic. The next release gate is physical
## Known Limits

- Physical 486-class board timing is still pending.
- Physical assistant stress proof is pending until a real 486-class DOS
machine returns `ASTRESS.LOG`; QEMU already rehearses the same batch path.
- Pentium hardware is not required for the solid 486-focused release; it is an
optional scaling datapoint.
- The generic model is intentionally small; the release claims evidence-gated
Expand Down
6 changes: 5 additions & 1 deletion hardware/HWVALID.BAT
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ if exist HWVALID.LOG del HWVALID.LOG
if exist QUAL.LOG del QUAL.LOG
if exist PERF.LOG del PERF.LOG
if exist ASSIST.LOG del ASSIST.LOG
if exist ASTRESS.LOG del ASTRESS.LOG
if exist ASSISTC.LOG del ASSISTC.LOG

echo HW_CAPTURE_BEGIN>HWVALID.LOG
Expand Down Expand Up @@ -46,6 +47,9 @@ goto done
echo Running ASSIST.EXE --scripted...
echo HW_STEP^|assistant>>HWVALID.LOG
ASSIST.EXE --scripted > ASSIST.LOG
echo Running ASSIST.EXE --stress-probe...
echo HW_STEP^|assistant_stress>>HWVALID.LOG
ASSIST.EXE --stress-probe > ASTRESS.LOG
goto done

:missing_exe
Expand All @@ -62,5 +66,5 @@ goto done
echo HW_CAPTURE_END>>HWVALID.LOG
echo.
echo Hardware validation capture complete.
echo Copy HWVALID.LOG, QUAL.LOG, PERF.LOG, ASSIST.LOG, ASSISTC.LOG,
echo Copy HWVALID.LOG, QUAL.LOG, PERF.LOG, ASSIST.LOG, ASTRESS.LOG, ASSISTC.LOG,
echo and HWNOTES.TXT back to the host for verification.
8 changes: 8 additions & 0 deletions hardware/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@ C:\GPT2\MODEL\GPT2FX.BIN
C:\GPT2\MODEL\GPT2EXP.BIN
C:\GPT2\MODEL\VOCAB.BIN
C:\GPT2\PACKS\PACKS.TXT
C:\GPT2\PACKS\CHAT\...
C:\GPT2\PACKS\DOSHELP\...
C:\GPT2\PACKS\OFFICE\...
C:\GPT2\PACKS\DEV\...
C:\GPT2\PACKS\PORTABLE\...
C:\GPT2\GPT2SRC\ASSIST.BAS
C:\GPT2\HWVALID.BAT
C:\GPT2\HWNOTES.TXT
Expand All @@ -43,9 +46,14 @@ HWVALID.LOG
QUAL.LOG
PERF.LOG
ASSIST.LOG
ASTRESS.LOG
ASSISTC.LOG
```

`ASSIST.LOG` is the five-pack scripted assistant proof. `ASTRESS.LOG` is the
50-reply stress probe for CHAT, DOSHELP, OFFICE, DEV, and PORTABLE on the same
machine.

Fill in `HWNOTES.TXT` with CPU, clock, RAM, DOS version, storage, cache/turbo
state, FreeBASIC version, and any setup notes.
The `Machine key:` value must match the host `--machine-key` argument used when
Expand Down
Loading