Skip to content

automate_observation: add --events for machine-readable progress #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
billy-lau merged 1 commit into from
Sep 29, 2026
Merged

automate_observation: add --events for machine-readable progress #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions uraniborg/docs/automate_observation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,167 @@ A requested serial that is not connected is not silently skipped: it is logged
as an error, reported as `FAILED` in the final summary, and makes the script
exit with code `1`. The remaining requested devices are still observed.

### Machine-Readable Progress (`--events`)

Log messages are meant for people and may change between versions. Wrappers
such as CI jobs or device-lab scripts should use `--events` instead: it writes
a stream of progress events as [JSON Lines](https://jsonlines.org/), one JSON
object per line, flushed as each event happens.

```bash
# Write events to a file (overwritten if it exists):
python3 automate_observation.py --events run.jsonl

# Or stream them on stdout:
python3 automate_observation.py --events - | my-wrapper
```

With `--events -`, stdout carries **only** events. Everything else that would
otherwise have gone to stdout is sent to stderr, including output from child
processes such as the verifier and the Xiaomi manual-install prompt. Logging
always goes to stderr. Without `--events`, nothing changes: log output, exit
codes and the results layout are the same. If the `--events` destination
cannot be opened, the script exits with code `1` before doing anything else.
If writing events fails partway through (for example, the reading process
exits), a warning is logged and the run continues without events.

#### Event schema (version 1)

Every event has these fields:

| Field | Meaning |
| :--- | :--- |
| `v` | Schema version, currently `1` |
| `ts` | UTC timestamp, ISO 8601 with milliseconds, e.g. `2026-01-02T03:04:05.678Z` |
| `type` | One of the event types below |

Optional fields are left out rather than set to `null`.

| `type` | Fields | When |
| :--- | :--- | :--- |
| `run_started` | `argv` (arguments, without the script name), `pid` | First event of every run |
| `step` | `step`, `state` (`started` / `finished` / `failed`), `device`\*, `duration_ms`\*\*, `message`\* | Around each phase (see below) |
| `devices` | `devices` (list of `{serial, unauthorized, model?, product?, device?}`), `selected` (serials to observe), `missing` (requested via `--serial` but not connected) | Once, after listing devices |
| `device_started` | `device` | Before processing each selected device |
| `device_finished` | `device`, `status`, `results_dir`\*, `error`\* | After each selected device, and once for each missing serial |
| `run_finished` | `exit_code`, `ok`, `summary`, `error`\* | Last event of a run that ends normally, exits early, raises, or is stopped by Ctrl-C or `SIGTERM` (see [End of stream](#end-of-stream)) |

\* only when relevant. \*\* only on `finished` / `failed`.

Both `error` fields are `{reason, message}`: `reason` is a stable code (see
[Error reasons](#error-reasons)) to match on, and `message` is human-readable
text that may change. Step `message` is free text.

**Steps.** Run-level steps have no `device` field and run in this order:
`build_hubble` (only without `-H`), `verify_hubble`, `check_adb`,
`start_adb_server`, `list_devices`. Per-device steps carry `device`:
`uninstall_previous` (only if Hubble was already installed), `install_hubble`,
`launch_hubble`, `wait_for_results`, `extract_results`, `extract_selinux`, and,
with `--perform_inclusion_proof_check`, `inclusion_proof_prefetch` (only when
pre-fetching is attempted) and `inclusion_proof_check`. A failed
`inclusion_proof_prefetch` is not fatal: verification falls back to fetching
entries on demand.

**Device status.** `device_finished.status` and `summary[serial].status` use the
same four outcomes as the final log summary:

| `status` | Meaning |
| :--- | :--- |
| `success` | Results collected, and the inclusion proof check (if requested) completed |
| `partial_check_incomplete` | Results collected, but the inclusion proof check could not complete (for example, the package list was missing or unreadable, or the output could not be written) |
| `partial_error` | Results collected, but an unexpected error occurred afterwards, or the run was stopped (`error.reason` is `interrupted` or `terminated`) |
| `failed` | No results collected (including unauthorized and not-connected devices, and devices stopped before results were collected) |

> [!IMPORTANT]
> `success` does **not** mean every APK was found in the transparency log. A
> completed check records a per-split `inclusion_proof_verified` result, which
> may be `false`, in `packages_with_inclusion_proof_signal.txt` (or
> `preinstalled_packages_with_inclusion_proof_signal.txt`) inside
> `results_dir`. Read that file to decide whether the APKs are in the log.

`results_dir` is present whenever results were collected, including both
`partial_*` outcomes.

**Run outcome.** `run_finished.summary` maps each serial, in the order it was
requested (or discovered, without `--serial`), to `{status, results_dir?}`.
If the run is stopped early (`interrupted`, `terminated` or
`unexpected_error`), `summary` lists only the devices that reached
`device_finished`; devices that had not been processed yet are absent.
`ok` is `true` only when `exit_code` is `0` **and** there is no `error`. Treat
`ok`, not `exit_code`, as the success signal: when the script stops before
processing any device (for example, no device is connected), it still exits
with code `0`, but `run_finished.error` explains why.

#### Error reasons

`run_finished.error.reason`:

| `reason` | Meaning |
| :--- | :--- |
| `unsupported_platform` | The host OS is not supported |
| `android_sdk_not_found` | Rebuilding Hubble (no `-H`) failed: Android SDK not found |
| `hubble_build_failed` | Rebuilding Hubble failed (Gradle or symlink error) |
| `invalid_hubble_apk` | The `-H` path is not an `.apk` |
| `adb_not_found` | `adb` is not installed |
| `adb_server_failed` | The adb server could not be started |
| `adb_devices_failed` | `adb devices` failed (without `--serial`) |
| `no_devices` | No device is connected (without `--serial`) |
| `unexpected_error` | An unhandled exception; `exit_code` is `1` |
| `interrupted` | Interrupted with Ctrl-C; `exit_code` is `130` |
| `terminated` | Stopped by `SIGTERM`; `exit_code` is `143` |

`device_finished.error.reason`:

| `reason` | Meaning | `status` |
| :--- | :--- | :--- |
| `not_connected` | Requested via `--serial` but not connected | `failed` |
| `unauthorized` | ADB is not authorized on the device | `failed` |
| `uninstall_failed` | A previous Hubble installation could not be removed | `failed` |
| `install_failed` | Hubble could not be installed | `failed` |
| `launch_failed` | Hubble could not be launched | `failed` |
| `no_results` | Hubble produced no results in time | `failed` |
| `extract_failed` | Results could not be pulled from the device | `failed` |
| `inclusion_proof_check_incomplete` | The inclusion proof check could not complete | `partial_check_incomplete` |
| `unexpected_error` | An unhandled exception while processing the device | `failed` or `partial_error` |
| `interrupted` | Ctrl-C while processing the device | `failed` or `partial_error` |
| `terminated` | `SIGTERM` while processing the device | `failed` or `partial_error` |

For the last three, `status` is `partial_error` if results had already been
collected, otherwise `failed`.

Example, for one device with inclusion proofs:

```json
{"v": 1, "ts": "2026-01-02T03:04:05.678Z", "type": "run_started", "argv": ["--events", "-", "--perform_inclusion_proof_check", "--verifier_path=/path/to/verifier"], "pid": 4242}
{"v": 1, "ts": "2026-01-02T03:04:05.690Z", "type": "step", "step": "verify_hubble", "state": "started"}
{"v": 1, "ts": "2026-01-02T03:04:05.691Z", "type": "step", "step": "verify_hubble", "state": "finished", "duration_ms": 1}
...
{"v": 1, "ts": "2026-01-02T03:04:06.100Z", "type": "devices", "devices": [{"serial": "ABCDEF012345", "unauthorized": false, "model": "Pixel_9"}], "selected": ["ABCDEF012345"], "missing": []}
{"v": 1, "ts": "2026-01-02T03:04:06.101Z", "type": "device_started", "device": "ABCDEF012345"}
{"v": 1, "ts": "2026-01-02T03:04:06.102Z", "type": "step", "step": "install_hubble", "state": "started", "device": "ABCDEF012345"}
...
{"v": 1, "ts": "2026-01-02T03:09:41.310Z", "type": "device_finished", "device": "ABCDEF012345", "status": "success", "results_dir": "/path/to/results/google/.../000"}
{"v": 1, "ts": "2026-01-02T03:09:41.312Z", "type": "run_finished", "exit_code": 0, "ok": true, "summary": {"ABCDEF012345": {"status": "success", "results_dir": "/path/to/results/google/.../000"}}}
```

#### End of stream

`run_finished` cannot be written if the process is killed with `SIGKILL`, the
Python interpreter crashes, or the host goes down. **A stream that ends without
`run_finished` means the run failed.** Do not infer success from the
`device_finished` events seen so far.

With `--events`, `SIGTERM` (how CI systems usually enforce timeouts and
cancellations) is caught so that `run_finished` can be written. The script then
still terminates by `SIGTERM`, so the parent process sees the same exit status
as before. Without `--events`, `SIGTERM` handling is unchanged.

#### Compatibility

The schema is a stable contract. New event types and new fields may be added
without changing `v`, so consumers should ignore types and fields they do not
recognize. Removing a field or changing what it means requires a new `v`.

## Performing Inclusion Proof Checks

You can automatically verify extracted package APK splits against Android Binary
Expand Down
Loading
Loading