Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/vibecad-workflow-harness.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: VibeCAD workflow harness

on:
pull_request:
push:
branches:
- main
workflow_dispatch:

permissions:
contents: read

concurrency:
group: vibecad-workflow-harness-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
harness:
name: Fake-channel workflow assertions
runs-on: ubuntu-24.04
timeout-minutes: 10
env:
PYTHONPATH: src/Mod/VibeCAD
steps:
- name: Check out source
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4

- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.11"

- name: Install pytest
run: python -m pip install --disable-pip-version-check pytest

- name: Run click-channel workflow tests without a TypeSafe key
run: python -m pytest -q src/Mod/VibeCAD/vibecad_tests/test_workflow_harness.py

- name: Run the fake-channel harness entry point
run: python src/Mod/VibeCAD/VibeCADWorkflowHarness.py --fake-channel
16 changes: 15 additions & 1 deletion docs/vibecad-agent-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,11 @@ curl -s -H "Authorization: Bearer %VIBECAD_TOKEN%" ^
-d "{\"kind\":\"ribbon\",\"text\":\"Aero\"}" ^
http://127.0.0.1:8766/v1/ui/click

curl -s -H "Authorization: Bearer %VIBECAD_TOKEN%" ^
-H "Content-Type: application/json" ^
-d "{\"kind\":\"command\",\"text\":\"VibeCADRibbonNew\"}" ^
http://127.0.0.1:8766/v1/ui/click

curl -s -H "Authorization: Bearer %VIBECAD_TOKEN%" ^
http://127.0.0.1:8766/v1/screenshot

Expand Down Expand Up @@ -267,6 +272,7 @@ python "%VIBECAD_CLI%" close
python "%VIBECAD_CLI%" ui-menus
python "%VIBECAD_CLI%" ui-ribbon
python "%VIBECAD_CLI%" ui-click --kind ribbon --text Aero
python "%VIBECAD_CLI%" ui-click --kind command --text VibeCADRibbonNew
python "%VIBECAD_CLI%" screenshot --path C:\Evidence\vibecad.png
python "%VIBECAD_CLI%" run --path C:\Models\part.FCStd --script C:\Work\edit.py
python "%VIBECAD_CLI%" run --python "result = [obj.Name for obj in App.ActiveDocument.Objects]"
Expand Down Expand Up @@ -321,7 +327,7 @@ owner's network controls.
| POST | `/v1/close` | optional `document`, explicit `discard_unsaved` | Close without silently discarding a modified document |
| GET | `/v1/ui/menus` | | Live top-level menu names, indices, visibility, and screen geometry |
| GET | `/v1/ui/ribbon` | | Live ribbon names, workbenches, indices, selection, and screen geometry |
| POST | `/v1/ui/click` | `{"kind":"menu|ribbon","text":"..."}`, optional exact PID/index | Activate one semantic Qt target without moving or clicking the OS cursor |
| POST | `/v1/ui/click` | `{"kind":"menu|ribbon|command","text":"..."}`, optional exact PID/index | Activate one semantic Qt target without moving or clicking the OS cursor |
| POST | `/v1/run` | `{"python":"..."}` or `{"script":"..."}` plus optional `path`, `recompute` | Exec against the active doc |
| GET | `/v1/operations/{operation_id}` | | Read the in-memory state/result of a client-identified operation without entering the document thread |
| GET/POST | `/v1/aero` | operation payload for POST | Bounded Aero context and operations |
Expand Down Expand Up @@ -409,6 +415,14 @@ window's visible, enabled top-level menus and enabled ribbon tabs. This keeps th
tester reusable across checkouts without assuming that an optional product
feature is installed.

`kind=command` is additive: it activates one named ribbon button or menu
action (`VibeCADRibbonNew`, `Sketcher_NewSketch`, `PartDesign_Pad`,
`Std_Export`, or the matching command id / tooltip) through the same
in-process click site. The workflow harness in
[vibecad-workflow-harness.md](vibecad-workflow-harness.md) uses that channel
for New document, Sketch then pad, and Export. The visible tour stays the
demo; it does not become the CI suite.

### Visible-window screenshots

`GET /v1/screenshot` captures the visible VibeCAD main window under the private
Expand Down
48 changes: 48 additions & 0 deletions docs/vibecad-workflow-harness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# VibeCAD click-channel workflow harness

`src/Mod/VibeCAD/VibeCADWorkflowHarness.py` drives three operator workflows
through the existing authenticated loopback click channel
(`POST /v1/ui/click` / `VibeCADAgentCli ui-click`). It does not move the OS
cursor and it does not replace
[`Invoke-VibeCAD-VisibleTour.ps1`](../Invoke-VibeCAD-VisibleTour.ps1). The
tour remains the human-watchable demo.

## Workflows

| Workflow | Clicks | Post-click checks |
| --- | --- | --- |
| `new_document` | `command` `VibeCADRibbonNew` (`Std_New`) | Semantic click verified; document count +1; active document present |
| `sketch_then_pad` | New document, `ribbon` `Model`, `Sketcher_NewSketch`, `PartDesign_Pad` | Model tab selected; sketch and pad objects present |
| `export` | New document, `command` `Std_Export` | Semantic click verified; active document still present |

Each step records the click payload plus document/object state. Code owns
timeout, click success, and pass/fail.

## Run

Live GUI (same agent home / token as the visible tour):

```sh
python src/Mod/VibeCAD/VibeCADWorkflowHarness.py --gui-only
```

Headless / CI (fake click channel, no display, no TypeSafe key):

```sh
python src/Mod/VibeCAD/VibeCADWorkflowHarness.py --fake-channel
python -m pytest -q src/Mod/VibeCAD/vibecad_tests/test_workflow_harness.py
```

Set `PYTHONPATH=src/Mod/VibeCAD` when running from the repository root.

## Optional Jev classification

After a step, one System One request may classify:

- `landed` (Noul): did the step actually complete
- `failure_class` (Choice): `geometry_bug`, `wrong_button`, `model_lied`, `flake`
- `progress` (Score): how far the workflow got

This path is default-off. A live call happens only when both
`VIBECAD_JEV_JUDGE=1` and `TYPESAFE_API_KEY` are set. Low confidence never
changes a green result. CI mocks or skips the judge. Do not commit a key.
3 changes: 3 additions & 0 deletions src/Mod/VibeCAD/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ set(VibeCAD_Scripts
VibeCADGeometryInspection.py
VibeCADPlacementInspection.py
VibeCADGui.py
VibeCADJevJudge.py
VibeCADWorkflowHarness.py
VibeCADTokenUsage.py
VibeCADLegacyArchitecture.py
VibeCADLiveAcceptanceOracle.py
Expand Down Expand Up @@ -1626,6 +1628,7 @@ if(BUILD_TEST AND BUILD_GUI)
vibecad_tests/test_agent_control_grok_bot.py
vibecad_tests/test_dev_launcher_contract.py
vibecad_tests/test_visible_operator_contract.py
vibecad_tests/test_workflow_harness.py
vibecad_tests/test_reference_image_downscale.py
vibecad_tests/test_ollama_inspect.py
vibecad_tests/test_live_acceptance_oracle.py
Expand Down
2 changes: 1 addition & 1 deletion src/Mod/VibeCAD/VibeCADAgentCli.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ def build_parser() -> argparse.ArgumentParser:
ui_click_parser.add_argument(
"--kind",
required=True,
choices=("ribbon", "menu"),
choices=("ribbon", "menu", "command"),
help="Target family to activate.",
)
ui_click_parser.add_argument("--text", required=True, help="Exact visible target text.")
Expand Down
Loading
Loading