Repository navigation
test(executor-e2e): add create-github-issue and set-github-issue-type scenarios #1798
Description
Task: add create-github-issue + set-github-issue-type scenarios to executor-e2e
Add deterministic Stage 3 end-to-end coverage for the two GitHub issue safe
outputs in githubnext/ado-aw. Both currently ship with zero runtime
coverage — their only proof is a wiremock unit test.
Background
create-github-issue and set-github-issue-type were added by PR #1670
(merged, unreleased as of v0.48.0). That PR deliberately deferred E2E coverage
because the smoke suite was being reworked in parallel (PR #1791). The rework
has since landed and removed the smoke-failure-reporter smoke case, which
was the only thing exercising create-github-issue at runtime. Net effect:
coverage went from "thin" to "none".
executor-e2e excludes it by design today. tests/executor-e2e/README.md
says: "Excluded (out of scope or GitHub-only): the GitHub-only
create-issue." That exclusion predates the tools being first-class public
safe outputs, and is what you are reversing.
Confusable, do not be misled: scripts/ado-script/src/executor-e2e/github-issue.ts
is the harness's own failure reporter (it files an issue when scenarios
fail). It is NOT a scenario under test. You may reuse its REST plumbing, but
do not mistake it for existing coverage.
What to build
Add scripts/ado-script/src/executor-e2e/scenarios/github-issue.ts exporting
githubIssueScenarios, and register it in scenarios/index.ts (append to
allScenarios — the order is deterministic and existing entries must not move).
Cover, at minimum:
create-github-issue— files an issue, asserts it exists on GitHub with
the expected title (including the configuredtitle-prefix), body, and
config-injected labels.set-github-issue-type— sets the issue type on an existing issue.- The same-run temporary-ID handoff — this is the highest-value case and
the one with the weakest current proof.create-github-issueaccepts an
optionaltemporary_id, andset-github-issue-typeaccepts an
issue_numberthat may be either a real number or that temporary id. A
single executor run must resolve the handoff. A wrong REST shape or a
scoping failure here is invisible today.
Consider also (use judgement, do not gold-plate):
allowed-labelsrejection — config is default-deny for labels, unlike
set-github-issue-type.allowedwhich is default-allow. Use
expectedFailurefor the rejection path.- Clearing an issue type:
issue_type: ""is the documented clear operation.
The contract you must follow
Read scripts/ado-script/src/executor-e2e/scenario.ts first — it is the
authoritative contract and is well commented. Key points:
- A scenario is
{ tool, config, setup, ndjson, assert, cleanup }with an
optionalid(setidwhen several scenarios share onetool),
files,env, andexpectedFailure. ndjson()returns the executor entry without thenamefield — the
runner injectsname: <tool>.- No LLM is involved. You craft the NDJSON directly, run the real
ado-aw executebinary, then assert via REST. ScenarioContextgives youbuildId,prefix(tool)
(ado-aw-det-<buildId>-<tool>),workDir,log, andrest.- Throw
SkipErrorfromsetup()when a precondition is missing, so an
incomplete environment records as skipped, not failed. - Cleanup gotcha: if
setup()throws a non-SkipError,cleanup()is
NOT called — tear down anything you partially created before rethrowing.
Seescenarios/pr.tssetupPrfor the established pattern. assert()may populate state thatcleanup()needs (e.g. the created issue
number read off the executor result) — do that before any fallible check,
so a later assertion failure still leaves cleanup able to tear down.
scenarios/wiki.ts is the cleanest model: env-var override, REST discovery,
SkipError, deterministic naming, real cleanup.
Two things that make this genuinely different from every existing scenario
1. It asserts against GitHub, not ADO. ScenarioContext.rest is AdoRest
and will not help you. Every existing scenario asserts via ADO REST. You need
GitHub REST for setup/assert/cleanup. Reuse the helpers in
executor-e2e/github-issue.ts (findOpenIssueByTitle, createGitHubIssue,
diagnoseGitHubAuthFailure) rather than writing a fourth GitHub client — lift
them into a shared module if that is cleaner, but do not duplicate them.
2. GitHub issues cannot be deleted. Every other scenario tears its objects
down completely. You cannot. cleanup() must close the issue (and prefer a
[e2e]-style title prefix plus the ado-aw-det-<buildId>-… marker so leaked
issues are identifiable and greppable). Call this limitation out explicitly in
a comment and in the README — a reviewer will otherwise reasonably ask why
cleanup is weaker here. Consider whether the target repo should be a dedicated
scratch repo rather than one carrying real issues.
Environment
- Definition
2550(executor e2e) already hasEXECUTOR_E2E_GITHUB_TOKEN
provisioned — no new secret provisioning is required. Confirm the token's
scope actually permits issue-type mutation on the target repo; issue types
are an org-level construct and may need more than plain Issues write. If it
does not,SkipErroris the correct response, not a hard failure. - Existing env vars follow the
E2E_*convention (E2E_ISSUE_REPO,
E2E_WIKI_NAME,E2E_QUEUE_PIPELINE_ID, …). Follow it, and treat an
unexpanded ADO macro (literal$(NAME)) as unset —github-issue.tsalready
has that guard, reuse it. - Issue types are a closed set defined by the repository/org owner. Discover
the available types via REST rather than hardcoding a name, andSkipError
when none exist — a hardcoded type will rot.
Validation
cd scripts/ado-script
npm run typecheck
npx vitest run src/executor-e2e
npm run build:executor-e2eAdd unit coverage alongside the existing src/executor-e2e/__tests__/. Prove
the temporary-ID handoff with an assertion that genuinely fails if the
resolution breaks — mutation-check it by deliberately breaking the handoff and
confirming your test goes red. A test that passes whether or not the feature
works is worse than no test.
Then update tests/executor-e2e/README.md: remove create-issue from the
"Excluded (out of scope or GitHub-only)" line, document the new scenarios and
any new env vars, and state the close-not-delete cleanup limitation.
Out of scope
- Do not add these tools to the smoke suite (
tests/smoke/cases.json) — the
point of executor-e2e is deterministic Stage 3 coverage with no agent. - Do not add a GitHub PAT to the smoke
agenticlane. Issue test(smoke): report scheduled smoke failures as GitHub issues from the orchestrator #1796 covers
orchestrator-side failure reporting separately and puts the token on the
orchestrator, not the lane. - Do not change the Rust implementation of either tool. If you find a genuine
bug, report it rather than fixing it inline.
Definition of done
- Scenarios registered and passing against the real project
- Temporary-ID handoff proven, mutation-checked
- Cleanup closes every issue it creates, limitation documented
- Missing preconditions skip rather than fail
- README exclusion line corrected
npm run typecheck,npx vitest run src/executor-e2e,
npm run build:executor-e2eall green
Activity
docs(smoke): restore safe-output authoring guidance lost in the lock-…
feat(smoke): replace per-case pipelines with lane-based smoke suite (#…