Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/skills/standalone-test-repro-cmd/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
name: standalone-test-repro-cmd
description: >
Resolve a CoreCLR runtime test from a generated .cmd or .sh wrapper path for
the standalone-test-repro workflow. Use when the requested test is identified
by a CI or Helix wrapper path, with or without a named test scenario.
Continue with the parent standalone-test-repro skill after resolution.
---

# Resolve a Runtime Test Wrapper

This is the wrapper-path resolution sub-skill for
[`../standalone-test-repro/SKILL.md`](../standalone-test-repro/SKILL.md).
Resolve the exact source test, owning project, and effective invocation, then
continue with the parent's shared extraction steps.

## Input

Accept a generated `.cmd` or `.sh` path copied from a CI failure, for example:

```text
JIT\Directed\ConvertToInt\checked\ConvertToInt.cmd
```

The path may be absolute, relative to a Helix work item or runtime test output
root, and may use either directory separator. The surrounding request may also
name a scenario such as `jitstress1`; preserve it separately from the path for
the parent workflow.

## Resolution

1. Normalize the identifier:
- Trim quotes, Markdown formatting, leading test-result labels, and trailing
punctuation.
- Remove command-line arguments while retaining the path through `.cmd` or
`.sh`.
- Convert separators only for host filesystem operations.
- Do not treat a separately supplied scenario as part of the wrapper path.
2. If the wrapper exists locally, read it and record:
- The managed assembly or executable
- `corerun` arguments, including `-p` and `-e`
- Test arguments and expected exit code
- Environment variables
- Working-directory assumptions
- Copied data or native dependencies
3. Map an artifact or Helix path back to `src/tests/`:
- Strip prefixes through the runtime test output root.
- Remove configuration-only artifact path segments.
- Search the remaining relative directory for the wrapper stem, likely
`.csproj`, source file, and matching `AssemblyName`.
4. If the wrapper is unavailable, use the path and stem to search under
`src/tests/`; derive invocation settings from the project.
5. Inspect the owning project, applicable imports, and its explicit or evaluated
`Compile` items.
6. Confirm the assembly contains only the requested runnable test. If it
contains multiple independently runnable tests, stop and ask for a fully
qualified method name.

Do not assume a wrapper only runs `corerun Test.dll`. Its host options,
environment, arguments, setup, and working directory may be required to
reproduce the failure. When available, the generated wrapper is the ground
truth for the effective invocation.

Resolution is complete only when the exact source method, owning project, and
effective invocation are known. Return any requested scenario unchanged for
canonicalization by the `test-scenario-env` skill.
65 changes: 65 additions & 0 deletions .github/skills/standalone-test-repro-method/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
name: standalone-test-repro-method
description: >
Resolve a CoreCLR runtime test from a fully qualified method name for the
standalone-test-repro workflow. Use for xUnit display names and methods in
merged or multi-test assemblies, with or without a named test scenario.
Continue with the parent standalone-test-repro skill after resolution.
---

# Resolve a Runtime Test Method

This is the method-name resolution sub-skill for
[`../standalone-test-repro/SKILL.md`](../standalone-test-repro/SKILL.md).
Resolve the exact source test, owning project, and test invocation, then
continue with the parent's shared extraction steps.

## Input

Accept a fully qualified method name copied from a CI failure, for example:

```text
Namespace.TypeName.MethodName
```

It may include an assembly prefix, xUnit display-name arguments, quotes,
parentheses, nested-type notation, or surrounding CI log text. The surrounding
request may also name a scenario such as `jitstress1`; preserve it separately
from the method identifier for the parent workflow.

## Resolution

1. Normalize the identifier:
- Trim quotes, Markdown formatting, leading result labels, and trailing
punctuation.
- Preserve xUnit argument text for theory-case resolution.
- Remove an assembly prefix only after verifying it is an assembly name.
- Never reduce the identifier to only the final method segment.
- Do not treat a separately supplied scenario as part of the method name.
2. Parse the longest plausible namespace/type prefix and method name. Account
for nested types (`+` in reflection names and `.` in C# source) and generic
arity suffixes.
3. Search C# sources under `src/tests/` for the exact method declaration and
confirm its containing type and namespace. Prefer symbol information when
available; otherwise combine targeted searches for method, type, and
namespace.
4. Find the owning `.csproj` by its explicit or evaluated `Compile` items.
Account for wildcard includes and merged projects such as JIT regression
assemblies.
5. Inspect the project and source for:
- `[Fact]`, `[Theory]`, custom attributes, or `TestEntryPoint`
- Static or instance invocation and constructor requirements
- Fixtures, setup, cleanup, and disposal
- Async and generic return or argument handling
- Conditional compilation and platform guards
- Test data and the specific failing theory row
6. If display-name arguments identify a theory row, reproduce exactly that
row. If not, generate invocations for all locally resolvable rows and label
them. Use the `ask_user` tool when runtime-discovered data or ambiguity
prevents a faithful choice.
7. If multiple source methods still match, present the concrete candidates with
`ask_user`; do not guess.

Resolution is complete only when the exact source method, owning project, and
required invocation semantics are known. Return any requested scenario
unchanged for canonicalization by the `test-scenario-env` skill.
Loading