Skip to content

feat: add experimental coverage command - #2993

Open
ariesclark wants to merge 18 commits into
Redocly:mainfrom
ariesclark:feat/coverage-command
Open

feat: add experimental coverage command#2993
ariesclark wants to merge 18 commits into
Redocly:mainfrom
ariesclark:feat/coverage-command

Conversation

@ariesclark

@ariesclark ariesclark commented Jul 30, 2026

Copy link
Copy Markdown

What/Why/How?

Note

This was heavily assisted by Anthropic's Opus 5. I've reviewed the code to the best of my ability, but if there's any obvious issues I didn't catch, let me know.

drift reports where traffic and the description disagree, but says nothing about the parts no traffic reached. A clean drift run means only that nothing was wrong in the portion the traffic covered. coverage measures that portion.

The command parses a traffic log, matches each exchange to a documented operation through drift's matcher, walks each body against the schema that describes it, and reports what nothing reached: operations, properties, oneOf/anyOf branches, and component schemas.

It adds no runtime dependencies. Spec loading reuses @redocly/openapi-core; traffic parsing, operation matching, and mime selection come from drift.

It bundles the description a second time with dereference: false. drift's loader dereferences, deep-cloning every $ref target, so nothing under paths shares identity with components.schemas. Without that identity a value cannot be traced back to the component it came from, and coverage is reported per component.

Reference

Stacked on #2992, #2994. This branch includes that commit until it merges.

Testing

Unit tests cover the engine: $ref resolution, declared properties, union-site enumeration, branch matching, walking, and summarizing. E2E runs the built CLI over a HAR fixture in stylish, JSON, --all, and --schema modes, plus the missing-traffic-path case. The fixture composes User with allOf so inherited properties are credited to the schema that declares them.

Check yourself

  • This PR follows the contributing guide
  • All new/updated code is covered by tests
  • Core code changed? - Tested with other Redocly products (internal contributions only)
  • New package installed? - Tested in different environments (browser/node)
  • Documentation update has been considered

Security

  • The security impact of the change has been considered
  • Code follows company security practices and guidelines

The command reads traffic logs and a description, both already handled as untrusted input by drift, and writes the report only to the path given by --output.


Note

Medium Risk
Large new experimental command reuses drift matching and shared path-regex logic, so regressions could affect drift routing; behavior is heavily tested but the feature surface (schema walking, unions, dual spec load) is non-trivial.

Overview
Introduces the experimental coverage command: it ingests the same traffic formats as drift, matches exchanges via drift’s matcher, walks request/response bodies against documented schemas (with dereference: false bundling so coverage can attribute hits to component schemas), and reports unexercised operations, parameters, responses, properties, union branches, and schemas in stylish or JSON output (--schema, --all, --output).

drift path compilation now supports segments that mix literals and multiple parameters (e.g. /instances/{worldId}:{instanceId}), with scoring so more specific templates win—fixing missed matches for both drift and coverage.

respect --har-output now populates HAR postData (and bodySize) for string/URLSearchParams bodies via buildPostData, so replays can validate request bodies.

Docs, changesets, unit/e2e tests, and a root /coverage/ .gitignore entry (so Vitest reports don’t hide the command source tree) round out the change; drift loader helpers are exported for coverage reuse.

Reviewed by Cursor Bugbot for commit 9f52750. Bugbot is set up for automated code reviews on this repo. Configure here.

@ariesclark
ariesclark requested review from a team as code owners July 30, 2026 16:28
@changeset-bot

changeset-bot Bot commented Jul 30, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 5990dea

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@redocly/cli Minor
@redocly/openapi-core Minor
@redocly/respect-core Minor
@redocly/client-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@ariesclark ariesclark changed the title Feat/coverage command feat: add experimental coverage command Jul 30, 2026
Comment thread packages/cli/src/commands/coverage/engine/analyse.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/walk.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
Comment thread packages/cli/src/commands/coverage/index.ts
Comment thread packages/cli/src/commands/coverage/index.ts
Comment thread packages/cli/src/commands/coverage/index.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/walk.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/walk.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
Comment thread packages/cli/src/commands/coverage/engine/sites.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
@ariesclark

Copy link
Copy Markdown
Author

It really is endless. 😓

Comment thread packages/cli/src/commands/coverage/engine/analyse.ts
Comment thread packages/cli/src/commands/coverage/reporter.ts
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit dc7f6b4. Configure here.

Comment thread packages/cli/src/commands/coverage/engine/schema.ts Outdated
Comment thread packages/cli/src/commands/coverage/engine/schema.ts
@vadyvas

vadyvas commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution
We'll review it next week and get back to you 🙂

Comment on lines +22 to +24
This is the opposite direction from [`drift`](./drift.md).
`drift` judges the traffic against the description and reports what disagrees; it is silent about a description that is never put to the test.
A `drift` run with no findings is only as meaningful as the share of the description the traffic actually covered, and that share is what `coverage` measures.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants