feat(evals): add auth0 CLI B2B organization setup eval - #178
Draft
developerkunal wants to merge 8 commits into
Draft
feat(evals): add auth0 CLI B2B organization setup eval#178developerkunal wants to merge 8 commits into
developerkunal wants to merge 8 commits into
Conversation
Goal-only CLI eval that drives a full B2B organizations setup through the auth0 CLI: a protected API with scopes, RBAC roles, a regular-web app and an M2M app, an M2M client grant, two organizations each with an enabled connection, and an org-admin invitation. Grading combines event graders over the successful command trace with a trace-aware judge over the manifest the agent writes. Includes an optional harness.json and verify.js for runners that provision a live tenant; both are inert to the framework loader and build.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sanchitmehtagit
left a comment
sanchitmehtagit
left a comment
Contributor
There was a problem hiding this comment.
Nice eval! The live-tenant verify.js is a great addition — asserting the full B2B graph exists on the server (not just in the agent's manifest) closes a gap that the trace and file/judge graders can't cover on their own. The PROMPT/graders structure looks clean and follows the level conventions.
Just one small nit inline, and one non-blocking follow-up that's out of scope for this PR:
- The
ranCommandsInOrderprimitive this eval relies on (already onmain) passes vacuously if it's ever called with an emptystepsarray — worth a construction-time guard + test at some point, but nothing to change here.
|
|
||
| // Management API list endpoints answer either a bare array or a wrapped object | ||
| // (e.g. { clients: [...] }) depending on pagination params. Normalize both. | ||
| function asList(res, key) { |
Contributor
There was a problem hiding this comment.
Nit / totally optional: asList is already nicely defensive, and since api() throws on both a non-zero exit and non-JSON output, it only ever sees valid parsed JSON here — so this is safe as-is. If you ever want to make a malformed response easier to debug, distinguishing "key present but not an array" from "resource genuinely empty" could help, but definitely not worth holding the PR for. Just flagging in case it's useful later. 🙂
Move the live-tenant declaration for the B2B org setup eval from a colocated harness.json into PROMPT.md frontmatter (provision: auth0-tenant), alongside the existing skills field. The runner reads the frontmatter directly, so harness.json is removed and verify.js is auto-detected from the eval directory.
developerkunal
force-pushed
the
eval/auth0-cli-b2b-org-setup
branch
from
August 10, 2026 08:39
3d6a1b2 to
56d9d7d
Compare
The recommender now tells us what went wrong and where the fault lies, and the report ranks those findings so a repeat across models is visible at a glance. Recommendations - Diagnose every agent run, including the no-tool control. If correct work fails a check with no skill in context, the check is the suspect, and that was the one run we were skipping. Skill files are sent only when the skill was really in context, so a control run cannot invent "the skill should say X". - Each finding carries what_happened, what_should_have_happened, evidence, and a root_cause of skill / model / grader / environment. - Send the ordered run trace (commands, MCP calls, failures with error text) and the reference pool per file, walking references/ recursively. - A failed analysis comes back with its reason instead of an empty result, and the report shows that reason rather than "no recommendations for this run". Secrets - redactSecrets masks credential values as [REDACTED SECRET] in anything sent to a model: the recommender's trace and MCP args, and the command trace appended to an LLM judge. Values are replaced rather than lines dropped, so a security judge still sees that a secret occupied that position; Auth0 ids stay readable. - The B2B security judge now reads the marker as exposure. Report - New "What to fix" section folds every run's findings into one list, grouped by category, root cause, and normalised context, ranked by run count then severity. The count of runs that could not be analysed is stated, not hidden. - Runs that stopped at the turn limit keep their graders, trace, and metrics instead of collapsing to a bare error card. Graders - ranCommandOneOf takes args and accepts nested arrays as AND groups, so one grader can allow a subcommand or the raw auth0 api call while still requiring the command to name the resource. The B2B graders use it throughout. - judge() requires a real question, not just a stray question mark. Fixes - docker.test.ts cleaned up symlinks with rmSync, which throws EISDIR on a link to a directory. Two tests were failing before this. - Slimmed the B2B prompt to the goal and the artifact; anything about how to use the CLI is guidance the skill should be supplying.
…g-setup # Conflicts: # AGENTS.md
The Bearer/Basic rule matched the scheme word anywhere, and `Basic` is also a value of the CLI's --auth-method flag, so the trace the judge reads had the next flag replaced by the redaction marker and the judge called it a leaked secret. Require the Authorization header around the scheme. A bare bearer token is still caught by the JWT and opaque-token rules.
Judges were failing runs for things the runs did right. The command trace now states what the format does and does not show. Output is not captured, so an absent output is not evidence a command did nothing, and each command runs in its own shell, so an id re-declared as a literal later is one read from an earlier command rather than an invented one. A judge asked to confirm the end state had read both as the agent fabricating data. The security judge treated every [REDACTED SECRET] marker as a leak, including the ones the harness writes when masking a flag value on a resource-creation command. It now looks for the marker where a secret could actually escape. The manifest judge wanted resolved names in invitation.org and invitation.role while the prompt only showed an ellipsis, so an agent that copied the org_/rol_ ids as the neighbouring fields ask was marked wrong. The prompt now names the expected form.
The recommender only had somewhere to file a skill or grader problem, so a broken CLI command or a defect in the eval itself came out as a skill finding. Categories now cover the CLI, the eval, the docs, the MCP server, and agent efficiency, each gated on evidence from the run. The report renders findings inside each run's Recommendations panel instead of a combined block above the details, where a reader met every model's findings before seeing a single run and had to scroll back to work out which run a line came from. The panel leads with counts by severity and puts the fix location beside the labels. aggregateRecommendations still answers which problem repeats across runs; it clusters on the wording, since keying on context merged unrelated defects in one reference file into a single row. Findings worded entirely in short common words produced an empty signature, which matched nothing, so two runs reporting one verbatim each got their own row and both ranked as one-offs. Fall back to the whole text.
…a command A model reply that quotes a shell command as evidence can emit a ```bash fence before the ```json fence. Taking only the first fence in the reply handed the bash snippet to JSON.parse and dropped every finding in the run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an
auth0_cli_b2b_org_setupeval under a newclicategory that measures how well agents configure a complete B2B (organizations) setup using only theauth0CLI, against a tenant the CLI is already logged into. It complements the SDK-integration quickstarts with a pure tenant-configuration task.The prompt is goal-only and names no command or flag, so the agent has to work out the CLI surface itself. It asks for a protected API with
read:reports/write:reports/manage:members, two RBAC roles wired to those scopes, a Regular Web App and a Machine-to-Machine app, an M2M client grant between them, two organizations each with an enabled login connection, and an org-admin invitation, then asmoke-b2b-manifest.jsoncapturing every created resource.Grading combines event graders over the agent's successful command trace (
ranCommand/ranCommandOneOf/wroteFile) with a trace-aware judge (includeCommandTrace: true) that reads the manifest and cross-checks it against the commands actually run. The client grant and org enabled-connections have no public CLI subcommand, so those steps are matched by their Management API endpoint (client-grants,enabled_connections) rather than a dedicated command. A dedicated L3 judge fails the run if any client secret leaks into the manifest or the trace. Structure mirrors the existing evals: ascending L3/L4/L5 graders and a final levelless holistic judge.The eval also ships an optional
harness.jsonandverify.jsfor a runner that provisions a live Auth0 tenant. Both are inert to the framework here (the loader reads onlyPROMPT.mdandgraders.ts, and tsc compiles onlysrc/**/*.ts), so they do not affect build, lint, or discovery.Test plan
tsc --noEmitpasses for the auth0-evals appeslintpasses on the newgraders.tsprettier --checkclean on all new filesauth0_cli_b2b_org_setupunder categorycli