docs: ship custom skill.md to fix autogenerator inaccuracies#3
Merged
Conversation
The Mintlify autogenerator keeps producing skill.md content that
contradicts the source docs — RESTful priors over source content. After
two rounds of strengthening the source pages (adding explicit Notes,
restructuring sections), the same inaccuracies still appear in the
generated /skill.md:
- Revoke documented as POST .../{key_id}/revoke (actual:
DELETE .../{key_id})
- Webhook rotate documented without the /org/ segment
- "Same as API key" storage analogy applied to the webhook secret
(which is stored plaintext, not hashed)
- completed_with_partial surfaced as if active
- Revoke described as admin-only
- Confidence triggers condensed inaccurately
- Usage endpoints omitted
Per Mintlify's spec, a skill.md at the project root overrides
auto-generation. This commit ships that override, written to the
agentskills.io v0 spec (frontmatter + body). Content is grounded in
verified facts from sherlock/src:
- api/routes_auth.py:254 → /api/v2/org/webhook-secret/rotate
- api/routes_auth.py:314 → DELETE /api/v2/org/api-keys/{key_id}
- api/routes_auth.py:318-321,337-338 → revoke role rules
- db/mongodb.py:513 → webhook_secret stored plaintext
- services/aggregator.py:499-530 → confidence triggers
- services/aggregator.py:354 → score band thresholds
Also adds a brief "Organization identifier" section to
authentication.mdx noting that org_id is opaque (no fixed alphabet) —
fixes the org_<hex> hallucination in the previous skill.md.
The override file is 306 lines (well under the 500-line guidance from
the agentskills.io spec).
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Usage / analytics endpoints are dashboard-only — they are not part of the public API contract. Removing them from the published docs so external customers don't build integrations against them. - Deleted the four api/usage/*.mdx pages (totals, scans, priority, warmer-health) and the empty parent directory. - Removed the Usage group from docs.json navigation. - skill.md: dropped the Usage table; added an explicit note that usage data is dashboard-only so agents stop hallucinating GET paths. - dashboard-overview.mdx: the Usage sidebar row now reads "Dashboard-only" instead of linking to API pages. Removed the "Get usage totals" link from the Home row and reworded the role-gating table. - api/errors.mdx: dropped "or usage endpoints" from the Org-settings-not-found row. Cross-repo follow-up: the corresponding endpoints in sherlock/src/api/routes_auth.py (get_usage, get_usage_scans, get_usage_priority, warmer-health) should be gated to dashboard sessions only — API-key auth on these paths should return 403, to match the new docs.
User intent: nothing in customer-facing docs should help a reader
reverse-engineer Tumban's detection pipeline. The previous pass
exposed too much — strategy names, judge model, bump_up/bump_down
adjudication, internal score arithmetic.
Scoring (concepts/recommendations.mdx)
- Removed the "three independent detection strategies" paragraph and
the bump_up/bump_down adjudication detail.
- Replaced with a contract-only explanation: what the score means
("confidence a violation is present"), what fields are part of the
public contract, and that other fields are opaque and may change.
Confidence (concepts/confidence.mdx)
- Removed the exact trigger table that listed blocklist thresholds,
multi-strategy agreement, contextual-model confidence pass-through,
and judge invocation.
- Replaced with a semantic table (what high/medium/low mean to a
reviewer). Kept the "never low when no_flags" gotcha — it's a
contract-level invariant, not mechanism.
Field surface (api/scans/get.mdx)
- Dropped the ResponseField blocks for strategy_scores and
judge_model_invoked, removed them from the example JSON, and
removed the "Strategy scores" bullet from the dashboard section.
Folded them into the existing "internal, opaque, subject to change"
disclaimer (which I also generalised — no longer enumerates the
specific internal fields).
Reason codes (reference/reason-codes.mdx)
- Reworded descriptions that explicitly named the judge model or the
"contextual model" / classifier internals. Codes themselves
(JUDGE_BUMP_UP, LLM_API_ERROR, etc.) are left literal because the
API emits them verbatim — cross-repo follow-up below.
- Stripped the source-trail MDX comment that cited
aggregator.py:_aggregate_reason_codes and strategies/llm_decision.py.
Cross-doc cleanup
- evidence-index, scans/get, webhooks/payload: replaced
"the contextual model cited" wording with "Tumban cited".
- index.mdx: removed "multi-strategy detection pipeline".
- skill.md: rewrote Confidence and Score-band sections to match the
scrubbed concepts pages; emphasises that score internals are not
part of the public contract.
Cross-repo follow-ups for sherlock
- API still returns strategy_scores and judge_model_invoked on
GET /api/v2/scans/{scan_id}. Remove from the response model so
customers can't inspect detection internals via the field surface.
- Reason codes JUDGE_BUMP_UP, JUDGE_BUMP_DOWN, LLM_API_ERROR,
LLM_PARSE_ERROR are emitted with internal terminology baked into
the enum strings. Consider renaming (e.g. SCORE_ADJUSTED_UP,
ANALYSIS_ERROR) so the public reason_codes list doesn't leak the
same mechanics this PR scrubbed from prose.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Mintlify autogenerator keeps emitting
/skill.mdcontent that contradicts the source docs — it's letting REST priors override the actual page content. After two rounds of strengthening the source pages (explicit `` callouts, restructured sections, sweeping prose), the same inaccuracies still appear in the generated skill:Per Mintlify's spec, a `skill.md` at the project root overrides auto-generation. This PR ships that override, plus one small source-doc fix.
What's in this PR
1. `skill.md` at repo root (306 lines, agentskills.io v0 spec). Grounded in verified facts from `sherlock/src`:
Each "summariser commonly gets wrong" fact has its own emphatic section so a future re-summarise can't drop it.
2. `authentication.mdx` — new "Organization identifier" section noting that `org_id` is an opaque case-sensitive string. Fixes the `org_` hallucination in the autogen.
After this lands
https://docs.tumban.com/skill.md. Verify all 7 audit items disappear from the served file.Test plan
https://docs.tumban.com/skill.mdand confirm it matches this PR's content (not the stale autogenerated version).sherlock/srcline cited in the table above.