Skip to content

feat: show config source and recent tool calls through tool output - #266

Merged
docdyhr merged 3 commits into
mainfrom
feat/diagnostics-in-tool-output
Oct 9, 2026
Merged

docdyhr merged 3 commits into
mainfrom
feat/diagnostics-in-tool-output

Conversation

@docdyhr

@docdyhr docdyhr commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Pull Request

Description

Claude Desktop does not capture an installed extension's stderr. Checked across ~/Library/Logs/Claude/mcp-server-*.log: servers configured by hand in claude_desktop_config.json have thousands of raw stderr lines, while every installed extension (Bear Notes, Filesystem, Control Chrome, Word, PDF Tools, this one) has none. So extra logger.* output can never tell a user which config file is live or which tool call failed. This exposes both through tool output instead (open items from the v4.0.3 debug test results: "log has no tool names", "which config loaded").

Type of Change

  • ✨ New feature (non-breaking change which adds functionality)

Changes Made

  • ServerConfiguration.getLoadInfo() records how the last load configured the server: mode, loaded file and its source, site IDs, and (single-site) the paths that were searched
  • wp_test_auth reports it on success and in the failure message (bad credentials, unreachable site, timeout keep their typed error): **Configuration:** multi-site, loaded from <path> (<source>) + **Configured sites:** …, or single-site (environment variables | client-supplied configuration); no multi-site config file found in: …. The install-folder source carries a warning that an extension update replaces that folder. Omitted if nothing was recorded.
  • PerformanceMonitor keeps the last 50 tool calls in memory (getRecentToolCalls()): tool, site, status, duration and, for a failure, HTTP status, error code and the error's type (class name). Call parameters and error text are never stored — messages can quote the arguments (validation errors, upload paths). Failures that don't throw carry a fixed code: tool_reported_failure, site_required, site_not_found.
  • wp_performance_stats (category=tools) returns them as recentCalls, newest first (10 in summary format, 50 in detailed)
  • ToolRegistry passes the thrown error and the resolved site to the tracker; MetricsCollector.endToolExecution(id, ok, error?, siteId?) forwards them
  • Tool descriptions updated; docs/api regenerated; src/config/AGENTS.md and src/performance/AGENTS.md updated
  • The existing tracker assertions in ToolRegistry/MetricsCollector tests changed to the new argument lists; ~25 new tests

Testing

  • Unit tests pass (npm test): 87 files / 2711 tests (after merging main and addressing review); the new tests were red first
  • TypeScript compilation, lint, format:check, markdownlint (the first commit attempt failed on an MD013 line length in src/performance/AGENTS.md; rewrapped, hook not bypassed)
  • Manual: built server over stdio with a two-site config in a temp HOME and a local WordPress stub. wp_test_auth printed the config path, source and site1, site2; after a 403 on wp_get_site_settings, wp_performance_stats listed it with statusCode: 403, errorCode: "rest_forbidden".

Notes for reviewers

  • feat: so semantic-release will cut a minor release (4.1.0).
  • In-memory only: nothing is written to disk and the list resets when the server restarts.
  • Not tested against Claude Desktop itself.

Documentation

  • Documentation updated
  • CHANGELOG.md updated — generated by semantic-release

Security

  • No credentials in the output; error text and call parameters are not recorded

🤖 Generated with Claude Code

Summary by Sourcery

Expose configuration provenance and recent tool-call outcomes through tool responses so installed extensions can provide actionable diagnostics without relying on host-captured logs.

New Features:

  • Expose the active configuration source and configured sites through wp_test_auth, including single-site fallback diagnostics.
  • Add in-memory recent tool-call diagnostics to wp_performance_stats, including tool, site, status, duration, and safe failure metadata.

Enhancements:

  • Propagate resolved sites and failure context through tool execution metrics while avoiding storage of call parameters and error messages.
  • Update generated API documentation and contributor guidance for the new diagnostics.

Documentation:

  • Document configuration reporting in wp_test_auth and recent-call reporting in wp_performance_stats.

Tests:

  • Add coverage for configuration load reporting, recent-call retention and ordering, failure metadata sanitization, and tool tracking propagation.

Claude Desktop does not capture an installed extension's stderr (every
extension log under ~/Library/Logs/Claude has zero server output, while
hand-configured servers have thousands of lines), so log lines cannot tell a
user which config file is live or which tool call failed. Expose both through
tools instead:

- wp_test_auth reports the configuration in use: the multi-site file that was
  loaded, where it came from (with a warning for the install folder, which an
  extension update replaces) and the site IDs, or the single-site fallback and
  the paths that were searched. ServerConfiguration records this in
  getLoadInfo().
- wp_performance_stats (category=tools) lists the last 50 tool calls, newest
  first: tool, site, status, duration and, for a failure, the HTTP status,
  error code and a truncated message. Call parameters are never stored. The
  registry now passes the thrown error and the resolved site to the tracker.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 07:32
@sourcery-ai

sourcery-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Sorry @docdyhr, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days and 4 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T07:35:19.493766Z ce7ed05 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the docs Documentation changes label Oct 9, 2026
@sourcery-ai

sourcery-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds tool-output diagnostics for installed extensions that lack host stderr capture: wp_test_auth reports configuration provenance, while wp_performance_stats exposes bounded, parameter-free recent tool-call history with resolved sites and failure metadata. Execution tracking is extended to propagate the thrown error and resolved site, with documentation and tests updated.

Sequence diagram for tool-output diagnostics

sequenceDiagram
    participant User
    participant ToolRegistry
    participant MetricsCollector
    participant PerformanceMonitor
    participant PerformanceTools

    User->>ToolRegistry: execute tool
    ToolRegistry->>MetricsCollector: startToolExecution(toolName, parameters, siteId)
    ToolRegistry->>ToolRegistry: resolve site and run tool
    alt tool succeeds or fails
        ToolRegistry->>MetricsCollector: endToolExecution(executionId, success, failure, siteId)
        MetricsCollector->>PerformanceMonitor: recordToolCall(toolName, responseTime, success, details)
        PerformanceMonitor->>PerformanceMonitor: retain latest 50 calls without parameters
    end
    User->>PerformanceTools: wp_performance_stats(category=tools)
    PerformanceTools->>MetricsCollector: getRecentToolCalls(limit)
    MetricsCollector->>PerformanceMonitor: getRecentToolCalls(limit)
    PerformanceMonitor-->>PerformanceTools: newest-first call records
    PerformanceTools-->>User: recentCalls with status and failure metadata
Loading

Sequence diagram for configuration provenance in wp_test_auth

sequenceDiagram
    participant User
    participant AuthTools
    participant ServerConfiguration

    User->>AuthTools: wp_test_auth
    AuthTools->>ServerConfiguration: getLoadInfo()
    ServerConfiguration-->>AuthTools: mode, configPath, source, siteIds, searched
    AuthTools-->>User: authentication result with Configuration and Configured sites
    alt single-site configuration
        AuthTools-->>User: environment-variable mode and searched paths
    end
Loading

File-Level Changes

Change Details Files
Expose configuration provenance through authentication output.
  • Record the last load mode, file/source, site IDs, and searched fallback paths.
  • Append multi-site or single-site configuration diagnostics to wp_test_auth, including an install-folder update warning.
  • Document the new diagnostic behavior.
src/config/ServerConfiguration.ts
src/tools/auth.ts
docs/api/tools/wp_test_auth.md
docs/api/README.md
src/config/AGENTS.md
docs/api/openapi.json
docs/api/summary.json
tests/config/ServerConfiguration.test.js
tests/tools/auth.test.js
Add bounded in-memory history of recent tool executions.
  • Store the newest 50 calls with tool, resolved site, status, duration, and failure status/code/message while excluding parameters.
  • Expose newest-first records through PerformanceMonitor and MetricsCollector.
  • Return 10 summary or 50 detailed records from wp_performance_stats under recentCalls, with ISO timestamps.
  • Document the diagnostics output and retention/security constraints.
src/performance/PerformanceMonitor.ts
src/performance/MetricsCollector.ts
src/tools/performance/PerformanceTools.ts
docs/api/tools/wp_performance_stats.md
src/performance/AGENTS.md
tests/performance/PerformanceMonitor.test.js
tests/performance/MetricsCollector.test.js
tests/server/ToolRegistry.test.js
Propagate execution failure context and the actually resolved site into tool-call metrics.
  • Extend the tracker completion API with optional error and site ID arguments.
  • Capture thrown errors in the registry and forward them during finalization, including failures from site resolution or handlers.
  • Update tracker assertions and add coverage for failure metadata and recent-call output.
src/server/ToolRegistry.ts
src/performance/MetricsCollector.ts
tests/server/ToolRegistry.test.js
tests/performance/MetricsCollector.test.js

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce7ed055a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/tools/auth.ts Outdated
Comment thread src/server/ToolRegistry.ts
Comment thread src/config/ServerConfiguration.ts Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failure paths omit configuration diagnostics, and retained error messages can indirectly expose call parameters.

2 open findings
What changed in this PR

Exposes configuration-source and recent tool-call diagnostics through MCP output.

Changes:

  • Tracks configuration load details and reports them via wp_test_auth.
  • Retains and exposes the 50 most recent tool calls.
  • Adds tests and regenerates API documentation.
File Description
src/​config/​ServerConfiguration.ts Records configuration load information.
src/​config/​AGENTS.md Documents configuration diagnostics.
src/​performance/​PerformanceMonitor.ts Retains recent tool-call records.
src/​performance/​MetricsCollector.ts Forwards failure and site metadata.
src/​performance/​AGENTS.md Documents recent-call tracking.
src/​server/​ToolRegistry.ts Passes resolved sites and errors to tracking.
src/​tools/​auth.ts Adds configuration details to successful auth output.
src/​tools/​performance/​PerformanceTools.ts Exposes recent calls in performance statistics.
tests/​config/​ServerConfiguration.test.js Tests load diagnostics.
tests/​performance/​MetricsCollector.test.js Tests metadata forwarding.
tests/​performance/​PerformanceMonitor.test.js Tests retention, ordering, and truncation.
tests/​server/​ToolRegistry.test.js Tests registry tracking integration.
tests/​tools/​auth.test.js Tests successful configuration reporting.
docs/​api/​tools/​wp_test_auth.md Documents configuration reporting.
docs/​api/​tools/​wp_performance_stats.md Documents recent-call diagnostics.
docs/​api/​openapi.json Regenerates OpenAPI descriptions.
docs/​api/​README.md Refreshes generated tool documentation.
docs/​api/​summary.json Updates generation metadata.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/performance/PerformanceMonitor.ts Outdated
Comment thread src/tools/auth.ts Outdated
docdyhr and others added 2 commits October 9, 2026 10:22
Resolves the MetricsCollector conflict with #267 (keeps getRecentToolCalls and
getActiveAlerts and both type imports) and the generated docs timestamp.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- wp_test_auth only reported the configuration on success, but a failed test
  (bad credentials, unreachable site, timeout) is exactly when a user needs to
  know which configuration is live. The failure message now carries it; the
  typed error (status, code) is preserved.
- Client-supplied configuration (an MCP client's mcpConfig, which takes
  precedence over the environment per field) was reported as "environment
  variables". getLoadInfo() now records singleSiteSource.
- Recent calls no longer retain any error text: validation and upload errors
  quote the offending argument or path, so truncation did not honour "call
  parameters are never recorded". Only the HTTP status, error code and error
  type (class name) are kept.
- Failures that do not throw (a result with success:false / status
  "unavailable", a missing or unknown site) now record a fixed reason code
  (tool_reported_failure, site_required, site_not_found) instead of nothing,
  again without echoing the arguments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@docdyhr
docdyhr merged commit bd744ba into main Oct 9, 2026
34 checks passed
@docdyhr
docdyhr deleted the feat/diagnostics-in-tool-output branch October 9, 2026 08:58
docdyhr pushed a commit that referenced this pull request Oct 9, 2026
## [4.1.0](v4.0.11...v4.1.0) (2026-10-09)

### 🚀 Features

* show config source and recent tool calls through tool output ([#266](#266)) ([bd744ba](bd744ba))
@docdyhr

docdyhr commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

🎉 This PR is included in version 4.1.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation changes released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants