feat: add structured data and API interaction tools

openspec/changes/archive/2026-08-25-structured-data-api-tools/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-25

openspec/changes/archive/2026-08-25-structured-data-api-tools/design.md

@@ -0,0 +1,62 @@
+## Context
+
+The madz project currently provides tools for web search, content extraction, file operations, and email. There is no structured API interaction capability — the agent must fall back to shell commands (curl, jq, yq) or rely on ad-hoc LLM reasoning for REST API calls, GraphQL queries, JSON/YAML manipulation, and webhook management. This is inconsistent and error-prone for office and marketing workflows.
+
+The project uses a tool factory pattern: each tool is a plain async function with a Zod input schema, registered in `src/tools/index.js` with permission tiers. Tools live in `src/tools/` and tests mirror the structure in `tests/unit/`.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Provide structured, validated tools for REST API, GraphQL, JSON, YAML, data transformation, and webhook management
+- Enforce URL allowlist security per AGENTS.md §1.2 on all network tools
+- Follow existing tool pattern: Zod schema → impl function → registration
+- Include comprehensive unit and integration tests
+
+**Non-Goals:**
+- Subscription support for GraphQL (deferred)
+- Embedded webhook server (agent-facing tool only)
+- File I/O for JSON/YAML/CSV tools (in-memory operations only)
+- OAuth/OIDC authentication flows (Bearer, Basic, API Key only)
+- Response caching (considered but deferred)
+
+## Decisions
+
+1. **Single tool per capability**: Each capability (REST, GraphQL, JSON, YAML, data, webhook) gets its own file in `src/tools/`. This follows the existing pattern and keeps tools focused.
+
+2. **Native `fetch` API**: Use Node.js 24+ built-in `fetch` for REST requests rather than adding axios or node-fetch. Zero additional dependencies, modern API, consistent with the runtime.
+
+3. **graphql-request for GraphQL**: Lightweight library (v6.x) that supports queries, mutations, and schema introspection. Avoids the heavier @apollo/client which is React-focused.
+
+4. **jsonpath-plus for JSONPath**: Well-maintained v8.x library supporting JSONPath expressions for path-based JSON access.
+
+5. **js-yaml for YAML**: Established v4.x library with load/dump and schema validation.
+
+6. **csv-parse/csv-generate for CSV**: Same author as csv-stringify, reliable and well-maintained v6.x.
+
+7. **HMAC-SHA256 for webhook verification**: Industry standard, supported natively by Node.js `crypto` module.
+
+8. **URL allowlist enforcement**: All network tools validate URLs against an allowlist before making requests. Disallow file://, gopher://, dict:// schemes. Reject internal IPs unless explicitly allowed.
+
+## Risks / Trade-offs
+
+- **Risk**: Adding 5 new npm dependencies increases bundle size.
+  → **Mitigation**: All dependencies are lightweight, well-maintained, and commonly used. graphql-request is ~50KB, jsonpath-plus ~30KB, js-yaml ~100KB.
+
+- **Risk**: Webhook HMAC verification could be bypassed if secret is weak.
+  → **Mitigation**: Document best practices for secret generation. Tool accepts any secret string; security is user responsibility.
+
+- **Risk**: GraphQL query depth/complexity limits could be circumvented.
+  → **Mitigation**: graphql-request supports depth limiting via custom plugins. Default limits (depth: 10, complexity: 1000) are configurable.
+
+- **Risk**: URL allowlist configuration could be forgotten by users.
+  → **Mitigation**: Tool returns clear error messages when URL is not on allowlist, with instructions on how to add it.
+
+## Migration Plan
+
+No migration needed — this is a net-new feature. All tools are additive to the existing tool registry.
+
+## Open Questions
+
+- Should webhook management include delivery status tracking? (deferred to v2)
+- Should the REST client support request/response middleware for logging? (deferred)
+- Should JSON/YAML tools support file I/O in addition to in-memory operations? (deferred, but could be added via a separate `filesystem:read/write` permission)

openspec/changes/archive/2026-08-25-structured-data-api-tools/proposal.md

@@ -0,0 +1,44 @@
+## Why
+
+The existing tools handle web search and content extraction, but there is no structured API interaction capability. Office and marketing workflows frequently need to call REST APIs (CRM, analytics, project management), query GraphQL endpoints, manage webhooks, and manipulate JSON/YAML data. Currently the agent must fall back to shell commands (curl, jq, yq) or rely on ad-hoc LLM reasoning, which is inconsistent and error-prone.
+
+## What Changes
+
+- Add REST API client tool supporting authenticated GET/POST/PUT/DELETE/PATCH requests with configurable headers, body, and authentication (Bearer, Basic, API Key)
+- Add GraphQL client tool for executing queries and mutations with schema introspection, depth/complexity limits
+- Add JSON manipulation tool for parse, transform, filter, and serialize operations with JSONPath-based access
+- Add YAML manipulation tool mirroring JSON tool structure with js-yaml parsing/dumping
+- Add data transformation tool for format conversion between JSON, YAML, and CSV with mapping rules
+- Add webhook management tool for create, list, delete, and verify (HMAC-SHA256) actions
+- Register all tools in `src/tools/index.js` with appropriate permissions
+- Add dependencies: graphql-request, jsonpath-plus, js-yaml, csv-parse, csv-generate
+
+## Capabilities
+
+### New Capabilities
+- `api-client`: REST API client with authentication, timeout, URL allowlist, response sanitization
+- `graphql-client`: GraphQL query/mutation execution with schema introspection and query limits
+- `json-manipulation`: JSON parse, transform, filter, serialize with JSONPath-based access
+- `yaml-manipulation`: YAML parse, transform, filter, serialize with path-based access
+- `data-transformation`: Format conversion between JSON, YAML, CSV with mapping rules
+- `webhook-management`: Webhook create, list, delete, verify with HMAC-SHA256 validation
+
+### Modified Capabilities
+- None
+
+## Impact
+
+- New files: `src/tools/api.js`, `src/tools/graphql.js`, `src/tools/json.js`, `src/tools/yaml.js`, `src/tools/data.js`, `src/tools/webhook.js`
+- Modified: `src/tools/index.js` (register new tools), `package.json` (add dependencies)
+- New tests: `tests/api.test.js`, `tests/graphql.test.js`, `tests/json.test.js`, `tests/yaml.test.js`, `tests/data.test.js`, `tests/webhook.test.js`
+- New config: `data/webhooks.json` (webhook registrations)
+- Security: All network tools enforce URL allowlist per AGENTS.md §1.2
+- Permissions: Network tools require `network:outbound`; data tools require `filesystem:read/write`
+
+## Non-goals
+
+- Subscription support for GraphQL (out of scope for v1)
+- Embedded webhook server (agent-facing tool only; no server component)
+- File I/O for JSON/YAML/CSV tools (in-memory operations only)
+- OAuth/OIDC authentication flows (Bearer, Basic, API Key only)
+- Response caching (considered but deferred)

openspec/changes/archive/2026-08-25-structured-data-api-tools/specs/api-client/spec.md

@@ -0,0 +1,72 @@
+## ADDED Requirements
+
+### Requirement: REST client executes authenticated HTTP requests
+The REST API client SHALL support GET, POST, PUT, DELETE, and PATCH methods with configurable headers, body, and authentication.
+
+#### Scenario: Successful GET request
+- **WHEN** the user calls the REST tool with method "GET" and a valid URL
+- **THEN** the tool returns the response body, status code, and headers
+
+#### Scenario: POST request with JSON body
+- **WHEN** the user calls the REST tool with method "POST", a URL, and a JSON body
+- **THEN** the tool sends the request with Content-Type: application/json and returns the response
+
+#### Scenario: Bearer token authentication
+- **WHEN** the user provides auth.type "bearer" with a token
+- **THEN** the tool adds an Authorization: Bearer <token> header to the request
+
+#### Scenario: Basic authentication
+- **WHEN** the user provides auth.type "basic" with a token
+- **THEN** the tool adds an Authorization: Basic <base64(token)> header to the request
+
+#### Scenario: API Key authentication
+- **WHEN** the user provides auth.type "apikey" with a key and optional token
+- **THEN** the tool adds the API key header as configured
+
+### Requirement: REST client enforces URL allowlist
+The REST API client SHALL validate all request URLs against an allowlist before making outbound requests.
+
+#### Scenario: URL on allowlist succeeds
+- **WHEN** the request URL matches an entry in the allowlist
+- **THEN** the request proceeds normally
+
+#### Scenario: URL not on allowlist is rejected
+- **WHEN** the request URL does not match any entry in the allowlist
+- **THEN** the tool returns an error and does not make the request
+
+#### Scenario: Disallowed schemes are rejected
+- **WHEN** the request URL uses file://, gopher://, or dict:// scheme
+- **THEN** the tool returns an error regardless of allowlist
+
+#### Scenario: Internal IP addresses are rejected
+- **WHEN** the request URL resolves to an internal IP (127.0.0.1, 0.0.0.0, 169.254.169.254)
+- **THEN** the tool returns an error unless explicitly allowed
+
+### Requirement: REST client supports configurable timeouts
+The REST API client SHALL support configurable request timeouts with a default of 30 seconds.
+
+#### Scenario: Default timeout applies
+- **WHEN** no timeout is specified
+- **THEN** the request uses a 30-second default timeout
+
+#### Scenario: Custom timeout is respected
+- **WHEN** the user specifies a timeout of 5000 milliseconds
+- **THEN** the request times out after 5 seconds if not completed
+
+### Requirement: REST client sanitizes responses
+The REST API client SHALL strip sensitive headers from proxied responses.
+
+#### Scenario: Sensitive headers are stripped
+- **WHEN** the response includes Set-Cookie or WWW-Authenticate headers
+- **THEN** the tool removes these headers from the returned response
+
+### Requirement: REST client limits response size
+The REST API client SHALL limit response body size to prevent memory exhaustion.
+
+#### Scenario: Response within limit is returned
+- **WHEN** the response body is within the 10MB limit
+- **THEN** the full response body is returned
+
+#### Scenario: Response exceeding limit is rejected
+- **WHEN** the response body exceeds 10MB
+- **THEN** the tool returns an error with a size-exceeded message

openspec/changes/archive/2026-08-25-structured-data-api-tools/specs/data-transformation/spec.md

@@ -0,0 +1,45 @@
+## ADDED Requirements
+
+### Requirement: Data tool converts between JSON and YAML
+The data transformation tool SHALL convert data between JSON and YAML formats.
+
+#### Scenario: JSON to YAML conversion
+- **WHEN** the user provides JSON input with format "yaml" as target
+- **THEN** the tool returns the data as a YAML string
+
+#### Scenario: YAML to JSON conversion
+- **WHEN** the user provides YAML input with format "json" as target
+- **THEN** the tool returns the data as a JSON string
+
+### Requirement: Data tool converts between JSON and CSV
+The data transformation tool SHALL convert data between JSON and CSV formats.
+
+#### Scenario: JSON array to CSV
+- **WHEN** the user provides a JSON array of objects with format "csv" as target
+- **THEN** the tool returns a CSV string with headers and rows
+
+#### Scenario: CSV to JSON array
+- **WHEN** the user provides a CSV string with format "json" as target
+- **THEN** the tool returns a JSON array of objects with headers as keys
+
+### Requirement: Data tool applies mapping rules during conversion
+The data transformation tool SHALL apply mapping rules to transform field names during conversion.
+
+#### Scenario: Mapping rule applied during JSON to CSV
+- **WHEN** the user provides mapping rules with JSON to CSV conversion
+- **THEN** the tool applies the mapping to column headers in the CSV output
+
+#### Scenario: Mapping rule applied during CSV to JSON
+- **WHEN** the user provides mapping rules with CSV to JSON conversion
+- **THEN** the tool applies the mapping to object keys in the JSON output
+
+### Requirement: Data tool validates input format
+The data transformation tool SHALL validate that input data matches the specified format.
+
+#### Scenario: Invalid JSON input is rejected
+- **WHEN** the user provides invalid JSON with format "json"
+- **THEN** the tool returns an error with a descriptive message
+
+#### Scenario: Invalid CSV input is rejected
+- **WHEN** the user provides malformed CSV with format "csv"
+- **THEN** the tool returns an error with a descriptive message

openspec/changes/archive/2026-08-25-structured-data-api-tools/specs/graphql-client/spec.md

@@ -0,0 +1,60 @@
+## ADDED Requirements
+
+### Requirement: GraphQL client executes queries and mutations
+The GraphQL client SHALL execute GraphQL queries and mutations against a specified endpoint.
+
+#### Scenario: Successful query execution
+- **WHEN** the user provides a GraphQL query string and endpoint URL
+- **THEN** the tool sends the query and returns the response data
+
+#### Scenario: Mutation execution
+- **WHEN** the user provides a GraphQL mutation string and endpoint URL
+- **THEN** the tool sends the mutation and returns the result
+
+#### Scenario: Query with variables
+- **WHEN** the user provides variables alongside the query
+- **THEN** the tool serializes variables and includes them in the request
+
+#### Scenario: Named operation
+- **WHEN** the user provides an operationName
+- **THEN** the tool includes the operation name in the request
+
+### Requirement: GraphQL client supports schema introspection
+The GraphQL client SHALL support schema introspection queries.
+
+#### Scenario: Schema introspection request
+- **WHEN** the user requests schema introspection
+- **THEN** the tool executes the standard introspection query and returns the schema
+
+### Requirement: GraphQL client enforces query depth limits
+The GraphQL client SHALL limit query depth to prevent DoS via deeply nested queries.
+
+#### Scenario: Query within depth limit succeeds
+- **WHEN** the query depth is within the configured limit (default: 10)
+- **THEN** the query executes normally
+
+#### Scenario: Query exceeding depth limit is rejected
+- **WHEN** the query depth exceeds the configured limit
+- **THEN** the tool returns an error indicating depth exceeded
+
+### Requirement: GraphQL client enforces query complexity limits
+The GraphQL client SHALL limit query complexity to prevent DoS via complex queries.
+
+#### Scenario: Query within complexity limit succeeds
+- **WHEN** the query complexity is within the configured limit (default: 1000)
+- **THEN** the query executes normally
+
+#### Scenario: Query exceeding complexity limit is rejected
+- **WHEN** the query complexity exceeds the configured limit
+- **THEN** the tool returns an error indicating complexity exceeded
+
+### Requirement: GraphQL client supports configurable timeouts
+The GraphQL client SHALL support configurable request timeouts with a default of 30 seconds.
+
+#### Scenario: Default timeout applies
+- **WHEN** no timeout is specified
+- **THEN** the request uses a 30-second default timeout
+
+#### Scenario: Custom timeout is respected
+- **WHEN** the user specifies a timeout of 10000 milliseconds
+- **THEN** the request times out after 10 seconds if not completed

openspec/changes/archive/2026-08-25-structured-data-api-tools/specs/json-manipulation/spec.md

@@ -0,0 +1,56 @@
+## ADDED Requirements
+
+### Requirement: JSON tool parses JSON strings
+The JSON manipulation tool SHALL parse JSON strings into structured objects.
+
+#### Scenario: Valid JSON string is parsed
+- **WHEN** the user provides a valid JSON string with action "parse"
+- **THEN** the tool returns the parsed JavaScript object
+
+#### Scenario: Invalid JSON string is rejected
+- **WHEN** the user provides an invalid JSON string
+- **THEN** the tool returns an error with a descriptive message
+
+### Requirement: JSON tool serializes objects to JSON strings
+The JSON manipulation tool SHALL serialize JavaScript objects to JSON strings.
+
+#### Scenario: Object is serialized to JSON
+- **WHEN** the user provides an object with action "serialize"
+- **THEN** the tool returns a valid JSON string
+
+#### Scenario: Object with options is serialized
+- **WHEN** the user specifies pretty-printing options
+- **THEN** the tool returns a formatted JSON string with indentation
+
+### Requirement: JSON tool transforms data
+The JSON manipulation tool SHALL transform JSON data using mapping rules.
+
+#### Scenario: Simple key mapping
+- **WHEN** the user provides a mapping rule to rename keys
+- **THEN** the tool returns the transformed JSON with renamed keys
+
+#### Scenario: Nested transformation
+- **WHEN** the user provides a mapping rule for nested paths
+- **THEN** the tool returns the transformed JSON with nested changes applied
+
+### Requirement: JSON tool filters data with JSONPath
+The JSON manipulation tool SHALL filter JSON data using JSONPath expressions.
+
+#### Scenario: JSONPath filter returns matching values
+- **WHEN** the user provides a JSONPath expression
+- **THEN** the tool returns all matching values from the JSON data
+
+#### Scenario: JSONPath filter returns empty result
+- **WHEN** the JSONPath expression matches no values
+- **THEN** the tool returns an empty array
+
+### Requirement: JSON tool supports path-based access
+The JSON manipulation tool SHALL access JSON values using path-based expressions.
+
+#### Scenario: Simple path access
+- **WHEN** the user provides a dot-notation path
+- **THEN** the tool returns the value at that path
+
+#### Scenario: Array index access
+- **WHEN** the user provides a path with array index
+- **THEN** the tool returns the value at the specified array index