api-probe

Containerized API validation tool for post-deployment functional testing in CI/CD pipelines.

Features

✅ REST & GraphQL Support
✅ XML/SOAP Support (XPath)
✅ Multiple Execution Contexts (test different users/accounts)
✅ Variable Substitution in requests AND validations
✅ Output Variable Capture with expressions (len, has, empty)
✅ Conditional Probe Execution (ignore field with expressions)
✅ Conditional Validation (skip headers/body based on response)
✅ Parallel Group Execution (flat and staged with isolated variable scopes)
✅ Include Directive (!include) for external files
✅ Rich Validations (status, headers, body)
✅ Timeout, Retry, Debug support
✅ Response Time Validation
✅ Progress Reporting to stderr
✅ Silent Success / Verbose Failure
✅ Docker-First, CI/CD Native

Quick Start

Local Development

# One command - auto-setup and run
./run.sh examples/passing/simple.yaml

# Run all examples
chmod +x run-examples.sh
./run-examples.sh

Docker

# Build
docker build -t api-probe .

# Run
docker run --rm \
  -v $(pwd)/examples:/configs \
  api-probe /configs/passing/simple.yaml

Example

executions:
  - name: "Production User"
    vars:
      - ACCOUNT: "123456789"
      - API_KEY: "${PROD_API_KEY}"  # From environment
  
  - name: "Staging User"
    vars:
      - ACCOUNT: "987654321"
      - API_KEY: "${STAGING_API_KEY}"

probes:
  - name: "Login"
    type: rest
    endpoint: "${BASE_URL}/auth"
    method: POST
    headers:
      Content-Type: "application/json"
    body: !include includes/login-body.json  # External file
    validation:
      status: 200
      response_time: 1000  # Must respond within 1 second
      body:
        equals:
          account_id: "${ACCOUNT}"  # Variable substitution in validation!
    output:
      TOKEN: "access_token"
      HAS_PREMIUM: "has(body.premium)"  # Expression evaluation
  
  - name: "Get Profile"
    type: rest
    endpoint: "${BASE_URL}/profile"
    headers:
      Authorization: "Bearer ${TOKEN}"
    validation:
      status: 200
      body:
        present: ["id", "email"]
  
  - name: "Get Premium Features"
    type: rest
    endpoint: "${BASE_URL}/features"
    ignore: "!HAS_PREMIUM"  # Skip if not premium user
    validation:
      status: 200
      body:
        ignore: "empty(body.features)"  # Skip validation if empty
        present: ["features[0].name"]

Multiple Execution Contexts

Test different user accounts, regions, or environments in one run:

executions:
  - name: "US East User"
    vars:
      - CLIENT_ID: "client-123"
      - REGION: "us-east-1"
  
  - name: "EU West User"
    vars:
      - CLIENT_ID: "client-456"
      - REGION: "eu-west-1"

probes:
  - name: "API Test"
    endpoint: "https://api.example.com/${REGION}/data"
    headers:
      X-Client: "${CLIENT_ID}"
    validation:
      body:
        equals:
          region: "${REGION}"  # Validates against execution-specific value

Each execution runs independently with isolated variables.

See examples/passing/executions-block.yaml for details.

Conditional Execution

Skip probes or validation based on previous results:

probes:
  - name: "Get Offers"
    endpoint: "https://api.example.com/offers"
    output:
      OFFER_COUNT: "len(body.offers)"  # Capture count using expression
      HAS_PREMIUM: "has(body.premium)"
  
  - name: "Process Rich Offers"
    endpoint: "https://api.example.com/process"
    ignore: "OFFER_COUNT <= 2"  # Skip if not enough offers
    validation:
      status: 200
  
  - name: "Validate Premium Features"
    endpoint: "https://api.example.com/user"
    validation:
      status: 200
      body:
        ignore: "!HAS_PREMIUM"  # Skip body validation if not premium
        present:
          - "premium.tier"
          - "premium.benefits"

Expression Functions:

• len(VAR) - Get length of array/string/dict
• has(VAR) - Check if exists and not empty
• empty(VAR) - Check if empty or None

Operators: ==, !=, >, <, >=, <=, &&, ||, !

See SCHEMA_SPECIFICATION.md for details.

Parallel Groups

Groups support two modes: flat (all probes in parallel) and staged (stages in parallel, probes within each stage sequential).

Flat Group

All probes run at the same time:

probes:
  - name: "Sequential Test"
    endpoint: "https://api.example.com/test"
  
  - group:
      probes:
        - name: "Parallel Test 1"
          endpoint: "https://api.example.com/delay/2"
        - name: "Parallel Test 2"
          endpoint: "https://api.example.com/delay/2"
        - name: "Parallel Test 3"
          endpoint: "https://api.example.com/delay/2"
  # Group completes in ~2 seconds instead of 6 seconds

Staged Group

Stages run in parallel, but probes within each stage run sequentially. Each stage has an isolated variable scope — output captured in one stage does not leak to sibling stages:

probes:
  - group:
      name: "Multi-User Auth Flow"
      stages:
        - name: "User A"
          probes:
            - name: "Auth - User A"       # runs first
              type: rest
              endpoint: "${AUTH_URL}"
              output:
                TOKEN_A: "data.token"
            - name: "API Call - User A"   # runs after Auth - User A
              type: graphql               # skipped if TOKEN_A not set
              endpoint: "${BASE_URL}/graphql"
              headers:
                Authorization: "Bearer ${TOKEN_A}"

        - name: "User B"                  # runs in parallel with User A
          probes:
            - name: "Auth - User B"
              type: rest
              endpoint: "${AUTH_URL}"
              output:
                TOKEN_B: "data.token"     # isolated — not visible in User A stage
            - name: "API Call - User B"
              type: graphql
              endpoint: "${BASE_URL}/graphql"
              headers:
                Authorization: "Bearer ${TOKEN_B}"

See examples/passing/groups-parallel.yaml for details.

Include Directive

Keep large request bodies in separate files:

probes:
  - name: "Create User"
    type: rest
    endpoint: "https://api.example.com/users"
    method: POST
    headers:
      Content-Type: "application/json"
    body: !include includes/user-profile.json
    validation:
      status: 201

See examples/passing/include-directive.yaml for details.

AI-Assisted Probe Generation

api-probe ships with AI skills for Claude Code and GitHub Copilot that turn natural language and existing API sources into ready-to-run probe files. The skills understand the full schema, handle auth detection, DAG ordering, !include externalisation, and per-probe validation — so you describe what you want and the AI does the rest.

Supported inputs

Paste or reference any combination of:

• curl commands
• Collection files — Postman, Bruno, Insomnia, OpenAPI/Swagger
• HAR files
• A plain description of your endpoints

The skill also scans your codebase directly (Express, FastAPI, Spring Boot, NestJS, Quarkus, Rails, Laravel, Go, .NET, and more) to extract routes, response shapes, and auth patterns without any file provided.

Install the skills

Install via Homebrew, then run the init command from your project root:

brew install hemantobora/tap/api-probe
api-probe init

The installer detects your project type and lets you pick which AI tool to configure — use arrow keys to select, then press Enter:

  Which AI tool would you like to configure?

    ❯  GitHub Copilot
       Claude Code

GitHub Copilot — installs two prompt files to .github/prompts/:

.github/prompts/api-probe-generate.prompt.md
.github/prompts/api-probe-sync.prompt.md

Attach them in VS Code agent mode with #api-probe-generate or reference them via @workspace.

Claude Code — installs two slash commands to .claude/commands/api-probe/:

.claude/commands/api-probe/generate.md   →  /api-probe:generate
.claude/commands/api-probe/sync.md       →  /api-probe:sync

Use /api-probe:generate to create a new probe file, /api-probe:sync to keep an existing one in sync after API changes.

Upgrade

Upgrade api-probe via Homebrew, then re-run init from your project root. If the bundled skills are newer than what is installed, all files for that tool are updated automatically:

brew upgrade hemantobora/tap/api-probe
api-probe init

The installer tracks versions in .api-probe/skills-manifest.json.

Uninstall

api-probe destroy

Removes only the files placed by init. Empty parent directories (.github/prompts/, .claude/commands/api-probe/) are cleaned up. Directories with other content are left untouched.

Local development

If you are working on the skills themselves, you can run the installer directly from the repository:

python3 skills/install.py        # init
python3 skills/install.py destroy

---

Documentation

• GETTING_STARTED.md - Installation and basic usage
• SCHEMA_SPECIFICATION.md - Complete YAML syntax
• DOCKER.md - Container usage and CI/CD integration
• XML_SOAP_GUIDE.md - XPath expressions and SOAP testing
• INCLUDE_DIRECTIVE.md - YAML include directive usage
• PROMPT.md - Copy-paste prompt for any LLM (ChatGPT, etc.)

Examples

Passing Examples (Expected: ✓ Silent Success)

• simple.yaml - Basic REST API probes
• comprehensive.yaml - All validation keywords
• complex-validation.yaml - Advanced patterns
• graphql.yaml - GraphQL API testing
• xml-soap.yaml - XML/SOAP with XPath
• executions-block.yaml - Multiple execution contexts
• multi-context.yaml - Multi-user testing
• no-executions.yaml - Single run with env vars
• groups-parallel.yaml - Parallel group execution
• include-directive.yaml - External file includes
• advanced-features.yaml - JSONPath + parallel groups

Failing Examples (Expected: ✗ Verbose Errors)

• test-failures.yaml - Basic intentional failures
• validation-failures.yaml - All validator failures
• variable-validation-failures.yaml - Variable validation errors
• group-failures.yaml - Failures in parallel groups
• execution-names-in-reports.yaml - Execution names in failure output
• multiple-execution-failures.yaml - Different failures per execution
• auto-generated-names-failures.yaml - Auto-generated execution names

Running All Examples

# Set executable permission
chmod +x run-examples.sh

# Run all passing and failing examples
./run-examples.sh

This will run all 15 passing examples and 7 failing examples with proper environment variables.

CI/CD Integration

GitHub Actions

- name: API Tests
  run: |
    docker run --rm \
      -v ${{ github.workspace }}/configs:/configs \
      -e PROD_API_KEY="${{ secrets.PROD_API_KEY }}" \
      -e STAGING_API_KEY="${{ secrets.STAGING_API_KEY }}" \
      api-probe:latest /configs/tests.yaml

Concourse

- task: api-tests
  image: api-probe-image
  params:
    PROD_API_KEY: ((prod-api-key))
    STAGING_API_KEY: ((staging-api-key))
  run:
    path: api-probe
    args: ["/configs/tests.yaml"]

See DOCKER.md for more CI/CD examples.

Installation

Homebrew (Recommended)

brew install hemantobora/tap/api-probe

Docker

docker build -t api-probe .
docker run --rm api-probe --help

Local Development

./run.sh examples/passing/simple.yaml

See GETTING_STARTED.md for details.

Status

Version: 2.7.0
Status: Production Ready

Version History

• v2.7.0

  • Added staged groups (stages:) — stages run in parallel, probes within each stage run sequentially
  • Each stage has an isolated variable scope (output does not leak between stages)
  • Retry and output capture warnings now printed atomically with their probe (no interleaving)
  • Output capture failure leaves variable unset — downstream probes are skipped rather than running with "None"
  • All-skipped runs now treated as failure for CI safety
  • Final report shows per-run probe counts in summary
  • Final report shows correct validation state (Passed, Passed (no validation), Passed (validation skipped))
  • Fixed in-group live output — removed duplicate probe name from result line

• v2.6.0

  • Added verify field for per-probe SSL/TLS certificate validation skip
  • Added !int, !bool, !float, !str type coercion tags for typed variable substitution
  • Unified output path convention — bare paths for body fields, headers. prefix for response headers
  • Fixed session isolation — fresh HTTP session per thread (no connection pool sharing)
  • Fixed debug output through print lock (no interleaving in concurrent executions)
  • Added response time display for every probe in report output
  • Added duplicate probe name detection (warning on validate)
  • Added retry config validation (max_attempts >= 1, delay >= 0)
  • Fixed skipped probe symbol consistency — ⊗ used in both live progress and final report

• v2.5.0

  • Added success reporting for probe execution
  • Executing probes in separate threads per execution block
  • Fixed the documentation in DOCKER.md

• v2.4.0

  • Added expression evaluation in output field (len, has, empty functions)
  • Added expression evaluation in ignore field (len, has, empty functions)
  • Added ignore field in validation headers and body sections
  • Expressions support operators: ==, !=, >, <, >=, <=, &&, ||, !
  • Expressions can access response data: status, body., headers.

• v2.3.0

  • Added ignore field for probes and groups (conditional execution)
  • Added name field for groups (with auto-generation)
  • Improved parallel group progress reporting with names

• v2.2.0

  • Added timeout field for request timeouts
  • Added retry configuration for automatic retries
  • Added debug flag for request/response logging
  • Added response_time validation for performance checks
  • Added status pattern matching (2xx, 3xx, 4xx, 5xx)
  • Added progress reporting to stderr
  • Fixed length validator for root-level arrays ($)

• v2.1.0

  • Added delay field for probes
  • Added length validator for arrays and strings

• v2.0.0

  • Added executions block
  • Added variable substitution in validation
  • Added parallel groups
  • Added XML/SOAP support
  • Added !include directive
  • Enhanced variable resolution

• v1.0.0

  • Initial release

License

See LICENSE