Skip to content

Add VPC DNS diagnostics MCP - #37

Open
ddericco wants to merge 9 commits into
aws-samples:mainfrom
ddericco:feature/aws-vpc-dns-diagnostics-mcp
Open

Add VPC DNS diagnostics MCP#37
ddericco wants to merge 9 commits into
aws-samples:mainfrom
ddericco:feature/aws-vpc-dns-diagnostics-mcp

Conversation

@ddericco

@ddericco ddericco commented Jul 30, 2026

Copy link
Copy Markdown

Description

Adds mcp/aws-vpc-dns-diagnostics-mcp, an MCP server for AWS DevOps Agent that diagnoses VPC DNS resolution in beyond that the AWS APIs can determine.

  • dns_probe_* (Mode A): runs comparative, multi-resolver DNS resolution inside an EC2 instance via SSM Run Command. It returns what a name actually resolves to from that subnet, which resolver answered (hostname.bind), and how a custom or hybrid resolver diverges from the VPC resolver. The EC2, VPC, and Route 53 describe APIs return DNS configuration but do not have visibility into what a client actually sees.

  • dns_simulate_* (Mode B): read-only, pre-change validation. It builds the VPC's effective config as the union of directly attached and Route 53 Profile-inherited constructs, applies a proposed change symbolically, and reports which currently-resolving names would break. A seven-level precedence engine and six trap detectors name the mechanism rather than just the affected zones: VPCE-shadow-NXDOMAIN, broad-FORWARD-sweep, flag-AND-mismatch, DNS-Firewall-block, Profile-union-shift, resolver-disabled.

  • list_sops / get_sop: serves 16 diagnostic runbooks bundled in the deployment package, so the agent fetches interpretation guidance at runtime rather than carrying it in context.

  • Companion skill to help the agent decide when to invoke and define a consistent investigation sequence

Transport follows the documented DevOps Agent path: FastMCP over Streamable HTTP via Lambda Web Adapter, on a Lambda Function URL with AuthType: AWS_IAM and RESPONSE_STREAM, registered with the mcpserversigv4 service type. No AgentCore Gateway or OAuth front end required.

Type of change

  • New skill
  • New custom agent
  • Update to an existing skill or agent
  • Documentation or infrastructure change

This is a new MCP server under mcp/. There is no checkbox for that, so ticking the closest option.

Testing

  • Unit: 88 tests, no AWS calls. Cross-account denial paths use fake sessions raising real botocore exceptions. sam build and sam validate --lint both clean.

  • Live testing: deployed and drove all six tools over a SigV4-signed Function URL. Unsigned requests return 403; signed return 200 with an mcp-session-id. Mode A reproduced a custom-resolver divergence (10.42.200.99 from a local resolver vs 10.42.200.10 from the PHZ, with getent following the custom resolver). Mode B read the live effective config and correctly predicted a broad . FORWARD rule would sweep seven names.

  • Security review: passed the MCP security review skill. Five findings from the first pass were remediated and a re-review returned PASS with no findings and no new issues. Guard tests assert the IAM grants cannot silently re-widen.

  • Agent Skill Eval: report.json shows "passed": true at the root, overall score 0.96, with audit 100/100, functional 0.9 (grade A), and trigger passing. Six functional evals and eight trigger queries, six of them negative. Artifacts committed under skills/aws-vpc-dns-investigation/evals/.

Notes:

  • test-infra/ holds CloudFormation fixtures that reproduce each diagnostic scenario end to end, including a two-account provider/consumer pair for the cross-account opacity cases. Let me know if these should be removed.
  • No credentials exist, so Secrets Manager is not used; the function contacts only AWS control-plane APIs, so it needs no VPC attachment. Both deviations from the reference pattern are documented in the README.

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

ddericco and others added 5 commits July 31, 2026 10:33
Adds mcp/aws-vpc-dns-diagnostics-mcp, an MCP server for AWS DevOps Agent that
diagnoses VPC DNS resolution two ways no describe API can.

Mode A (dns_probe_*) runs live, comparative, multi-resolver DNS resolution inside
an EC2 instance via SSM Run Command, returning what a name actually resolves to
from that subnet, which resolver answered (hostname.bind), and how a custom or
hybrid resolver diverges from the VPC resolver. The EC2, VPC, and Route 53
describe APIs return DNS configuration; they never return that ground truth.

Mode B (dns_simulate_*) is read-only pre-change validation. It builds the VPC's
effective configuration as the union of directly attached and Route 53
Profile-inherited constructs, applies a proposed change symbolically, and reports
which currently-resolving names would break. A seven-level precedence engine and
six trap detectors (VPCE-shadow-NXDOMAIN, broad-FORWARD-sweep, flag-AND-mismatch,
DNS-Firewall-block, Profile-union-shift, resolver-disabled) name the mechanism
rather than only the affected names.

list_sops and get_sop serve 16 diagnostic runbooks bundled in the deployment
package, so the agent fetches interpretation guidance at runtime.

Transport and auth follow the documented DevOps Agent path: FastMCP over
Streamable HTTP via Lambda Web Adapter, exposed on a Lambda Function URL with
AuthType AWS_IAM and RESPONSE_STREAM, registered with the mcpserversigv4 service
type. No AgentCore Gateway or OAuth front end is required.

Security model. The central Lambda execution role holds only sts:AssumeRole and
assumes one of two per-target-account scoped roles per call. Mode B's role never
holds ssm:SendCommand. The probe role's sole privileged grant is ssm:SendCommand
resource-scoped to one purpose-built SSM document that accepts three
allowedPattern-validated parameters and renders a fixed read-only probe set
(dig, cat /etc/resolv.conf, resolvectl, getent). The server sends structured
parameters, never a command string. Account, region, VPC, and resolver allowlists
are enforced in one place, and the server refuses to start on a wildcard when
StageName=prod. There are no credentials to store, so Secrets Manager is not
used; the function needs no VPC attachment because it contacts only AWS
control-plane APIs. Both deviations from the common reference pattern are
documented in the README.

Cross-account constructs shared via RAM or contributed through a Route 53 Profile
may be enumerable but opaque. Denied detail reads become OPAQUE markers rather
than failing the model build, and an opaque firewall rule is evaluated first
because a hidden block list may cover any name. Reporting says 'cannot determine
from this account' rather than 'not affected'.

Validation. 77 unit tests, no AWS calls. Deployed to a test account and drove all
six tools over a SigV4-signed Function URL: unsigned requests return 403, signed
return 200 with an mcp-session-id; Mode A reproduced a custom-resolver divergence
(10.42.200.99 via a local resolver vs 10.42.200.10 from the PHZ, with getent
following the custom resolver); Mode B read the live effective config and
correctly predicted a broad '.' FORWARD rule would sweep seven names. Live
testing surfaced three defects since fixed: a dependency layer built for the
build host's platform rather than manylinux2014_x86_64, two missing EC2 read
grants for DHCP discovery, and add_resolver_rule ignoring the documented
target_ips shape.

test-infra/ holds CloudFormation fixtures that reproduce each diagnostic scenario
end to end, including a two-account provider/consumer pair for the cross-account
opacity cases.

Licensed under Apache-2.0.
Security review callout: with no AllowedAccounts value the deployment defaulted
to '*', which is an implicit allow-all account scope. For a DevOps Agent
integration the account allowlist is the boundary that stops the server assuming
a role into an arbitrary account, so it must be stated explicitly.

Three separate routes reached allow-all, and the template default was only one of
them:
- deploying without overriding the AllowedAccounts parameter
- passing '*' explicitly
- the env var being absent, because _load_allowlist() fell back to '*'

Changing only the template would have left the other two open, and the existing
_enforce_prod_allowlists() gate covers wildcards solely when STAGE_NAME=prod,
leaving a dev or staging deployment attached to a real Agent Space unprotected.

template.yaml: AllowedAccounts loses its Default and gains
AllowedPattern '^[0-9]{12}$' with a ConstraintDescription, so CloudFormation
rejects a missing or malformed value per list item before the function boots.
Confirmed via validate-template that DefaultValue is now null.

src/server.py: new _enforce_account_allowlist() runs at import, unconditionally
and in every stage. It refuses unset, empty, whitespace-only, any entry equal to
'*', and any entry that is not a 12-digit account ID. Because it runs before the
allowlists load, _load_allowlist()'s '*' fallback is unreachable for accounts and
_validate()'s allow-all-on-empty branch cannot be hit for accounts either. Both
docstrings now record that invariant.

Regions, VPCs, and resolvers deliberately keep their permissive defaults.
Accounts are known in advance; requiring ALLOWED_VPCS up front would force a
redeploy to diagnose a new VPC. Those three stay guarded by the prod gate, and
resolvers additionally by literal-IP-only fail-closed behaviour plus a startup
warning.

Docs: README parameter table, Controls table, and Safety blocked list;
ARCHITECTURE fail-closed table.

Tests: 88 passing, up from 77. Eleven new tests in TestAccountAllowlistRequired
cover unset, empty, whitespace-only, wildcard, wildcard mixed with a real
account, malformed, the prod-stage belt-and-braces case, three valid forms, and a
guard asserting template.yaml never regains a Default. Verified the new tests
fail when the fix is reverted.

Also removes a duplicate _enforce_prod_allowlists() call left at module level by
the edit that inserted the new gate. Harmless but confusing; caught in review.

Verified: security review re-run returns PASS with no findings; all five earlier
findings confirmed still cleared; no bypass found on the account path and no tool
reaches an AWS API without passing _preflight(). Redeployed to a test account and
confirmed the tools still work end to end.
@ddericco
ddericco force-pushed the feature/aws-vpc-dns-diagnostics-mcp branch from a7a76e9 to f636f56 Compare July 31, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant