Skip to content

feat(cloudformation): Add DevOps Agent alarm investigations template - #38

Open
LorenzoBoccaccia wants to merge 1 commit into
aws-samples:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn
Open

feat(cloudformation): Add DevOps Agent alarm investigations template#38
LorenzoBoccaccia wants to merge 1 commit into
aws-samples:mainfrom
LorenzoBoccaccia:add-alarm-investigations-cfn

Conversation

@LorenzoBoccaccia

Copy link
Copy Markdown

Description

Add a CloudFormation addon that forwards a single CloudWatch alarm to a DevOps Agent generic (HMAC) webhook to open an investigation, plus a combined cloudformation/ README documenting it alongside the existing skill-policies template.

The signing Lambda hardens the forwarding path: layered SSRF egress checks with connect-to-validated-IP pinning, HMAC-SHA256 request signing, alarm-ARN match to reject spoofed/mismatched events, retry-stable incidentId for dedup, staleness cutoff matching the retry window, and least-privilege IAM. Reserved concurrency and log retention are configurable; the webhook HMAC key is read from a Secrets Manager secret (SecretString or SecretBinary).

Type of change

  • New skill
  • New custom agent
  • Update to an existing skill or agent
  • Documentation or infrastructure change

Testing

This change is a CloudFormation automation add-on (not a skill), so Agent Skill
Eval does not apply
. It was validated at four levels — template, unit, live AWS,
and a real DevOps Agent webhook.

1. Template validation

  • aws cloudformation validate-template — passed (6 parameters recognized,
    CAPABILITY_IAM).
  • Full YAML parse with CloudFormation intrinsic tags — well-formed.

2. Local unit / adversarial tests of the inline signing Lambda (python3.12, the
target runtime) — 24/24 passing, covering:

  • SSRF egress classifier: blocks RFC1918, loopback, link-local, 169.254.169.254
    (IMDS), CGNAT 100.64/10, TEST-NET/reserved, and IPv4-embedded-in-IPv6
    (::ffff:, NAT64 64:ff9b::, 6to4); allows public v4/v6.
  • Log-injection: control chars / newlines / NUL stripped and ARN capped at 2048.
  • Secret parsing: raw string, JSON key fields, and UTF-8 SecretBinary; rejects
    empty / missing-field / non-string.
  • Staleness cutoff (8h) and ALARM-state gating.
  • Alarm-ARN match: mismatched / empty / missing resources[0] rejected and logged.
  • Retry-stable incidentId (event id, then alarmArn+state-timestamp; never
    wall-clock).
  • HMAC signature: base64(HMAC-SHA256(key, "timestamp:payload")).
  • End-to-end against a local TLS mock that recomputed the HMAC and validated the
    JSON schema through the real _post / pinned-TLS connect.

3. Live AWS deploy-and-teardown (throwaway dev account, all resources tagged,
fully torn down afterwards):

  • Stack reached CREATE_COMPLETE (IAM role, inline-packaged Lambda, EventBridge
    rule, log group, conditional reserved concurrency).
  • Invoked the deployed Lambda with a synthetic ALARM event: exercised Secrets
    Manager fetch → HMAC sign → SSRF resolve → TLS POST and handled the endpoint
    response.
  • Spoofed event (wrong alarm ARN) → skipped: alarm ARN mismatch.
  • Confirmed DeletionPolicy: Delete behavior; stack, log group, and secret removed.

4. Real DevOps Agent webhook contract (manual) — posted one signed request to a
live generic (HMAC) webhook: HTTP 200, confirming the signing scheme
(x-amzn-event-timestamp / x-amzn-event-signature) and the payload schema
(eventType/incidentId/action/priority/title/description/timestamp/
data) are accepted as-is.

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

Add a CloudFormation addon that forwards a single CloudWatch alarm to a DevOps Agent generic (HMAC) webhook to open an investigation, plus a combined cloudformation/ README documenting it alongside the existing skill-policies template.

The signing Lambda hardens the forwarding path: layered SSRF egress checks with connect-to-validated-IP pinning, HMAC-SHA256 request signing, alarm-ARN match to reject spoofed/mismatched events, retry-stable incidentId for dedup, staleness cutoff matching the retry window, and least-privilege IAM. Reserved concurrency and log retention are configurable; the webhook HMAC key is read from a Secrets Manager secret (SecretString or SecretBinary).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant