Skip to content

Cap runtime log retention and stop logging prompt text in the kernel - #55

Closed
zzkamzn wants to merge 1 commit into
aws-samples:mainfrom
zzkamzn:fix/runtime-log-hygiene
Closed

zzkamzn wants to merge 1 commit into
aws-samples:mainfrom
zzkamzn:fix/runtime-log-hygiene

Conversation

@zzkamzn

@zzkamzn zzkamzn commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Problem

  • Runtime log groups never expire. AgentCore creates /aws/bedrock-agentcore/runtimes/<runtime-id>-DEFAULT at the moment the runtime is created, with no retention, and nothing in the repo set one. On a live deployment every runtime's group shows retentionInDays: none. That includes groups of runtimes that were destroyed long ago.
  • Those groups hold user text.
    • The headless kernel logs invoke session=%s prompt=%.80r (runtimes/agent-sdk-kernel/src/main.py), so the first 80 characters of every prompt land there.
    • With agent_observability on, the same group also receives the otel-rt-logs event records with full prompt and answer text (docs/observability.md).
    • Anyone with logs:GetLogEvents on the account can read all of it, indefinitely.
  • The docs disagree about traces. docs/user-guide.md describes span-level traces as "per tool call, per model turn". docs/observability.md correctly says there are no per-turn LLM spans.

Change

  • Terraform. New terraform/modules/runtime/log_retention.tf.
    • Terraform cannot own these groups, because AgentCore creates them before the runtime resource returns. So, following the existing platform_version.tf pattern, a terraform_data with local-exec runs aws logs put-retention-policy for every runtime: interactive and sdk per platform version, plus mcp-tools.
    • The trigger is (runtime id, days), so replacing a runtime or changing the value reruns it.
    • New variable runtime_log_retention_days, default 30. It is validated against CloudWatch Logs' allowed values, and 0 leaves the groups as created. It is also added to terraform.tfvars.example.
  • deploy-cli. 30-runtime.sh sets the same policy, controlled by RUNTIME_LOG_RETENTION_DAYS, default 30.
  • Kernel. It logs prompt_chars=<n> instead of the prompt text.
  • Docs.
    • The user guide now says one span per invocation plus one per tool call, and links the gotcha.
    • observability.md notes the retention setting and that the group holds full prompt and answer text when trace delivery is on.

There are no new IAM permissions. The AWS CLI is already a stated Terraform prerequisite, and local-exec already depends on it.

Not changed

  • Existing orphaned groups. Groups of runtimes that no longer exist are not touched. They are outside Terraform state; delete them by hand if wanted.
  • aws/spans. Its retention is not set here. That group belongs to the account-level Transaction Search setup, which the repo asks operators to enable themselves.
  • trace_id on kernel logs. docs/observability.md says kernel stdout lines carry trace_id=… span_id=…. The kernel's own format string does not include them, but OTel log correlation in the -otel image may add them. I could not verify that without running the observability image, so that sentence is left as is.

Test plan

  • terraform fmt -check -recursive passes, and terraform validate passes with the new resource and variable.
  • bash -n passes on 30-runtime.sh.
  • Ran the exact put-retention-policy command shape against a non-existent group. It got ResourceNotFoundException, which shows the arguments parse and reach the service, with no side effects on real groups.
  • Confirmed on a live account that AgentCore creates the group at runtime-creation time with no retention. This is why Terraform cannot own it.

This came out of an inventory of DynamoDB and CloudWatch usage done for the #44 review. It is independent of #44.

🤖 Generated with Claude Code

AgentCore creates /aws/bedrock-agentcore/runtimes/<id>-DEFAULT when a
runtime is created, with no retention. Nothing in the repo set one, so the
group kept everything forever. That includes the headless kernel's
'invoke session=... prompt=...' line, which records the first 80 characters
of every user prompt, and, with agent_observability on, the otel-rt-logs
event records that carry full prompt and answer text.

Terraform cannot own the group because it exists as soon as the runtime
does. Following platform_version.tf, a terraform_data with local-exec runs
aws logs put-retention-policy for every runtime (interactive and sdk per
platform version, plus mcp-tools). The trigger is (runtime id, days). The new
variable runtime_log_retention_days defaults to 30, is validated against the
CloudWatch Logs values, and 0 leaves the groups unmanaged. deploy-cli does
the same with RUNTIME_LOG_RETENTION_DAYS.

The kernel now logs only the prompt length.

The user guide described span-level traces as "per model turn".
observability.md already states there are no per-turn LLM spans, so the
guide now says one span per invocation plus one per tool call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@odinwang

Copy link
Copy Markdown
Contributor

Thanks! This change is included in #66 (published together with the related platform fixes). This repository is now updated by snapshot PRs only, so closing this one.

@odinwang odinwang closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants