Skip to content

test(aws-sdk): stabilize DynamoDB trace assertions - #9675

Merged
tlhunter merged 2 commits into
masterfrom
BridgeAR/2026-08-04-fix-aws-sdk-dynamodb-flake
Aug 7, 2026
Merged

test(aws-sdk): stabilize DynamoDB trace assertions#9675
tlhunter merged 2 commits into
masterfrom
BridgeAR/2026-08-04-fix-aws-sdk-dynamodb-flake

Conversation

@BridgeAR

@BridgeAR BridgeAR commented Aug 4, 2026

Copy link
Copy Markdown
Member

Summary

Use the existing 5s LocalStack window for DynamoDB trace assertions.

Why

LocalStack may respond after the mock agent's 1s default expires, which removes the observer before the SDK callback completes.

@dd-octo-sts

dd-octo-sts Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 7.96 MB
Deduped: 8.62 MB
No deduping: 8.62 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@BridgeAR
BridgeAR marked this pull request as ready for review August 4, 2026 14:20
@BridgeAR
BridgeAR requested a review from a team as a code owner August 4, 2026 14:20
@BridgeAR
BridgeAR requested review from tlhunter and removed request for a team August 4, 2026 14:20
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 4, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.55% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 76090b3 | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.55%. Comparing base (b9b4e0b) to head (76090b3).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #9675   +/-   ##
=======================================
  Coverage   98.55%   98.55%           
=======================================
  Files         966      966           
  Lines      138858   138858           
  Branches    11993    12084   +91     
=======================================
+ Hits       136845   136846    +1     
+ Misses       2013     2012    -1     
Flag Coverage Δ
aiguard 56.99% <ø> (-0.03%) ⬇️
aiguard-integration 55.90% <ø> (ø)
apm-bucket-0 57.28% <ø> (+0.01%) ⬆️
apm-bucket-1 63.39% <ø> (-0.02%) ⬇️
apm-bucket-2 62.24% <ø> (-0.03%) ⬇️
apm-bucket-3 59.82% <ø> (-0.03%) ⬇️
apm-capabilities-tracing 62.36% <ø> (ø)
apm-integrations-aerospike 56.30% <ø> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript 61.22% <ø> (-0.03%) ⬇️
apm-integrations-couchbase 56.72% <ø> (-0.03%) ⬇️
apm-integrations-http 61.93% <ø> (-0.02%) ⬇️
apm-integrations-kafkajs 61.74% <ø> (-0.03%) ⬇️
apm-integrations-next 59.42% <ø> (-0.03%) ⬇️
apm-integrations-prisma 58.53% <ø> (-0.02%) ⬇️
appsec 72.17% <ø> (-0.05%) ⬇️
appsec-express_fastify_graphql 69.48% <ø> (-0.02%) ⬇️
appsec-integration 50.18% <ø> (+<0.01%) ⬆️
appsec-kafka_ldapjs_lodash 63.44% <ø> (-0.02%) ⬇️
appsec-mongodb-core_mongoose_mysql 66.92% <ø> (-0.02%) ⬇️
appsec-next 56.67% <ø> (-0.02%) ⬇️
appsec-node-serialize_passport_postgres 66.33% <ø> (-0.02%) ⬇️
appsec-sourcing_stripe_template 64.75% <ø> (-0.03%) ⬇️
debugger 64.28% <ø> (-0.04%) ⬇️
instrumentations-bucket-0 51.68% <ø> (-0.03%) ⬇️
instrumentations-bucket-1 59.41% <ø> (-0.03%) ⬇️
instrumentations-bucket-10 60.90% <ø> (-0.02%) ⬇️
instrumentations-bucket-11 61.63% <ø> (-0.02%) ⬇️
instrumentations-bucket-12 51.69% <ø> (-0.03%) ⬇️
instrumentations-bucket-13 52.43% <ø> (-0.03%) ⬇️
instrumentations-bucket-14 51.65% <ø> (-0.03%) ⬇️
instrumentations-bucket-2 53.14% <ø> (-0.03%) ⬇️
instrumentations-bucket-3 53.49% <ø> (-0.03%) ⬇️
instrumentations-bucket-4 58.78% <ø> (-0.03%) ⬇️
instrumentations-bucket-5 50.29% <ø> (-0.02%) ⬇️
instrumentations-bucket-6 60.52% <ø> (-0.03%) ⬇️
instrumentations-bucket-7 58.12% <ø> (-0.03%) ⬇️
instrumentations-bucket-8 59.10% <ø> (-0.03%) ⬇️
instrumentations-bucket-9 57.45% <ø> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 50.97% <ø> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.24% <ø> (ø)
llmobs-ai_anthropic_bedrock 62.84% <ø> (-0.02%) ⬇️
llmobs-bucket-1 61.34% <ø> (-0.02%) ⬇️
llmobs-openai 61.81% <ø> (-0.02%) ⬇️
llmobs-openai-agents_vertex-ai 59.83% <ø> (-0.02%) ⬇️
llmobs-sdk 66.57% <ø> (-0.03%) ⬇️
master-coverage 98.55% <ø> (?)
openfeature 55.71% <ø> (ø)
openfeature-unit 53.22% <ø> (-0.03%) ⬇️
platform-core_esbuild_instrumentations-misc 41.22% <ø> (-0.02%) ⬇️
platform-integration 60.50% <ø> (ø)
platform-shimmer_unit-guardrails_webpack 38.88% <ø> (-0.02%) ⬇️
plugins-bucket-0 56.93% <ø> (-0.02%) ⬇️
plugins-bucket-1 54.05% <ø> (ø)
plugins-bucket-11 61.50% <ø> (-0.03%) ⬇️
plugins-bucket-17 61.32% <ø> (-0.03%) ⬇️
plugins-bucket-18 61.95% <ø> (-0.02%) ⬇️
plugins-bucket-19 61.34% <ø> (-0.03%) ⬇️
plugins-bucket-20 63.76% <ø> (-0.03%) ⬇️
plugins-bucket-4 58.32% <ø> (-0.03%) ⬇️
plugins-bullmq_cassandra_cookie 61.40% <ø> (-0.03%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 56.36% <ø> (-0.03%) ⬇️
plugins-fetch_fs_generic-pool 58.23% <ø> (-0.07%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 64.18% <ø> (-0.03%) ⬇️
plugins-hapi_hono_ioredis 59.92% <ø> (-0.03%) ⬇️
plugins-knex_langgraph_ldapjs 55.06% <ø> (-0.03%) ⬇️
plugins-light-my-request_limitd-client_lodash 58.39% <ø> (-0.03%) ⬇️
plugins-mariadb_memcached_mercurius 61.32% <ø> (-0.02%) ⬇️
plugins-mongodb_mongodb-core_mongoose 59.28% <ø> (-0.03%) ⬇️
plugins-multer_mysql_mysql2 58.86% <ø> (-0.03%) ⬇️
plugins-nats_node-serialize_opensearch 60.42% <ø> (-0.03%) ⬇️
plugins-passport-http_pino_postgres 58.60% <ø> (-0.03%) ⬇️
plugins-process_pug_redis 57.40% <ø> (-0.03%) ⬇️
plugins-undici_url_valkey 58.04% <ø> (-0.03%) ⬇️
plugins-vm_winston_ws 59.61% <ø> (-0.03%) ⬇️
profiling 61.54% <ø> (-0.03%) ⬇️
serverless-aws-sdk-aws-sdk 55.12% <ø> (-0.02%) ⬇️
serverless-aws-sdk-base-inject-field 50.92% <ø> (-0.03%) ⬇️
serverless-aws-sdk-bedrockruntime 54.65% <ø> (-0.02%) ⬇️
serverless-aws-sdk-client 56.22% <ø> (-0.03%) ⬇️
serverless-aws-sdk-dynamodb 55.50% <ø> (-0.02%) ⬇️
serverless-aws-sdk-eventbridge 49.71% <ø> (-0.02%) ⬇️
serverless-aws-sdk-kinesis 59.09% <ø> (-0.02%) ⬇️
serverless-aws-sdk-lambda 57.24% <ø> (-0.02%) ⬇️
serverless-aws-sdk-s3 55.59% <ø> (-0.02%) ⬇️
serverless-aws-sdk-serverless-peer-service 59.35% <ø> (-0.02%) ⬇️
serverless-aws-sdk-sns 59.90% <ø> (-0.02%) ⬇️
serverless-aws-sdk-sqs 60.32% <ø> (-0.02%) ⬇️
serverless-aws-sdk-stepfunctions 55.42% <ø> (-0.02%) ⬇️
serverless-aws-sdk-util 51.45% <ø> (-0.03%) ⬇️
serverless-bucket-0 54.02% <ø> (-0.11%) ⬇️
serverless-bucket-1 58.87% <ø> (-0.03%) ⬇️
test-optimization-cucumber 71.12% <ø> (+0.01%) ⬆️
test-optimization-cypress 64.91% <ø> (+0.06%) ⬆️
test-optimization-jest 72.50% <ø> (-0.15%) ⬇️
test-optimization-mocha 72.17% <ø> (+0.03%) ⬆️
test-optimization-playwright-playwright-atr 59.93% <ø> (-0.03%) ⬇️
test-optimization-playwright-playwright-efd 60.06% <ø> (+0.05%) ⬆️
test-optimization-playwright-playwright-final-status 60.23% <ø> (-0.02%) ⬇️
test-optimization-playwright-playwright-impacted-tests 59.76% <ø> (+0.13%) ⬆️
test-optimization-playwright-playwright-reporting 60.93% <ø> (-0.05%) ⬇️
test-optimization-playwright-playwright-test-management 60.75% <ø> (-0.05%) ⬇️
test-optimization-playwright-playwright-test-span 59.98% <ø> (-0.08%) ⬇️
test-optimization-selenium 59.12% <ø> (-0.10%) ⬇️
test-optimization-testopt 57.65% <ø> (+0.07%) ⬆️
test-optimization-vitest 73.38% <ø> (+0.02%) ⬆️
test-optimization-vitest-browser 58.99% <ø> (+0.04%) ⬆️
test-optimization-webdriverio 65.50% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

## Summary

Use the existing 5s LocalStack window for DynamoDB trace assertions.

## Why

LocalStack may respond after the mock agent's 1s default expires, which removes the observer before the SDK callback completes.
@tlhunter
tlhunter force-pushed the BridgeAR/2026-08-04-fix-aws-sdk-dynamodb-flake branch from c5c8b7d to aea0e92 Compare August 6, 2026 18:29
@tlhunter

tlhunter commented Aug 6, 2026

Copy link
Copy Markdown
Member

CI had a lot of seemingly unrelated failures so I force pushed a master rebase.

@tlhunter
tlhunter enabled auto-merge (squash) August 6, 2026 18:38
@pr-commenter

pr-commenter Bot commented Aug 6, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-07 12:18:08

Comparing candidate commit 76090b3 in PR branch BridgeAR/2026-08-04-fix-aws-sdk-dynamodb-flake with baseline commit b9b4e0b in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2312 metrics, 46 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-206.684ms; +201.542ms] or [-7.667%; +7.476%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-200.648ms; +255.450ms] or [-7.786%; +9.913%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-157.527ms; +165.118ms] or [-5.052%; +5.296%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-183679.332µs; +182284.999µs] or [-6.239%; +6.192%]

scenario:appsec-control-20

  • unstable execution_time [-134.580ms; +131.015ms] or [-8.138%; +7.922%]

scenario:appsec-control-24

  • unstable execution_time [-114.921ms; +112.916ms] or [-9.151%; +8.991%]

scenario:appsec-control-26

  • unstable execution_time [-126.310ms; +131.210ms] or [-10.064%; +10.454%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-12874.482µs; +13705.870µs] or [-4.915%; +5.233%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-14.220ms; +18.128ms] or [-5.521%; +7.038%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-28167.984µs; +28960.317µs] or [-5.060%; +5.202%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-2116.947ms; +3432.433ms] or [-25.745%; +41.743%]
  • unstable execution_time [-2099.355ms; +3458.611ms] or [-23.507%; +38.727%]
  • unstable instructions [-17.9G instructions; +29.7G instructions] or [-26.869%; +44.447%]
  • unstable max_rss_usage [-8.734MB; +12.384MB] or [-5.488%; +7.782%]
  • unstable throughput [-921.767op/s; +542.465op/s] or [-24.842%; +14.620%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-1972.736ms; +3094.873ms] or [-24.309%; +38.136%]
  • unstable execution_time [-2067.572ms; +3184.451ms] or [-23.492%; +36.182%]
  • unstable instructions [-16.8G instructions; +26.4G instructions] or [-25.497%; +39.980%]
  • unstable max_rss_usage [-6.988MB; +11.279MB] or [-4.418%; +7.131%]
  • unstable throughput [-882.239op/s; +587.078op/s] or [-23.498%; +15.637%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-2280.447ms; +761.670ms] or [-23.765%; +7.938%]
  • unstable execution_time [-2308.909ms; +743.115ms] or [-22.390%; +7.206%]
  • unstable instructions [-20.5G instructions; +6.8G instructions] or [-25.704%; +8.490%]
  • unstable throughput [-145.847op/s; +452.030op/s] or [-4.537%; +14.062%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-3727.956ms; +3712.535ms] or [-36.146%; +35.997%]
  • unstable execution_time [-3854.984ms; +3836.575ms] or [-34.901%; +34.734%]
  • unstable instructions [-32730.2M instructions; +32701.4M instructions] or [-37.826%; +37.792%]
  • unstable max_rss_usage [-12899.806KB; +12200.606KB] or [-8.026%; +7.591%]
  • unstable throughput [-737.135op/s; +756.042op/s] or [-23.869%; +24.481%]

scenario:dogstatsd-aggregated-20

  • unstable cpu_usage_percentage [-6.548%; +4.580%]
  • unstable execution_time [-87.607ms; +124.252ms] or [-6.788%; +9.628%]
  • unstable throughput [-797669.413op/s; +569999.366op/s] or [-6.785%; +4.848%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-465.329ms; +225.214ms] or [-9.776%; +4.731%]
  • unstable execution_time [-455.745ms; +227.885ms] or [-9.426%; +4.713%]
  • unstable throughput [-78621.885op/s; +164604.783op/s] or [-4.529%; +9.483%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-26

  • unstable cpu_usage_percentage [-2.180%; +9.154%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-14.541MB; +24.676MB] or [-3.736%; +6.339%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-4.652MB; +9.617MB] or [-3.670%; +7.588%]

scenario:plugin-graphql-long-with-depth-off-26

  • unstable max_rss_usage [-31028.809KB; +32525.381KB] or [-14.678%; +15.386%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-589.678ms; +612.078ms] or [-5.119%; +5.313%]
  • unstable execution_time [-598.205ms; +627.484ms] or [-5.091%; +5.340%]
  • unstable throughput [-3.661op/s; +3.511op/s] or [-5.346%; +5.127%]

scenario:plugin-pg-service-26

  • unstable cpu_usage_percentage [-6.181%; +6.449%]
  • unstable execution_time [-86.711ms; +77.518ms] or [-9.510%; +8.502%]
  • unstable throughput [-435125.655op/s; +512680.635op/s] or [-6.492%; +7.649%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-3831.695KB; +4163.695KB] or [-4.826%; +5.244%]

@tlhunter
tlhunter merged commit eeab14c into master Aug 7, 2026
978 of 981 checks passed
@tlhunter
tlhunter deleted the BridgeAR/2026-08-04-fix-aws-sdk-dynamodb-flake branch August 7, 2026 12:23
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 7, 2026
## Summary

Use the existing 5s LocalStack window for DynamoDB trace assertions.

## Why

LocalStack may respond after the mock agent's 1s default expires, which removes the observer before the SDK callback completes.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Aug 7, 2026
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 7, 2026
## Summary

Use the existing 5s LocalStack window for DynamoDB trace assertions.

## Why

LocalStack may respond after the mock agent's 1s default expires, which removes the observer before the SDK callback completes.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Aug 7, 2026
sabrenner pushed a commit that referenced this pull request Aug 7, 2026
## Summary

Use the existing 5s LocalStack window for DynamoDB trace assertions.

## Why

LocalStack may respond after the mock agent's 1s default expires, which removes the observer before the SDK callback completes.
sabrenner pushed a commit that referenced this pull request Aug 7, 2026
## Summary

Use the existing 5s LocalStack window for DynamoDB trace assertions.

## Why

LocalStack may respond after the mock agent's 1s default expires, which removes the observer before the SDK callback completes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants