From 9450b6f65ad053d98c679e612e7a6342f0ed930d Mon Sep 17 00:00:00 2001 From: Sameeran Kunche Date: Tue, 19 May 2026 22:09:06 +0000 Subject: [PATCH 1/4] Activate tests/ffe/test_flag_eval_metrics.py across released tracers Ruby v2.32.0, Python v4.7.0, Go v2.8.0, Node.js v5.99.0 (express4 only), .NET v3.44.0, and Java v1.62.0 now ship FFE evaluation metrics. Java keeps the per-class FFL-1972 overrides so CI XPASSes will flag the tests that can be trimmed. Co-Authored-By: Claude Opus 4.7 (1M context) --- manifests/dotnet.yml | 2 +- manifests/golang.yml | 2 +- manifests/java.yml | 2 +- manifests/nodejs.yml | 5 ++++- manifests/python.yml | 2 +- manifests/ruby.yml | 2 +- 6 files changed, 9 insertions(+), 6 deletions(-) diff --git a/manifests/dotnet.yml b/manifests/dotnet.yml index cad11d470b6..bc752779f76 100644 --- a/manifests/dotnet.yml +++ b/manifests/dotnet.yml @@ -702,7 +702,7 @@ manifest: tests/ffe/test_dynamic_evaluation.py::Test_FFE_Flag_Parse_Error_Isolation: bug (FFL-2184) tests/ffe/test_dynamic_evaluation.py::Test_FFE_Unknown_Operator_Tolerance: bug (FFL-2184) tests/ffe/test_exposures.py: v3.36.0 - tests/ffe/test_flag_eval_metrics.py: missing_feature (FFL-2257 - Requires Datadog.FeatureFlags.OpenFeature 2.3.0 NuGet package to be published) + tests/ffe/test_flag_eval_metrics.py: v3.44.0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integrations/crossed_integrations/test_kafka.py::Test_Kafka: v2.0.0-prerelease tests/integrations/crossed_integrations/test_kinesis.py::Test_Kinesis_PROPAGATION_VIA_MESSAGE_ATTRIBUTES: missing_feature diff --git a/manifests/golang.yml b/manifests/golang.yml index 0733c38a01b..24d700ccc76 100644 --- a/manifests/golang.yml +++ b/manifests/golang.yml @@ -876,7 +876,7 @@ manifest: tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Unavailable: v2.4.0 tests/ffe/test_dynamic_evaluation.py::Test_FFE_Unknown_Operator_Tolerance::test_unknown_operator_errors: bug (FFL-2182) tests/ffe/test_exposures.py: v2.6.0-dev # Easy win for chi, echo, gin, net-http, net-http-orchestrion, uds-echo and version 2.5.0 - tests/ffe/test_flag_eval_metrics.py: v2.9.0-dev + tests/ffe/test_flag_eval_metrics.py: v2.8.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex: irrelevant (Go validates regex at config load time) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980) tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) diff --git a/manifests/java.yml b/manifests/java.yml index 9a5ae0023e6..b31c0b39238 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3159,7 +3159,7 @@ manifest: "*": irrelevant spring-boot: v1.56.0 tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1729) - tests/ffe/test_flag_eval_metrics.py: missing_feature (FFL-1972) + tests/ffe/test_flag_eval_metrics.py: v1.62.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Config_Exists_Flag_Missing::test_ffe_eval_config_exists_flag_missing: bug (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_error_type: bug (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_reason: bug (FFL-1972) diff --git a/manifests/nodejs.yml b/manifests/nodejs.yml index c3a1e9132f8..ac7765b57d5 100644 --- a/manifests/nodejs.yml +++ b/manifests/nodejs.yml @@ -1667,7 +1667,10 @@ manifest: "*": incomplete_test_app express4: *ref_5_77_0 tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1730) - tests/ffe/test_flag_eval_metrics.py: missing_feature + tests/ffe/test_flag_eval_metrics.py: + - weblog_declaration: + "*": incomplete_test_app + express4: *ref_5_99_0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: irrelevant (JS SDK requires targeting key) tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: *ref_5_71_0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: *ref_5_71_0 diff --git a/manifests/python.yml b/manifests/python.yml index 7d709c654ab..b37772b9b2e 100644 --- a/manifests/python.yml +++ b/manifests/python.yml @@ -1271,7 +1271,7 @@ manifest: tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Down_From_Start: v4.0.0 tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Unavailable: flaky (FFL-1622) tests/ffe/test_exposures.py: v4.2.0-dev - tests/ffe/test_flag_eval_metrics.py: v4.8.0-dev + tests/ffe/test_flag_eval_metrics.py: v4.7.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980) tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: v3.16.0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: v3.16.0 diff --git a/manifests/ruby.yml b/manifests/ruby.yml index d7819d1c89b..9c659ca97a7 100644 --- a/manifests/ruby.yml +++ b/manifests/ruby.yml @@ -1816,7 +1816,7 @@ manifest: tests/ffe/test_flag_eval_metrics.py: - weblog_declaration: "*": irrelevant - rails72: v2.32.0-dev + rails72: v2.32.0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integrations/crossed_integrations/test_kafka.py::Test_Kafka: - weblog_declaration: From 96666a854dd14084d5e45793b8dfad9681afb3cc Mon Sep 17 00:00:00 2001 From: Sameeran Kunche Date: Tue, 19 May 2026 23:11:01 +0000 Subject: [PATCH 2/4] [nodejs] Mark 5 FFE eval metric tests as bug (FFL-2313) dd-trace-js v5.103.0 fails to distinguish feature_flag.result.reason for static/split/type-mismatch paths (all report targeting_match) and emits error.type=general instead of parse_error. Tracked in FFL-2313 under FFL-1899. Co-Authored-By: Claude Opus 4.7 (1M context) --- manifests/nodejs.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/manifests/nodejs.yml b/manifests/nodejs.yml index ac7765b57d5..a8c77d84988 100644 --- a/manifests/nodejs.yml +++ b/manifests/nodejs.yml @@ -1671,6 +1671,11 @@ manifest: - weblog_declaration: "*": incomplete_test_app express4: *ref_5_99_0 + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-2313) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-2313) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex: bug (FFL-2313) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch: bug (FFL-2313) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-2313) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: irrelevant (JS SDK requires targeting key) tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: *ref_5_71_0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: *ref_5_71_0 From dc4ef75b932324c5a98b10d82ad6e9a226d401ed Mon Sep 17 00:00:00 2001 From: Sameeran Kunche Date: Tue, 19 May 2026 23:27:59 +0000 Subject: [PATCH 3/4] Trim Java overrides for v1.62.0 fixes and re-tag JS targeting key test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Java v1.62.0 actually shipped fixes for 16 of the 17 FFL-1972 tests (verified by CI XPASS on the prod spring-boot job). Only Test_FFE_Eval_Metric_Count still fails; the rest can run without overrides. Test_FFE_Eval_Targeting_Key_Optional for nodejs is more accurately a bug (FFL-1730) than irrelevant — the JS SDK errors on empty targeting key instead of treating it as optional. Co-Authored-By: Claude Opus 4.7 (1M context) --- manifests/java.yml | 18 ------------------ manifests/nodejs.yml | 8 +++++--- 2 files changed, 5 insertions(+), 21 deletions(-) diff --git a/manifests/java.yml b/manifests/java.yml index b31c0b39238..c4c752f82f1 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3160,25 +3160,7 @@ manifest: spring-boot: v1.56.0 tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1729) tests/ffe/test_flag_eval_metrics.py: v1.62.0 - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Config_Exists_Flag_Missing::test_ffe_eval_config_exists_flag_missing: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_error_type: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_reason: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-1972) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Count::test_ffe_eval_metric_count: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Different_Flags::test_ffe_eval_metric_different_flags: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-1972) - ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex - : bug (FFL-1972) - ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch - : bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Type_Mismatch::test_ffe_eval_metric_type_mismatch: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored::test_ffe_eval_nested_attributes_ignored: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_No_Config_Loaded::test_ffe_eval_no_config_loaded: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Default::test_ffe_eval_reason_default: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Disabled::test_ffe_eval_reason_disabled: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: bug (FFL-1972) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: bug (FFL-1972) tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0 tests/integration_frameworks/llm/openai/test_openai_llmobs.py: v1.61.0 diff --git a/manifests/nodejs.yml b/manifests/nodejs.yml index a8c77d84988..6825e2bbbff 100644 --- a/manifests/nodejs.yml +++ b/manifests/nodejs.yml @@ -1673,10 +1673,12 @@ manifest: express4: *ref_5_99_0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-2313) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-2313) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex: bug (FFL-2313) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch: bug (FFL-2313) + ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex + : bug (FFL-2313) + ? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch + : bug (FFL-2313) tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-2313) - tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: irrelevant (JS SDK requires targeting key) + tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: bug (FFL-1730) tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: *ref_5_71_0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: *ref_5_71_0 tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create: missing_feature # ephemeral cache TTL metrics not yet released From 9391bc658c4205837f47a9a506124a313d1e0703 Mon Sep 17 00:00:00 2001 From: Sameeran Kunche Date: Wed, 20 May 2026 05:33:31 +0000 Subject: [PATCH 4/4] Scope Java FFE eval metrics tests to spring-boot weblog only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous catch-all entry ran the tests on every Java weblog, which caused 404s on akka-http, jersey-grizzly2, play, ratpack, resteasy-netty3, spring-boot-3-native, vertx3, and vertx4 — none of which implement the /ffe endpoint. Matches the existing weblog_declaration pattern used by test_dynamic_evaluation.py and test_exposures.py. --- manifests/java.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/manifests/java.yml b/manifests/java.yml index c4c752f82f1..09ca1131a1b 100644 --- a/manifests/java.yml +++ b/manifests/java.yml @@ -3159,7 +3159,10 @@ manifest: "*": irrelevant spring-boot: v1.56.0 tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1729) - tests/ffe/test_flag_eval_metrics.py: v1.62.0 + tests/ffe/test_flag_eval_metrics.py: + - weblog_declaration: + "*": irrelevant + spring-boot: v1.62.0 tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Count::test_ffe_eval_metric_count: bug (FFL-1972) tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234) tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0