fix(config): idp-core observability

docs/src/deployment/observability.md

@@ -28,48 +28,83 @@ The app exposes health endpoints and uses OTLP export for traces and metrics whe
 
 ## Metrics
 
-### Metrics exported by Spring Boot and Micrometer
-
-The application exposes the standard Micrometer metrics and OTLP export can be enabled through environment variables.
-
-```yaml
-management:
-  tracing:
-    sampling:
-      probability: ${OTEL_TRACES_SAMPLER_ARG:0.1}
-    export:
-      enabled: ${OTEL_TRACING_ENABLED:false}
-
-  otlp:
-    metrics:
-      export:
-        url: ${OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}/v1/metrics}
-        enabled: ${OTEL_METRICS_EXPORT_ENABLED:false}
-        transport: ${OTEL_EXPORTER_OTLP_METRICS_PROTOCOL:${OTEL_EXPORTER_OTLP_PROTOCOL:http/protobuf}}
-
-  opentelemetry:
-    tracing:
-      export:
-        otlp:
-          endpoint: ${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}/v1/traces}
-          transport: ${OTEL_EXPORTER_OTLP_TRACES_TRANSPORT:${OTEL_EXPORTER_OTLP_PROTOCOL:http}}
-    resource-attributes:
-      deployment.environment: ${OTEL_ENV:local}
-      service.namespace: ${OTEL_SERVICE_NAMESPACE:idp-core}
-      service.name: ${OTEL_SERVICE_NAME:internal-developer-platform}
+### Native OpenTelemetry metrics
+
+The application uses `io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter` to configure the SDK, metric readers, and OTLP exporters. You do not need application-defined `OpenTelemetry` or `SdkMeterProvider` beans. The ingestion processors use the auto-configured instance.
+
+```yaml title="application.yml"
+otel:
+  metrics:
+    exporter: ${OTEL_METRICS_EXPORTER:otlp}
+  exporter:
+    otlp:
+      endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318}
+      protocol: ${OTEL_EXPORTER_OTLP_PROTOCOL:http/protobuf}
 ```
 
-### Custom business metrics
+The previous Spring Boot starter exported Micrometer metrics. Its SDK configuration could attach an existing meter provider, but did not create one for these native instruments. Setting `management.opentelemetry.metrics.export.otlp.*` did not activate collection. A `DefaultMeter` therefore discarded custom measurements even while traces worked.
 
-Business-level metrics are intentionally separate from Camel route telemetry. The application adds custom counters only for functional decisions that Camel does not expose as a reusable business signal.
+`camel-opentelemetry2-starter` provides Camel tracing, not automatic base route metrics. This correction restores the existing custom metrics; it does not add Camel base metrics or export Micrometer registries.
 
-Examples:
+### Diagnose missing metrics
+
+1. Restart the application and inspect `[OTEL SDK CHECK]`. `DefaultMeter` means no-op collection; `SdkMeter` confirms a native SDK meter, not successful delivery to Datadog.
+2. Temporarily set `OTEL_METRICS_EXPORTER=console,otlp` and `OTEL_METRIC_EXPORT_INTERVAL=5000`. The starter includes the logging exporter. Console and OTLP exporters can coexist without a manually created reader.
+3. Send a webhook, then wait for the export interval. Look for `idp.webhook.security.requests.total` or `idp.webhook.mapping.applied.total`. A filtered mapping still records attempts and filtered results, but does not create an entity.
+4. If console metrics appear but Datadog receives nothing, check exporter errors, collector metrics routing, authentication, and Datadog filters. Configure required request headers with `OTEL_EXPORTER_OTLP_METRICS_HEADERS`; resource attributes are not HTTP headers.
+5. Restore `OTEL_METRICS_EXPORTER=otlp` and the normal interval after diagnosis. Console metrics can contain operational identifiers and generate large logs.
+
+> [!WARNING]
+> On platforms that allocate CPU only during requests, background export can pause between requests. Check CPU allocation and instance lifetime if collection works but periodic export is delayed.
+
+For HTTP/protobuf, set `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` to the full collector URL, including `/v1/metrics`. Inside a container, `localhost` points to the application container. Use the collector service name, or `host.docker.internal` if the collector runs on your macOS host through Docker Desktop. The repository's Docker Compose file does not start a collector.
+
+### Verify the export path
 
-- `idp.webhook.security.requests.total{connector, security_type}`
-- `idp.webhook.security.result{connector, security_type, result}`
-- `idp.webhook.security.validation.errors.total{connector, security_type, reason}`
+Run the focused tests without a collector or Datadog credentials:
 
-These metrics reflect the real security decision made by the application and should not duplicate the standard Camel `route.*` metrics.
+```bash
+mvn -Dspring.profiles.active=test -Dtest=OpenTelemetryMetricsAutoConfigurationTest test
+```
+
+The tests distinguish these checks:
+
+- **SDK collection**: an in-memory exporter receives a custom counter and histogram with the expected measurements. A separate test checks the warning when you disable the SDK.
+- **OTLP HTTP transport**: the real exporter sends a `protobuf` request to a local server on an ephemeral port. The test checks the method, path, content type, configured header, and encoded metric, connector, and service identifiers. An HTTP 200 response completes the export; an HTTP 400 response reports failure. This check does not decode numeric points or exercise periodic scheduling.
+- **Datadog delivery**: these tests do not contact your deployed collector or Datadog. After deployment, send a webhook and confirm reception at the collector, then in Datadog. Check that the collector enables an OTLP receiver and a metrics pipeline connected to the Datadog exporter, or that your Datadog Agent enables OTLP metrics ingestion. A successful HTTP response alone does not prove downstream delivery.
+
+> [!WARNING]
+> Do not use `SdkMeterProvider.forceFlush().isSuccess()` as proof of HTTP acceptance. In this SDK version, the reader can complete the flush successfully even when the exporter receives HTTP 400. The transport tests observe the actual exporter result; in deployment, check exporter errors and collector reception.
+
+### Custom webhook ingestion metrics
+
+The application emits custom OpenTelemetry metrics in the webhook ingestion pipeline to capture business decisions and failures that are not exposed by the standard Camel route telemetry.
+
+The following metrics are produced by the ingestion, security and decoding processors:
+
+| Metric name                                      | Attributes                              | Meaning                                                                      |
+|--------------------------------------------------|-----------------------------------------|------------------------------------------------------------------------------|
+| `idp.webhook.security.requests.total`            | `connector`, `security_type`            | Number of security checks executed for a connector.                          |
+| `idp.webhook.security.result`                    | `connector`, `security_type`, `result`  | Result of the security decision (`allowed`, `rejected`, `error`).            |
+| `idp.webhook.security.validation.errors.total`   | `connector`, `security_type`, `reason`  | Security validation failures and malformed configuration errors.             |
+| `idp.webhook.payload.decoded.total`              | `connector`, `encoding`                 | Successful payload decodes for each encoding step.                           |
+| `idp.webhook.payload.decode.errors.total`        | `connector`, `reason`                   | Payload decode failures, including unsupported or corrupted encodings.       |
+| `idp.webhook.payload.size.bytes`                 | `connector`                             | Histogram of decoded payload size in bytes.                                  |
+| `idp.webhook.mapping.applied.total`              | `connector`, `mapping_id`, `action`     | Mapping execution attempts.                                                  |
+| `idp.webhook.mapping.filtered.total`             | `connector`, `mapping_id`, `reason`     | Mapping results rejected by the filter stage.                                |
+| `idp.webhook.mapping.result`                     | `connector`, `mapping_id`, `result`     | Mapping outcome (`MATCHED`, `FILTERED_OUT`, `ERROR`).                        |
+| `idp.webhook.mapping.errors.total`               | `connector`, `mapping_id`, `error_type` | Mapping exceptions and runtime failures.                                     |
+| `idp.webhook.mapping.latency.ms`                 | `connector`, `mapping_id`               | Mapping duration histogram in milliseconds.                                  |
+| `idp.webhook.entity.created.total`               | `template`, `connector`                 | Number of entities created during ingestion.                                 |
+| `idp.webhook.entity.updated.total`               | `template`, `connector`                 | Number of entities updated during ingestion.                                 |
+| `idp.webhook.entity.deleted.total`               | `template`, `connector`                 | Number of entities deleted during ingestion.                                 |
+| `idp.webhook.entity.patch.total`                 | `template`, `connector`, `action`       | Number of patch operations applied.                                          |
+| `idp.webhook.entity.validation.errors.total`     | `template`, `connector`, `reason`       | Entity validation and business-rule failures.                                |
+| `idp.webhook.entity.notfound.total`              | `template`, `connector`, `action`       | Missing entity operations during ingestion.                                  |
+| `idp.webhook.entity.conflict.total`              | `template`, `connector`, `reason`       | Conflict scenarios such as duplicate entities.                               |
+| `idp.webhook.functional.validation.failed.total` | `connector`, `template`, `reason`       | Functional validation failures captured after a mapping or entity operation. |
+
+These metric names are centralized in `IngestionObservabilityConstants` and are intentionally aligned with the connector, template, action, and reason dimensions used across the webhook processing pipeline.
 
 ## Logging
 
@@ -85,19 +120,23 @@ OpenTelemetry tracing is enabled through Spring Boot configuration. The applicat
 
 | Variable | Description | Default |
 | --- | --- | --- |
-| `OTEL_TRACING_ENABLED` | Enables tracing export | `false` |
-| `OTEL_METRICS_EXPORT_ENABLED` | Enables OTLP metrics export | `false` |
+| `OTEL_TRACES_EXPORTER` | Trace exporter; `none` disables export | `otlp` (`none` locally) |
+| `OTEL_METRICS_EXPORTER` | Metric exporters; `console,otlp` for diagnosis | `otlp` |
+| `OTEL_METRIC_EXPORT_INTERVAL` | Metric export interval in milliseconds | `60000` |
+| `OTEL_LOGS_EXPORTER` | Log exporter | `none` |
 | `OTEL_TRACES_SAMPLER_ARG` | Trace sampling ratio | `0.1` |
 | `OTEL_EXPORTER_OTLP_ENDPOINT` | Base OTLP endpoint | `http://localhost:4318` |
 | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Full traces endpoint | `<base>/v1/traces` |
 | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Full metrics endpoint | `<base>/v1/metrics` |
-| `OTEL_EXPORTER_OTLP_PROTOCOL` | Generic OTLP protocol fallback | `http/protobuf` for metrics, `http` for traces |
-| `OTEL_EXPORTER_OTLP_TRACES_TRANSPORT` | Trace-specific transport override | `OTEL_EXPORTER_OTLP_PROTOCOL` or `http` |
+| `OTEL_EXPORTER_OTLP_PROTOCOL` | OTLP protocol, `http/protobuf` or `grpc` | `http/protobuf` |
+| `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | Trace-specific protocol override | `OTEL_EXPORTER_OTLP_PROTOCOL` |
 | `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL` | Metrics-specific transport override | `OTEL_EXPORTER_OTLP_PROTOCOL` |
 | `OTEL_RESOURCE_ATTRIBUTES` | Resource metadata injected by the environment | - |
 
 The signal-specific variables win over the generic fallback, so local development and hosted deployments can both work without hard-coded vendor configuration.
 
+Replace the old `OTEL_TRACING_ENABLED` and `OTEL_METRICS_EXPORT_ENABLED` flags with exporter selection (`otlp` or `none`). Replace `*_TRANSPORT=http` with `*_PROTOCOL=http/protobuf`. The official starter reads `otel.*`, not the former `management.opentelemetry.*` export settings. The test profile disables network export for all signals.
+
 ### Important trace attributes
 
 The application sets trace attributes for the webhook pipeline, including the connector identifier and the security decision.
@@ -119,7 +158,9 @@ The app exposes health endpoints used by Docker and Kubernetes.
 | `/actuator/health/liveness`  | Liveness probe       |
 | `/actuator/health/readiness` | Readiness probe      |
 
-## Next steps
+---
+
+## Next Steps
 
 - **[Configuration](configuration.md)** - Environment and Spring Boot settings
 - **[Docker](docker.md)** - Docker deployment and OTLP examples

pom.xml

@@ -39,6 +39,14 @@
   <!--  Swagger version management to avoid conflicts -->
   <dependencyManagement>
     <dependencies>
+      <!-- Align the native metrics SDK and the official OTel Spring Boot starter. -->
+      <dependency>
+        <groupId>io.opentelemetry.instrumentation</groupId>
+        <artifactId>opentelemetry-instrumentation-bom</artifactId>
+        <version>2.32.0</version>
+        <type>pom</type>
+        <scope>import</scope>
+      </dependency>
       <dependency>
         <groupId>io.netty</groupId>
         <artifactId>netty-bom</artifactId>
@@ -180,6 +188,12 @@
       <scope>test</scope>
     </dependency>
 
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-sdk-testing</artifactId>
+      <scope>test</scope>
+    </dependency>
+
     <dependency>
       <groupId>org.mock-server</groupId>
       <artifactId>mockserver-client-java</artifactId>
@@ -332,9 +346,10 @@
       <artifactId>camel-rest-starter</artifactId>
     </dependency>
 
+    <!-- Autoconfigures native OTel metrics, readers and exporters, not Micrometer metrics. -->
     <dependency>
-      <groupId>org.springframework.boot</groupId>
-      <artifactId>spring-boot-starter-opentelemetry</artifactId>
+      <groupId>io.opentelemetry.instrumentation</groupId>
+      <artifactId>opentelemetry-spring-boot-starter</artifactId>
     </dependency>
 
     <dependency>
@@ -343,8 +358,8 @@
     </dependency>
 
     <dependency>
-      <groupId>org.apache.camel.springboot</groupId>
-      <artifactId>camel-micrometer-starter</artifactId>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-api</artifactId>
     </dependency>
 
   </dependencies>

src/main/java/com/decathlon/idp_core/infrastructure/adapters/ingestion/configuration/CamelTracingConfiguration.java

@@ -0,0 +1,49 @@
+package com.decathlon.idp_core.infrastructure.adapters.ingestion.configuration;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.spring.boot.CamelContextConfiguration;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+import io.opentelemetry.api.OpenTelemetry;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.context.propagation.ContextPropagators;
+import lombok.extern.slf4j.Slf4j;
+
+@Configuration
+@Slf4j
+public class CamelTracingConfiguration {
+
+  static final String CAMEL_INSTRUMENTATION_SCOPE = "camel";
+
+  @Bean
+  public Tracer camelTracer(OpenTelemetry openTelemetry) {
+    return openTelemetry.getTracer(CAMEL_INSTRUMENTATION_SCOPE);
+  }
+
+  @Bean
+  public ContextPropagators camelContextPropagators(OpenTelemetry openTelemetry) {
+    return openTelemetry.getPropagators();
+  }
+
+  @Bean
+  public CamelContextConfiguration camelTracingBridge(Tracer camelTracer,
+      ContextPropagators camelContextPropagators) {
+    return new CamelContextConfiguration() {
+      @Override
+      public void beforeApplicationStart(CamelContext camelContext) {
+        // Expose Spring SDK beans directly in Camel registry before route startup.
+        camelContext.getRegistry().bind("camelTracer", camelTracer);
+        camelContext.getRegistry().bind("camelContextPropagators", camelContextPropagators);
+        log.info(
+            "[OTEL CAMEL BRIDGE] Bound Tracer={} and ContextPropagators={} into Camel registry",
+            camelTracer.getClass().getName(), camelContextPropagators.getClass().getName());
+      }
+
+      @Override
+      public void afterApplicationStart(CamelContext camelContext) {
+        // No-op
+      }
+    };
+  }
+}

src/main/java/com/decathlon/idp_core/infrastructure/adapters/ingestion/configuration/IngestionMetricsConfiguration.java

@@ -0,0 +1,26 @@
+package com.decathlon.idp_core.infrastructure.adapters.ingestion.configuration;
+
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+import io.opentelemetry.api.OpenTelemetry;
+import io.opentelemetry.api.metrics.Meter;
+
+@Configuration
+public class IngestionMetricsConfiguration {
+
+  @Bean("ingestionMeter")
+  public Meter ingestionMeter(OpenTelemetry openTelemetry) {
+    return openTelemetry.getMeter("com.decathlon.idp_core.ingestion");
+  }
+
+  @Bean("securityMeter")
+  public Meter securityMeter(OpenTelemetry openTelemetry) {
+    return openTelemetry.getMeter("com.decathlon.idp_core.security");
+  }
+
+  @Bean("decodingMeter")
+  public Meter decodingMeter(OpenTelemetry openTelemetry) {
+    return openTelemetry.getMeter("com.decathlon.idp_core.decoding");
+  }
+}

src/main/java/com/decathlon/idp_core/infrastructure/adapters/ingestion/configuration/IngestionObservabilityConstants.java

@@ -0,0 +1,41 @@
+package com.decathlon.idp_core.infrastructure.adapters.ingestion.configuration;
+
+public final class IngestionObservabilityConstants {
+
+  public static final String SECURITY_REQUESTS_TOTAL = "idp.webhook.security.requests.total";
+  public static final String SECURITY_RESULT = "idp.webhook.security.result";
+  public static final String SECURITY_VALIDATION_ERRORS_TOTAL = "idp.webhook.security.validation.errors.total";
+
+  public static final String PAYLOAD_DECODED_TOTAL = "idp.webhook.payload.decoded.total";
+  public static final String PAYLOAD_DECODE_ERRORS_TOTAL = "idp.webhook.payload.decode.errors.total";
+  public static final String PAYLOAD_SIZE_BYTES = "idp.webhook.payload.size.bytes";
+
+  public static final String MAPPING_APPLIED_TOTAL = "idp.webhook.mapping.applied.total";
+  public static final String MAPPING_FILTERED_TOTAL = "idp.webhook.mapping.filtered.total";
+  public static final String MAPPING_RESULT = "idp.webhook.mapping.result";
+  public static final String MAPPING_ERRORS_TOTAL = "idp.webhook.mapping.errors.total";
+  public static final String MAPPING_LATENCY_MS = "idp.webhook.mapping.latency.ms";
+
+  public static final String ENTITY_CREATED_TOTAL = "idp.webhook.entity.created.total";
+  public static final String ENTITY_UPDATED_TOTAL = "idp.webhook.entity.updated.total";
+  public static final String ENTITY_DELETED_TOTAL = "idp.webhook.entity.deleted.total";
+  public static final String ENTITY_PATCH_TOTAL = "idp.webhook.entity.patch.total";
+  public static final String ENTITY_VALIDATION_ERRORS_TOTAL = "idp.webhook.entity.validation.errors.total";
+
+  public static final String FUNCTIONAL_VALIDATION_FAILED_TOTAL = "idp.webhook.functional.validation.failed.total";
+  public static final String ENTITY_NOT_FOUND_TOTAL = "idp.webhook.entity.notfound.total";
+  public static final String ENTITY_CONFLICT_TOTAL = "idp.webhook.entity.conflict.total";
+
+  public static final String ATTRIBUTE_CONNECTOR = "connector";
+  public static final String ATTRIBUTE_MAPPING_ID = "mapping_id";
+  public static final String ATTRIBUTE_ACTION = "action";
+  public static final String ATTRIBUTE_REASON = "reason";
+  public static final String ATTRIBUTE_TEMPLATE = "template";
+  public static final String ATTRIBUTE_RESULT = "result";
+  public static final String ATTRIBUTE_SECURITY_TYPE = "security_type";
+
+  public static final String SPAN_ATTRIBUTE_WEBHOOK_IDENTIFIER = "idp.webhook.identifier";
+
+  private IngestionObservabilityConstants() {
+  }
+}