By: Yvette Nartey
Automating DORA metrics (Lead Time for Changes & Deployment Frequency) by deterministically correlating asynchronous developer-workflow events into a single distributed trace.
A custom OpenTelemetry telemetry middleware that links Jira issues, GitLab commits, CI/CD pipelines, and deployments into one unified trace — without modifying or deeply integrating any of those tools. Built as a Bachelor's thesis in Software Engineering (Code University, Berlin), validated with a controlled experiment.
TL;DR: Standard distributed tracing breaks across asynchronous webhook boundaries, so cross-tool DORA metrics can't be computed automatically. This middleware generates deterministic Trace IDs by hashing the Jira issue key, forcing every event in a feature's lifecycle into the same trace. Result: 0% → 100% correlation success, 92.5% less trace fragmentation, and automated Lead Time for Changes with 0% error — at +1.1 ms overhead.
Modern delivery toolchains are fragmented by design: Jira tracks work, GitLab manages code and CI/CD, Jaeger/Prometheus watch production. Each tool is excellent in isolation — and blind to the others.
The root cause is architectural. Standard distributed tracing (Google Dapper, W3C Trace Context) relies on context propagation: trace IDs passed in HTTP headers between services during synchronous communication. But developer workflows are asynchronous — tools talk to each other through webhooks, and webhook payloads carry no trace context. The causal chain snaps at every tool boundary.
The practical consequence: you cannot automatically measure the DORA metrics that span tools — specifically Lead Time for Changes (commit → production) — because the commit event and the deployment event live in completely separate, unconnected traces. Teams fall back to manual data reconciliation or proprietary, vendor-locked platforms. Neither scales.
Research question: How can deterministic correlation of asynchronous developer-workflow events enable automated calculation of DORA metrics through a unified signal architecture?
Instead of relying on propagated (random) trace IDs, the middleware derives the Trace ID deterministically from a business identifier — the Jira issue key. Any event that references the same key (a Jira update, a commit on a TE-123-* branch, a pipeline, a deployment) hashes to the same 128-bit Trace ID and therefore lands in the same trace — even when the events occur days apart on different infrastructure.
Trace ID = HMAC-SHA256( TRACE_ID_HMAC_SECRET, normalize(jira_issue_key) )[:32 hex chars] # e.g. "TE-123" → stable 128-bit ID, unguessable without the secret
Span ID = random UUID # uniqueness per event
The stack runs as a "Local Production" environment via Docker Compose: SaaS webhooks (Jira Cloud, GitLab Cloud) arrive over an ngrok tunnel, get correlated by the telemetry middleware, and fan out as traces and metrics through the OpenTelemetry Collector.
| Component | Function |
|---|---|
| Telemetry Middleware | FastAPI service that correlates events and generates OTel signals |
| Jaeger | Distributed tracing backend for visualizing the happy-path developer workflow |
| Prometheus | Time-series database storing Lead Time and Deployment Frequency |
| Grafana | Visualizes metrics from Prometheus |
| Backstage | The IDP — surfaces signals in a developer-centric view |
- Custom OTel ID generator — intercepts trace creation to inject deterministic hashes instead of the SDK's default random IDs, using OTel's strict API/SDK separation.
- Content as the carrier — leverages OTel's Propagators API to treat the webhook JSON body (not HTTP headers) as the context carrier, mapping fields to OTel semantic conventions (
git.*,cicd.*,workflow.*). - Real-time metrics, no ETL — Lead Time is computed inline from the event stream via a stateful manager (commit timestamp → deployment timestamp), exported as a Prometheus histogram. No batch pipeline required.
- Structured logging via span events — logs attach to their parent span (
correlation.success,processing.complete, etc.), so logs and traces stay correlated without a separate aggregation stack. - Webhook reliability — async fire-and-forget pattern returns
200 OKimmediately so GitLab never disables the webhook under load; correlation work runs in a background task behind a thread-safe state manager. - Clean root spans — manually resets OTel context per request to defeat "ghost parent" traces injected by ngrok's auto-instrumentation.
A controlled experiment compared the deterministic strategy against standard OpenTelemetry across six runs (3 control, 3 experimental), holding the workflow, toolchain versions, and network constant.
| Metric | Standard OTel (control) | Deterministic (experimental) | Δ |
|---|---|---|---|
| Traces per workflow | 13.33 (fragmented) | 1.00 (unified) | −92.5% |
| Spans per trace | ~1 | 13.67 | consolidated |
| Correlation success rate | 0% (0/3) | 100% (3/3) | +100 pp |
| Automated LTC success | 0% (0/3) | 100% (3/3) | +100 pp |
| Automated LTC accuracy | N/A (no data) | 0% relative error vs. manual | — |
| Deployment Frequency | works | works | unchanged |
Performance (93 webhook events): mean processing latency 5.4 ms; deterministic strategy adds +1.1 ms vs. standard (MD5 + state lookup) — negligible against 50–200 ms webhook network latency. CPU < 1%, memory steady at ~75 MiB.
Cross-stack consistency: automated Lead Time values matched exactly across Jaeger → Prometheus → Grafana with zero data loss.
- A business identifier (Jira key) can serve as a stable backbone for distributed tracing, substituting determinism for propagation across long-running, asynchronous workflows.
- The distinction between counting (Deployment Frequency works without correlation) and linking (Lead Time requires it) — so the technique is precisely scoped to where it adds value.
- The middleware is lightweight and designed to scale horizontally — Trace IDs are derivable independently, so no central coordinator is needed. (This is an architectural property; a formal concurrent-load test was out of scope — see Limitations.)
Scope note: the entire experiment ran locally via Docker Compose ("production-in-a-box"). Jira and GitLab were live SaaS sources, but the processing/visualization stack and the deployment target were local. No cloud-platform deployment was validated.
Fair question — and for many teams, the honest answer is: don't. If you run a small team and just want deployment counts and basic DORA dashboards, GitLab's built-in metrics or a low-cost engineering-intelligence tool (LinearB, Sleuth) is the right call. This project's own analysis says exactly that: deterministic correlation is only worth the cost when you need to link events across tools (e.g. Lead Time), not merely count them.
This work is not a product competing with those tools. It's a validation of a technique — and that distinction is the point:
| Commercial platforms (Datadog CI, LinearB, Sleuth…) | This approach | |
|---|---|---|
| Cross-tool correlation | Proprietary integrations or heuristic matching | Deterministic, derived from a business key on open standards |
| Standard | Vendor-specific | Vendor-neutral OpenTelemetry — no lock-in |
| Data location | Your delivery data in their cloud | Self-hostable; you own and can audit it |
| Auditability | Black-box correlation logic | Transparent — the linkage is reproducible from the Jira key |
Where this genuinely beats buying:
- Vendor lock-in matters. The correlation rides on OpenTelemetry, the CNCF industry standard — not a proprietary agent. Swap any backend without re-instrumenting.
- Regulated industries (banking, healthcare). Proving exactly when a specific ticket reached production is often a legal/audit requirement. A deterministic, self-hosted, reproducible trail is stronger evidence than a SaaS black box.
- You want to understand the mechanism, not rent it. The expensive tools solve this problem too — behind a paywall and an opaque integration. This shows the underlying method working on standards anyone can adopt.
The honest trade-off: this requires OpenTelemetry fluency to build and operate. That's a real cost — and in 2026, OTel is the standard platform teams are expected to know, so it's a cost most platform orgs are already paying. The technique is precision-scoped: reach for it when cross-tool lineage, auditability, or vendor independence justify the engineering effort. Otherwise, buy the tool.
| Layer | Technology |
|---|---|
| Middleware | Python, FastAPI, OpenTelemetry Python SDK (custom ID generator + Propagators API) |
| Tracing backend | Jaeger 1.53 |
| Metrics | Prometheus 2.48 (histogram + counter) |
| Visualization | Grafana 10.2 |
| Internal Developer Platform | Backstage 1.44 (+ PostgreSQL catalog) |
| Telemetry routing | OpenTelemetry Collector (OTLP) |
| Connectivity | ngrok (public SaaS → local tunnel) |
| Orchestration | Docker Compose (production-in-a-box) |
| Event sources | Jira Cloud, GitLab Cloud (webhooks) |
.
├── backstage-ote/ # Internal Developer Portal (Backstage Monorepo)
│ ├── packages/
│ │ ├── app/ # React Frontend (The UI)
│ │ └── backend/ # Node.js Backend (Catalog & Plugins)
│ ├── app-config.yaml # Main Backstage Configuration
│ └── Dockerfile # IDP Container definition
├── docs/ # Project Documentation
│ └── thesis_evidence/ # Screenshots & Raw Data
├── telemetry-middleware/ # Python Service (Trace Correlation Core)
│ ├── app/
│ │ ├── main.py # Entry Point
│ │ ├── id_generator.py # Deterministic ID Logic
│ │ ├── routers/ # API Endpoints (Webhooks)
│ │ ├── schemas/ # Data Models (Mappers & Factory)
│ │ └── services/ # Business Logic (State & Processor)
│ └── Dockerfile
├── .gitlab-ci.yml # CI/CD Pipeline Definitions
├── docker-compose.full.yaml # Orchestrates the Full Stack (IDP + Middleware)
├── README.md
└── switch_experiment.sh # Utility script for thesis experiments
- Docker Desktop
- Ngrok to expose local middleware to Jira & GitLab Cloud. You can use other cloud options like Gitpod to get secure links
git clone https://github.com/NewerKey/idp-observ.git
cd idp-observcp .env.example .envThis launches the middleware, jaeger, grafana, ngrok, prometheus and Backstage
docker compose -f docker-compose.full.yaml up -d
The test suite runs directly inside the Docker container to test and validate the middleware implementation for environmental consistency.
- Run all Tests
Execute the full pytest suite inside the running middleware container:
docker exec -it telemetry-middleware pytest
- Run Specific Tests
For example, to run only the thesis hypothesis tests (Control vs. Experimental groups):
docker exec -it telemetry-middleware pytest tests/test_strategies.py -v
- View Coverage
To generate a coverage report inside the container:
docker exec -it telemetry-middleware pytest --cov=app tests/
- Telemetry Middleware: Deep Dive into the application structure, correlation logic and event mapping.
- Thesis Hypothesis: Explanation of the deterministic correlation theory
- Thesis Experiments Runs Documentation and Results
- Thesis Experiments Runs Artifacts
This is a validated proof-of-concept, scoped to a happy-path workflow. Known boundaries (and the production path):
- State is in-memory — a container restart loses Lead Time data. Next: Redis or SQLite/PostgreSQL persistence.
- Trace-ID hashing — resolved. Trace IDs are now keyed with HMAC-SHA256 (
TRACE_ID_HMAC_SECRET), truncated to 128 bits for OTel compatibility, instead of plain MD5 — unguessable without the secret, not just collision-resistant. (The experiment above was measured against MD5; only the hashing algorithm changed, not the correlation technique or its results.) - Webhook auth — resolved, with a correction. The original "Next: HMAC signature verification" line was imprecise — Jira and GitLab don't authenticate webhooks the same way. GitLab sends a static secret in
X-Gitlab-Token(a shared-secret compare, not a cryptographic signature); Jira Cloud's built-in webhooks have no signing mechanism at all, so a customX-Webhook-Secretheader (set via a Jira Automation rule) fills that gap. Both are now enforced — unauthenticated requests get a401. - Entirely local study — the full stack ran in Docker Compose on one machine; the deployment target was the GitLab artifact repo, not a live cloud. No cloud-platform deployment was validated. Next: real deploys to AWS/GCP/Azure/Kubernetes.
- Depends on developer discipline — the Jira key must appear in the branch/commit. Next: git-hook / CI validation to enforce it.
- 2 of 4 DORA metrics (LTC, DF). Next: Change Failure Rate + MTTR via failure/recovery tracking.
- Sync overhead — next: decouple ingestion from processing via a message queue (Kafka/RabbitMQ) + worker (Celery/FastStream).
- Deterministic Tracing: OpenTelemetry Trace Specifications - Understanding standard ID generation vs. the custom correlation implemented in this middleware.
- OTel Extensibility
- DORA Metrics: Google Cloud DevOps Research & Assessment (DORA) - The standard for measuring software delivery performance (Lead Time, Deployment Frequency).
- OpenTelemetry Python: OTel Python SDK Documentation - The library used for generating spans and context propagation.
- Backstage: Backstage.io Documentation - The framework powering the Internal Developer Portal (IDP).
- FastAPI: FastAPI Documentation - The web framework used for the middleware's webhook ingestion.
- Jaeger: Jaeger Tracing - The distributed tracing backend used for visualization.
- Prometheus: Prometheus.io - The monitoring system and time series database used for metrics.
- Ngrok: Ngrok Documentation - Used for exposing the local middleware to public cloud webhooks.
- Docker Compose: Compose Specification - Used for orchestrating the multi-container local stack.
Bachelor's thesis, B.Sc. Software Engineering — "Unified Developer Workflow Observability in Platform Engineering: Automating DORA Metrics via Deterministic Trace Correlation and Unified Signals." Builds on Forsgren et al. (Accelerate), Majors et al. (Observability Engineering), W3C Trace Context, Google's Dapper, and content-based correlation (Aguilera et al., Facebook Canopy).
Author: Yvette Nuerkie Nartey · Berlin