feat(tracing): add PostgreSQL tracing spans

[ThePumpingLemma]

Why

Expose PostgreSQL datastore latency within existing request traces so slow logical database operations can be identified without recording tenant data or query arguments.

What

• Add client spans around logical PostgreSQL operations across the database facade, search, audit, replica fencing, and command persistence
• Use a dedicated buzz_datastore target and db.system.name = "postgresql" for filtering and backend classification
• Exclude health-check database calls and scrub raw identifiers and errors from newly traced paths

Risk Assessment

Medium — this instruments frequently used datastore paths and increases trace volume when enabled, but does not change SQL execution or datastore behavior. Existing OpenTelemetry filtering controls export.

References

• Pre-push clippy and fast unit-test hooks passed

Generated with Amp

[ThePumpingLemma]

Some additional research on more detailed alternatives:

Buzz PostgreSQL trace correlation options

Decision being considered

Buzz currently has logical datastore spans around operations such as
insert_event, query_events, and audit_log. These show how much time a Buzz
operation spends in the datastore, but they do not identify or time each SQL
statement independently.

Two possible ways to add query-level visibility are:

1. Create an OpenTelemetry client span around every SQL query.
2. Add SQLCommenter support to SQLx so each sampled query carries W3C trace
   context in a SQL comment.

These approaches solve related but different problems. Query spans describe
application-side execution and appear in an APM flame graph. SQLCommenter sends
trace identity to PostgreSQL so database-side telemetry can be associated with
the application trace.

Option 1: manually add a span around every query

Resulting trace shape

ws.event
└── handle_event
    └── insert_event
        ├── postgres.query: INSERT events
        ├── postgres.query: UPDATE thread counters
        └── postgres.query: INSERT mentions

Each query span would normally be a CLIENT span with attributes such as:

db.system.name = postgresql
db.operation.name = INSERT
db.query.summary = INSERT events
db.query.text = <parameterized SQL, if approved>

Bind values, community IDs, event IDs, user data, and other high-cardinality or
sensitive values should not be recorded.

Advantages

• Gives exact application-side duration and errors for each SQL operation.
• Produces the conventional APM model expected by Datadog and other tracing
  backends.
• Preserves SQLx prepared-statement caching because the SQL sent to PostgreSQL
  is unchanged.
• Naturally nests query work beneath Buzz's logical datastore spans.
• Can include returned-row counts, affected-row counts, and SQLSTATE without
  exposing bind values.
• Works independently of PostgreSQL logging and DBM query sampling.
• Does not require database-side understanding of a propagation format.

Disadvantages

• Adding spans manually at every call site is a large and error-prone change.
  Buzz has hundreds of query, query_as, query_scalar, and QueryBuilder
  executions spread across pools, acquired connections, and transactions.
• New queries can easily be added without instrumentation.
• Every query-site wrapper must correctly handle errors, streaming results,
  cancellation, and early stream drops.
• It creates more spans. A single logical operation can execute several SQL
  statements, and high-traffic requests can therefore generate many child
  spans.
• Capturing or normalizing SQL adds CPU, allocation, telemetry payload, privacy,
  and cardinality costs.
• Query span names and attributes must remain low-cardinality and follow a
  consistent semantic convention.
• Application-side head sampling is needed to avoid constructing and exporting
  query spans for every production request.

Prior art

• SeaORM has an opt-in tracing-spans feature that creates one span per ORM
  operation.
• sqlx-otel and sqlx-tracing wrap SQLx executors to create query spans
  automatically rather than requiring manual query-site spans.
• SQLx itself sees every query in its driver and emits completion events through
  QueryLogger, but those are events rather than independently timed spans.

The strongest implementation would therefore not be literally manual. It would
add a SQLx-native hook or a carefully reviewed executor wrapper that covers
pools, connections, transactions, and streams centrally. The existing public
SQLx wrappers are currently young, have little visible adoption, and target
SQLx 0.8 rather than Buzz's SQLx 0.9. sqlx-otel also targets OpenTelemetry
0.31 while Buzz uses 0.32.

Option 2: add SQLCommenter propagation to SQLx

Resulting SQL

For a sampled query, SQLx would append the active W3C context:

SELECT * FROM events WHERE community_id = $1 AND id = $2
/*traceparent='00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01'*/

PostgreSQL treats the suffix as a comment, but database logs, activity samples,
and compatible monitoring products can observe it.

Advantages

• Carries exact trace identity across the application/database boundary.
• Can connect a PostgreSQL-side query sample, wait event, execution plan, or log
  entry to the originating trace when the monitoring system understands the
  SQLCommenter field.
• Uses the vendor-neutral W3C traceparent representation.
• OpenTelemetry's database semantic conventions describe SQLCommenter as an
  optional context propagation mechanism.
• A SQLx-level implementation could apply consistently without manually editing
  every query execution.
• The comment need not contain community IDs, user data, bind values, or other
  application identifiers.

Disadvantages

• OpenTelemetry currently marks database SQLCommenter propagation as
  Development and says it should not be enabled by default.
• Datadog must be verified to consume generic W3C SQLCommenter fields for the
  PostgreSQL DBM path in use. OpenTelemetry standardizing the comment does not
  guarantee a particular vendor correlates it.
• SQLx does not expose a per-statement query rewrite hook, so this cannot be
  implemented cleanly as a pool configuration option today.
• A unique traceparent changes the complete SQL string. SQLx keys its prepared
  statement cache by that string, which would otherwise produce effectively one
  cached statement per trace.
• Setting .persistent(false) avoids unbounded statement-cache growth, but
  forces sampled queries to be parsed, described, prepared, executed, and then
  closed rather than reusing a cached prepared statement.
• Comments increase wire bytes and can increase PostgreSQL and database-log
  volume.
• Query aggregation and normalization must strip or ignore the dynamic comment.
• SQLCommenter alone does not create an application-side query span. It can
  correlate database telemetry to a trace, but it does not add a timed query bar
  to the APM flame graph unless query spans are also created.
• Tail sampling in a collector happens too late to avoid SQL mutation and
  prepare overhead. Buzz must make the sampling decision before sending SQL.

Sampling behavior

SQL comments should only be injected when the active span context is valid and
sampled. With application-side parent-based ratio sampling:

Context	Inject comment	Preserve normal SQLx statement caching
Sampled	Yes	No, unless SQLx gains specialized support
Unsampled	No	Yes
Missing or invalid	No	Yes

This makes application-side sampling the relevant cost control. Collector-only
sampling cannot prevent query rewriting or PostgreSQL parse/prepare work.

Prior art

Prisma Engines is the strongest Rust implementation. Its Rust query engine
injects sampled W3C traceparent comments through its tokio-postgres-based
Quaint layer. Prisma added a specialized tracing cache that:

1. Removes the dynamic trace comment when deriving the metadata cache key.
2. Reuses parameter and result metadata for the stable underlying SQL.
3. Sends the comment-bearing SQL for each execution.
4. Avoids retaining a distinct prepared statement for every trace ID.

That is substantially better than simply disabling persistence, but Prisma
controls its PostgreSQL dispatch layer. Reproducing this correctly in Buzz would
likely require an upstream SQLx feature or maintaining changes against SQLx
internals.

Direct comparison

Concern	Per-query spans	SQLCommenter in SQLx
APM flame-graph timing	Yes	No, not by itself
PostgreSQL-side trace identity	No	Yes
Exact DBM query-sample correlation	Possible through span/query matching	Possible if DBM consumes the comment
Changes SQL sent to PostgreSQL	No	Yes
Prepared-statement cache impact	None	Severe without specialized handling
Application code impact	Large if manual; moderate with an executor wrapper	Small only if implemented inside SQLx; otherwise query construction must be centralized
Span volume	One additional span per sampled query	None by itself
Query parsing overhead	Attribute parsing/normalization only	Potential PostgreSQL re-prepare for every sampled execution
Sampling requirement	Recommended for span/export volume	Required to control SQL/cache overhead
Privacy concern	Query text and span attributes	Context stored in SQL/logs; no application data is required
Current Rust maturity	Established model; automatic SQLx wrappers are young	One strong Prisma implementation; no general SQLx support

Recommendation for Buzz

Do not add either mechanism to the current PostgreSQL datastore-spans PR.

The current logical spans provide useful operation-level latency with a small,
reviewable behavior change and no effect on query execution. SQLx's built-in
sqlx::query completion events can provide slow-query details correlated through
Buzz's trace-aware logs without introducing query spans or changing SQL.

If production traces show that logical operation timing is insufficient:

1. Prefer a SQLx-native or executor-level per-query span implementation over
   manually instrumenting every query site.
2. Require compatibility with SQLx 0.9 and Buzz's OpenTelemetry version.
3. Ensure the unsampled path skips SQL cloning, normalization, attribute-vector
   construction, and metrics unless explicitly enabled.
4. Start with parameterized query text disabled or privacy-reviewed, and use
   stable query names/summaries.
5. Benchmark sampled and unsampled request paths before broad rollout.

Consider SQLCommenter later only if all of the following are true:

1. Datadog's PostgreSQL DBM integration is proven to consume the W3C comment in
   Buzz's deployment.
2. SQLx supports trace-aware execution without fragmenting its prepared
   statement cache, preferably through an upstream feature modeled on Prisma's
   metadata-cache design.
3. Buzz uses application-side sampling and injects comments only for sampled
   contexts.
4. Database query grouping, logging volume, privacy, and parse/prepare overhead
   have been measured in staging.

In short: logical datastore spans now; SQLx slow-query events for diagnosis;
automatic query spans only if the additional APM detail proves necessary; and
SQLCommenter only after Datadog compatibility and prepared-statement behavior
are solved.

References

• OpenTelemetry database client spans and SQLCommenter:
  https://opentelemetry.io/docs/specs/semconv/db/database-spans/#sql-commenter
• SQLCommenter specification: https://google.github.io/sqlcommenter/spec/
• Datadog OpenTelemetry trace/DBM correlation:
  https://docs.datadoghq.com/opentelemetry/correlate/dbm_and_traces/
• Prisma Engines: https://github.com/prisma/prisma-engines
• SeaORM tracing spans:
  https://github.com/SeaQL/sea-orm/blob/master/src/database/tracing_spans.rs
• sqlx-otel: https://github.com/chmodas/sqlx-otel
• sqlx-tracing: https://github.com/jdrouet/sqlx-tracing
• SQLx QueryLogger:
  https://github.com/launchbadge/sqlx/blob/main/sqlx-core/src/logger.rs