feat(observability): implement universal 4-path OpenTelemetry tracing by chalmerlowe · Pull Request #18433 · googleapis/google-cloud-python

chalmerlowe · 2026-09-21T14:06:23Z

feat(observability): implement universal 4-path OpenTelemetry tracing

Problems Solved

Google Cloud Python client libraries support four communication paths: synchronous gRPC, asynchronous gRPC, synchronous REST (HTTP), and asynchronous REST (HTTP). Previously, distributed OpenTelemetry tracing was only wired for synchronous gRPC calls, leaving asynchronous and HTTP communications untraced. Additionally, earlier drafts of asynchronous tracing attempted to modify gRPC channels after creation, which violated the immutability rules of the underlying Python gRPC library, and did not consistently forward client options across all transport classes.

Solutions

This pull request provides a unified, cross-transport tracing implementation:

Universal 4-Transport Support:
- Synchronous gRPC (GrpcTransport): Continues using OpenTelemetry gRPC channel interceptors.
- Asynchronous gRPC (GrpcAsyncIOTransport): Supplies OpenTelemetry interceptors directly during channel creation, respecting the immutable design of asynchronous gRPC channels.
- Synchronous REST (RestTransport): Adds wire span tracking around HTTP requests with automatic W3C trace context header injection (traceparent).
- Asynchronous REST (AsyncRestTransport): Integrates HTTP wire span tracking and async context lifecycle handling with W3C header propagation.
Refined Transport Contracts & Cleanup:
- Replaced per-instance signature inspection with module-level constants to eliminate overhead during method invocation.
- Configured BaseTransport.kind to safely return an empty string by default instead of raising an exception.
- Updated REST and base transport initializers to accept and store client options, ensuring options are forwarded uniformly across all transport classes.
- Resolved an issue in asynchronous method wrappers where caller-specified compression settings were not correctly evaluated.
- Added task cancellation handling (asyncio.CancelledError) so spans are closed accurately when asynchronous tasks are cancelled.

Notes for Reviewers

This pull request targets template definitions in packages/gapic-generator and core helper functions in packages/google-api-core.
Per team coordination, generated client library code and integration goldens are not included in this change and will be updated in a follow-up step.
All additions maintain backward compatibility and preserve zero overhead when tracing is disabled.

gemini-code-assist

Code Review

This pull request implements OpenTelemetry tracing capabilities across gRPC and REST transports for generated clients. It introduces _observability modules, updates transport constructors to accept client_options, and wires tracing interceptors into the transport layers. The reviewer identified a critical issue in the _shared_macros.j2 template where start_http_span was called with incorrect arguments, which would cause a runtime TypeError. The suggested fix involves passing a properly constructed request object to the tracing utility.

gemini-code-assist · 2026-09-21T14:09:38Z

+        with _observability.start_http_span(
+            client_options,
+            method=method,
+            url=url,
+            url_template=uri,
+            headers=headers,
+            body=body,
+        ) as span:


The current call to _observability.start_http_span passes client_options as the first positional argument (which maps to request in the function signature) and passes method, url, headers, and body as keyword arguments. However, start_http_span does not accept these keyword arguments and expects a request object with those attributes. This will result in a TypeError at runtime when tracing is enabled.

To fix this, dynamically construct a lightweight request object using Python's built-in type() constructor and pass it as the request argument, while correctly passing url_template and client_options as keyword arguments.

with _observability.start_http_span( type("Request", (), {"method": method, "url": url, "headers": headers, "body": body})(), url_template=uri, client_options=client_options, ) as span:

chalmerlowe · 2026-09-21T14:18:09Z

        "DeleteOperation",
        request_type="operations_pb2.DeleteOperationRequest",
        response_type="None",
+        rpc_name="google.longrunning.Operations/DeleteOperation",


Note
As context for the reviewer:

For native methods (like Echo or GetSecret), the generator reads the service's .proto file directly, so constructing the name in the template is straightforward:

method_name="{{ '.'.join(method.meta.address.package) }}.{{ service.name }}/{{ method.name }}"

However, mixins don't live in the service’s proto. Mixin methods (GetOperation, GetIamPolicy, ListLocations) are synthetic—they are injected by the generator from the static catalog in gapic/schema/mixins.py

Without a name attribute, any mixin call (like polling an operation or checking IAM permissions) would be unable to start an OpenTelemetry method span, or would emit an unknown/nameless span that failed our contract checks.

chalmerlowe · 2026-09-21T17:13:27Z


 import abc
+import inspect
 from typing import {% if service.any_extended_operations_methods %}Any, {% endif %}Awaitable, Callable, Dict, Optional, Sequence, Union


The focus for this file is to get fundamental/core elements into the Base class to:

Help eliminate some checks that we were originally considering placing in the Transport classes, etc.

Cut down on some of the boilerplate in the Transport classes

chalmerlowe · 2026-09-22T14:17:49Z

    }
 {% endmacro %}

-{# TODO: This helper logic to check whether `kind` needs to be configured in wrap_method


Comment for Reviewers:

This logic got moved to the base transport.

daniel-sanche

There's a lot I haven't looked at yet, but wanted to get some of my first comments out

daniel-sanche · 2026-09-24T00:02:07Z

+        return self._wrap(gapic_v1.method.wrap_method, _WRAP_METHOD_SUPPORTS_TRACING, func, *args, **kwargs)
+
+    def _wrap_async_method(self, func, *args, **kwargs):
+        return self._wrap(gapic_v1.method_async.wrap_method, _ASYNC_WRAP_METHOD_SUPPORTS_TRACING, func, *args, **kwargs)


I'm finding all these wrappers very hard to follow. Is there any way we can filter these out instead of wrapping, like I suggested in the other PR?

If we do add extra helpers, I don't think we should need these three, along with additional _prep_wrapped_messages and _wrap_method implementations in the async transport. Can we try to simplify the wrapping logic?

EXPLANATION & RESOLVED

There were originally seven functions involved in wrapping transport methods. We cut that to four:

BaseTransport._wrap_method (sync wrapper in base.py.j2)

BaseTransport._wrap_async_method (async wrapper in base.py.j2)

REMOVED: BaseTransport._wrap (internal dispatcher in base.py.j2)
-> We inlined this directly into _wrap_method and _wrap_async_method to remove this layer of indirection. The downside is that we are duplicating about 17 lines of code.

REMOVED: GrpcAsyncIOTransport._wrap_method (override in grpc_asyncio.py.j2)

REMOVED: RestAsyncIOTransport._wrap_method (override in rest_asyncio.py.j2)
-> These overrides were an awkward detour. The async macro in _shared_macros.j2 now calls _wrap_async_method directly on the transport.

The remaining four methods serve clear, distinct purposes and cannot be eliminated. These first two are pre-existing dict initializers that map RPC callables for client dispatch:

BaseTransport._prep_wrapped_messages (precomputes sync method dict in base.py.j2)

Async Transports' _prep_wrapped_messages (precomputes async method dict in _shared_macros.j2)

These two act as the transports's local binding adaptors:

BaseTransport._wrap_method

BaseTransport._wrap_async_method:
They:

Shield against version skew: Check _WRAP_METHOD_SUPPORTS_TRACING to strip tracing keyword args (client_options, kind, method_name, is_streaming) when running against older google-api-core installations that do not accept them.

Inject instance state: Automatically pass self._client_options and self.kind to gapic_v1.method.wrap_method / gapic_v1.method_async.wrap_method across all RPCs.

Keep gapic_v1.method.wrap_method a pure static utility in google-api-core, completely decoupled from transport instance internals.

There is still severe code duplication between prep_wrappd_messages sync/async, and _wrap_method/wrap_method_async. And it seems surprising that in some places, async code is in shared_macros, and sometimes it is baked into the base class. I think there should be a way to generalize this a bit more, so logic doesn't need to be copied in multiple locations

This isn't really related to your observability changes though, so we don't have to spend too much time on this now. But if you want, I can try to suggest a patch for this

chalmerlowe · 2026-09-24T14:41:58Z

+    return span_name, span_attributes, resolved_headers
+
+
+class trace_http_request:


RESOLVED

Good questions/good ideas.

We stripped out the attribute processing code and put it in a separate function: _build_http_span_attributes

We merged the old start_http_span and trace_http_request functions into a single class with context management __entry__ and __exit__ points called trace_http_request.

NOTE: we strayed from custom and named the class using snake_case rather than PascalCase in alignment the principle that most context managers are more aligned with verbs/action versus nouns/objects and to mirror other context managers in the standard library:

contextlib.redirect_stdout

unittest.mock.patch

urllib.request.urlopen

are classes named in snake_case.

nit: For debugging purposes, I'd still suggest giving the class a PascalCase name, but have it accessed trough a snake_case method:

def trace_http_request(*args, **kwargs): return _TraceManager(*args, **kwargs) class _TraceContext: ...

chalmerlowe · 2026-09-28T18:21:47Z

    def host(self):
        return self._host

+    def _wrap_method(self, func, *args, **kwargs):


@ohmayr
Following up on our discussion:

We created _wrap_method in base.py primarily because google-api-core and the generated client libraries are released independently.

Preventing TypeError on older environments:
If newly generated client code passed our new tracing arguments (client_options, kind, method_name) directly into wrap_method, any user running the library with an older version of google-api-core would immediately hit TypeError: wrap_method() got an unexpected keyword argument. We can’t fix that inside wrap_method because older versions of google-api-core are already deployed in the wild.

Serving as a runtime bridge:
Sitting inside the generated transport, _wrap_method checks once at module import whether the installed google-api-core understands tracing. If it does, it passes the tracing configuration through; if it doesn’t, it cleanly strips those new kwargs so existing RPC calls continue working without error.

Keeping generated code DRY:
The transport instance already knows its own _client_options (tracer providers) and kind ("grpc", "rest", etc.). Having _wrap_method bind those automatically means the code generator doesn't have to repeat that boilerplate across every single RPC in the service's method table.

daniel-sanche · 2026-09-28T23:00:38Z

+        # The fallback below strips tracing-specific arguments when an older version
+        # of google-api-core is installed (which does not accept client_options, etc.).
+        for k in ["client_options", "method_name", "is_streaming", "kind"]:
+            kwargs.pop(k, None)


It looks like this change puts kind behind the _WRAP_METHOD_SUPPORTS_TRACING flag. But isn't kind already in use? Wouldn't stripping it here cause issues for async rest?

I agree with this. If we pop kind like this, we lose the logic where we call wrap_errors for grpc (by checking kind) which will result in a bug.

RESOLVED:

So this got complicated quick.
Turns out, since we support google-api-core all the way back to 2.14-ish...

2.14 to 2.19-ish no support for kind OR any tracing arguments.
In 2.19-ish Omair added support for kind to distinguish between gRPC and REST and prevent REST callables from being wrapped with gRPC callables, but still no support for tracing.
Now, in 2.36 we are introducing client_options, method_name, is_streaming AND continuing the original use of kind.

So we have three different scenarios to support when trying to pass args back to google-api-core and avoid a TypeError.

This now adapts to any of those situations.

NOTE: @daniel-sanche @ohmayr

I would like to move all of this into _compat and just have a three line stub here in Transport to call _compat._wrap_method OR _compat._wrap_async_method.

That will cut down on a lot of duplicate boilerplate and seems like the right place to deal with these version-specific intricacies.

I don't really wanna do that in this PR. I would rather focus as much as we can on bringing this PR to a close and doing a fast follow OR stripping this out of this PR and doing it separately.

I created an issue to track the overhaul and deduplication of all things wrap.

daniel-sanche · 2026-09-28T23:04:21Z

+        return self._wrap(gapic_v1.method.wrap_method, _WRAP_METHOD_SUPPORTS_TRACING, func, *args, **kwargs)
+
+    def _wrap_async_method(self, func, *args, **kwargs):
+        return self._wrap(gapic_v1.method_async.wrap_method, _ASYNC_WRAP_METHOD_SUPPORTS_TRACING, func, *args, **kwargs)


There is still severe code duplication between prep_wrappd_messages sync/async, and _wrap_method/wrap_method_async. And it seems surprising that in some places, async code is in shared_macros, and sometimes it is baked into the base class. I think there should be a way to generalize this a bit more, so logic doesn't need to be copied in multiple locations

This isn't really related to your observability changes though, so we don't have to spend too much time on this now. But if you want, I can try to suggest a patch for this

daniel-sanche · 2026-09-28T23:13:08Z

+    return span_name, span_attributes, resolved_headers
+
+
+class trace_http_request:


nit: For debugging purposes, I'd still suggest giving the class a PascalCase name, but have it accessed trough a snake_case method:

def trace_http_request(*args, **kwargs): return _TraceManager(*args, **kwargs) class _TraceContext: ...

chalmerlowe · 2026-09-29T17:23:29Z

@daniel-sanche I concur that we can prolly do more to reduce some duplication, but I believe some of this is in main. I don't wanna lose focus on getting this PR merged, so let's defer this action, I have created an Issue to track this.

There is still severe code duplication between prep_wrappd_messages sync/async, and _wrap_method/wrap_method_async. And it seems surprising that in some places, async code is in shared_macros, and sometimes it is baked into the base class. I think there should be a way to generalize this a bit more, so logic doesn't need to be copied in multiple locations

This isn't really related to your observability changes though, so we don't have to spend too much time on this now. But if you want, I can try to suggest a patch for this

chalmerlowe · 2026-09-29T20:16:42Z

        self._start_span_fn = None
        if (
            not is_streaming
-            and kind == "grpc"
+            and kind in self._SUPPORTED_TRACING_KINDS


Note
Comment for reviewers:
@ohmayr asked a question in another venue that was something like this "do we need to check these? the values are coming from the GAPIC layer so aren't they already constrained?"

wrap_method and method_async.wrap_method are public APIs in google-api-core, exposed to handwritten clients, custom transports, and test mocks (not just generator code).

Public API Contract & Guardrails:
Because callers can pass arbitrary inputs to kind (or inherit BaseTransport.kind == ""), we need an explicit whitelist of supported tracing kinds. Without it, an arbitrary or mock transport (kind="mock", kind="custom") would fall through and blindly emit OpenTelemetry spans with "rpc.system.name" set to "grpc", violating semantic conventions and generating dirty telemetry.

Controlled Scope:
We only want method tracing to initialize when we know the transport's wire semantics and how to map its attributes properly. Whitelisting supported kinds acts as a fail-safe boundary.

Clean Polymorphism:
Because _AsyncGapicCallable inherits from _GapicCallable, leveraging _SUPPORTED_TRACING_KINDS across the classes allows us to enforce this contract gracefully—letting sync allow ("grpc", "rest") and async allow ("grpc_asyncio", "rest_asyncio") using standard object-oriented design without duplicating the initialization or span-creation logic.

ohmayr · 2026-10-05T15:21:12Z

+_WRAP_METHOD_SUPPORTS_KIND = (
+    "kind" in inspect.signature(gapic_v1.method.wrap_method).parameters
+)
+_ASYNC_WRAP_METHOD_SUPPORTS_TRACING = (
+    "client_options" in inspect.signature(gapic_v1.method_async.wrap_method).parameters
+)
+_ASYNC_WRAP_METHOD_SUPPORTS_KIND = (
+    "kind" in inspect.signature(gapic_v1.method_async.wrap_method).parameters


have we considered the performance impact of calling inspect.signature?

RESOLVED

@ohmayr
Good catch! Calling inspect.signature 4 times per service module at import time added unnecessary import latency across generated libraries and forced import inspect into transport modules.

We eliminated inspect.signature completely:

Capability checks have also been moved into _compat.py:

WRAP_METHOD_SUPPORTS_TRACING is detected via an
O(1) attribute check (hasattr(_observability, "trace_http_request")), unified with the import of trace_http_request.

ASYNC_WRAP_METHOD_SUPPORTS_KIND checks WRAP_METHOD_SUPPORTS_TRACING or hasattr(gapic_v1.method_async, "_DEFAULT_ASYNC_TRANSPORT_KIND")

Transport base.py imports these boolean constants directly from ._compat.

Added comments in _compat.py documenting the minimum google-api-core versions (>= 2.36.0 and >= 2.29.0) required before these checks can be retired.

daniel-sanche

Looking close, but a few more comments

daniel-sanche · 2026-10-05T22:27:46Z

+                span_name,
+                kind=trace.SpanKind.CLIENT,
+                attributes=span_attributes,
+            )


Should we be setting record_exception=False and set_status_on_exception=False here?

I see we're doing some manual error recording in the __exit__ method here, so do we still need the build-in error handling?

daniel-sanche · 2026-10-05T22:56:05Z

-    response = {{ await_prefix }}getattr(session, method)(
-        "{host}{uri}".format(host=host, uri=uri),
-        timeout=timeout,
+    url = "{host}{uri}".format(host=host, uri=uri)


This is supposed to be a url_template (i.e., arguments replaces with placeholders). But it looks like this might be the full expanded URL?

daniel-sanche · 2026-10-05T23:01:01Z

+
+    # Temporarily set the ambient global tracer provider
+    original_provider = trace.get_tracer_provider()
+    trace.set_tracer_provider(global_provider)


Can we mokey-patch this? It seems like this modifies some global state, which can only be set once per process. I could see this causing hard-to-catch issues in the future

daniel-sanche · 2026-10-05T23:08:11Z

+        apply_interceptors = getattr(
+            grpc_helpers,
+            "apply_channel_interceptors",
+            lambda channel, interceptors: channel,


Could it be a problem that this does nothing on old versions of api_core?

Maybe we should add an implementation to _compat, like async. Or at least mention in the docstring that it requires a certain version of api_core. Or am I overthinking this?

…gapic-generator

…error recording, and templates - Hoist sync channel interceptors (apply_channel_interceptors) and async channel interceptors (apply_async_channel_interceptors) to _compat with active fallback wrapping, eliminating the noop lambda fallback in grpc.py.j2 - Hoist wrap_method tracing and kind support constants to _compat and eliminate inspect.signature module-load checks in base transports - Adopt hybrid error recording in _TraceContext.record_error to enrich span attributes (error.type, status.message) while allowing upstream OpenTelemetry defaults to handle exception events - Pass parameterized proto route template from http_options into trace_http_request url_template - Add rest_transport.kind assertion for AsyncResumableUploadServiceRestTransport to maintain 100% test coverage - Add unit test isolation for custom tracer provider and test cases for _compat interceptors

…d otel tracing

gemini-code-assist Bot reviewed Sep 21, 2026

View reviewed changes

chalmerlowe commented Sep 21, 2026

View reviewed changes

chalmerlowe self-assigned this Sep 21, 2026

chalmerlowe changed the title ~~feat(observability): implement universal 4-path OpenTelemetry tracing~~ feat(observability): [WIP] implement universal 4-path OpenTelemetry tracing Sep 21, 2026

chalmerlowe commented Sep 21, 2026

View reviewed changes

chalmerlowe commented Sep 22, 2026

View reviewed changes

daniel-sanche reviewed Sep 24, 2026

View reviewed changes

chalmerlowe commented Sep 24, 2026

View reviewed changes

chalmerlowe mentioned this pull request Sep 25, 2026

test(generator): sandbox verification of spec-driven telemetry compliance tests #18468

Draft

chalmerlowe marked this pull request as ready for review September 28, 2026 17:07

chalmerlowe requested a review from a team as a code owner September 28, 2026 17:07

chalmerlowe commented Sep 28, 2026

View reviewed changes

daniel-sanche reviewed Sep 28, 2026

View reviewed changes

chalmerlowe commented Sep 29, 2026

View reviewed changes

chalmerlowe added this to the [o11y] Tracing milestone Sep 30, 2026

chalmerlowe requested review from a team as code owners October 1, 2026 11:41

chalmerlowe requested review from shuoweil and removed request for a team October 1, 2026 11:41

chalmerlowe mentioned this pull request Oct 1, 2026

refactor(gapic-generator): deduplicate sync/async prep_wrapped_messages and transport method wrapping #18536

Open

chalmerlowe commented Oct 1, 2026

View reviewed changes

Comment thread ci/get_package_shards.py Outdated

chalmerlowe changed the title ~~feat(observability): [WIP] implement universal 4-path OpenTelemetry tracing~~ feat(observability): implement universal 4-path OpenTelemetry tracing Oct 1, 2026

ohmayr reviewed Oct 5, 2026

View reviewed changes

daniel-sanche reviewed Oct 6, 2026

View reviewed changes

chalmerlowe force-pushed the feat/otel-tracing-universal-4path branch 2 times, most recently from 4aee356 to 0566e94 Compare October 6, 2026 15:21

feat(observability): universal 4-path tracing in google-api-core and …

699f869

…gapic-generator

chalmerlowe force-pushed the feat/otel-tracing-universal-4path branch from 0566e94 to 4711333 Compare October 6, 2026 16:07

test(gapic): update bazel integration goldens for resumable upload an…

1ffdedb

…d otel tracing

chalmerlowe force-pushed the feat/otel-tracing-universal-4path branch from 4711333 to 1ffdedb Compare October 6, 2026 17:33

		return span_name, span_attributes, resolved_headers


		class trace_http_request:

Conversation

chalmerlowe commented Sep 21, 2026

Problems Solved

Solutions

Notes for Reviewers

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

daniel-sanche Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chalmerlowe Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chalmerlowe commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Sep 21, 2026 •

edited

Loading

daniel-sanche Sep 24, 2026 •

edited

Loading

chalmerlowe Sep 24, 2026 •

edited

Loading

chalmerlowe Sep 24, 2026 •

edited

Loading

daniel-sanche Sep 28, 2026 •

edited

Loading

chalmerlowe Sep 29, 2026 •

edited

Loading

chalmerlowe Sep 29, 2026 •

edited

Loading

chalmerlowe commented Sep 29, 2026 •

edited

Loading