Tracing does not work with mTLS token in AdkApp on Agent Runtime

[gwendal-lecren]

🔴 Required Information

Describe the Bug:
We are trying to deploy an ADK agent with an AdkApp wrapper to Agent Runtime, using Agent Identity instead of service account.
When we enable logging of prompt inputs and response outputs, we run into the issue described in this documentation:
https://docs.cloud.google.com/iam/docs/troubleshoot-auth-manager#401-error
Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

When we set GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES to "false", then it works.

It seems that the default built-in telemetry setup in AdkApp fails with certificate-bound mTLS access tokens.

NB: we followed this documentation https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-adk.

Steps to Reproduce:
Please provide a numbered list of steps to reproduce the behavior:

1. Deploy an agent with agent identity
2. Follow this documentation to setup tracing: https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-adk and that one: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/tracing#write-traces

Expected Behavior:
Input prompts and output answers should be logged properly

Observed Behavior:
Running in 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

Environment Details:

• ADK Library Version (pip show google-adk): 2.3.0
• Desktop OS:** [e.g., macOS, Linux, Windows] N/A
• Python Version (python -V): 3.14

Model Information:

• Are you using LiteLLM: No
• Which model is being used: gemini-2.5-pro

---

🟡 Optional Information

Regression:
Did this work in a previous version of ADK? If so, which one? No

Logs:

{

 "textPayload": "Error while writing to Cloud Logging\nTraceback (most recent call last):\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/grpc_helpers.py\", line 55, in error_remapped_callable\n  return callable_(*args, **kwargs)\n File \"/code/.venv/lib/python3.14/site-packages/grpc/_interceptor.py\", line 276, in __call__\n  response, ignored_call = self._with_call(\n               ~~~~~~~~~~~~~~~^\n    request,\n    ^^^^^^^^\n  ...<4 lines>...\n    compression=compression,\n    ^^^^^^^^^^^^^^^^^^^^^^^^\n  )\n  ^\n File \"/code/.venv/lib/python3.14/site-packages/grpc/_interceptor.py\", line 331, in _with_call\n  return call.result(), call\n      ~~~~~~~~~~~^^\n File \"/code/.venv/lib/python3.14/site-packages/grpc/_channel.py\", line 438, in result\n  raise self\n File \"/code/.venv/lib/python3.14/site-packages/grpc/_interceptor.py\", line 314, in continuation\n  response, call = self._thunk(new_method).with_call(\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n    request,\n    ^^^^^^^^\n  ...<4 lines>...\n    compression=new_compression,\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  )\n  ^\n File \"/code/.venv/lib/python3.14/site-packages/grpc/_channel.py\", line 1173, in with_call\n  return _end_unary_response_blocking(state, call, True, None)\n File \"/code/.venv/lib/python3.14/site-packages/grpc/_channel.py\", line 990, in _end_unary_response_blocking\n  raise _InactiveRpcError(state) # pytype: disable=not-instantiable\n  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ngrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAUTHENTICATED\n\tdetails = \"Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\"\n\tdebug_error_string = \"UNAUTHENTICATED:Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\"\n>\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/code/.venv/lib/python3.14/site-packages/opentelemetry/exporter/cloud_logging/__init__.py\", line 463, in _write_log_entries_to_client\n  client.write_log_entries(\n  ~~~~~~~~~~~~~~~~~~~~~~~~^\n    WriteLogEntriesRequest(entries=batch, partial_success=True)\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  )\n  ^\n File \"/code/.venv/lib/python3.14/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py\", line 1027, in write_log_entries\n  response = rpc(\n    request,\n  ...<2 lines>...\n    metadata=metadata,\n  )\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/gapic_v1/method.py\", line 128, in __call__\n  return wrapped_func(*args, **kwargs)\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_unary.py\", line 294, in retry_wrapped_func\n  return retry_target(\n    target,\n  ...<3 lines>...\n    on_error=on_error,\n  )\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_unary.py\", line 156, in retry_target\n  next_sleep = _retry_error_helper(\n    exc,\n  ...<6 lines>...\n    timeout,\n  )\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_base.py\", line 216, in _retry_error_helper\n  raise final_exc from source_exc\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_unary.py\", line 147, in retry_target\n  result = target()\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/timeout.py\", line 130, in func_with_timeout\n  return func(*args, **kwargs)\n File \"/code/.venv/lib/python3.14/site-packages/google/api_core/grpc_helpers.py\", line 57, in error_remapped_callable\n  raise exceptions.from_grpc_error(exc) from exc\ngoogle.api_core.exceptions.Unauthenticated: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",

 "insertId": "XXX",
 "resource": {
  "type": "aiplatform.googleapis.com/ReasoningEngine",
  "labels": {
   "location": "europe-west1",
   "reasoning_engine_id": "XXX",
   "resource_container": "XXX"
  }
 },
 "timestamp": "2026-07-04T16:11:55.229479Z",
 "severity": "ERROR",
 "labels": {
  "python_logger": "root"
 },
 "logName": "projects/XXX/logs/agent-logs",
 "sourceLocation": {
  "file": "/code/.venv/lib/python3.14/site-packages/opentelemetry/exporter/cloud_logging/__init__.py",
  "line": "468",
  "function": "_write_log_entries_to_client"
 },
 "receiveTimestamp": "2026-07-04T16:11:55.277809948Z"
}

How often has this issue occurred?:

• Always (100%)

[surajksharma07]

@gwendal-lecren this looks isolated to _get_agent_engine_logs_exporter in google_cloud.py: whenever OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental is set, it opens a raw gRPC channel straight off your credentials with no mTLS wiring, unlike the span exporter which already handles cert-bound tokens.
That's why traces work and logs don't.

There's already a change on main that drops that path in favor of stdout logging — could you try pulling adk-python from main (or unsetting that env var as a stopgap) instead of disabling GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES, and confirm it clears the 401 for you?

[gwendal-lecren]

Thanks @surajksharma07 - FYI I've just tried using git+https://github.com/google/adk-python.git@main#egg=google-adk (instead of google-adk==2.3.0) in the requirements.txt, but still running into the 401 unfortunately.