feat(gapic): add OpenTelemetry T3 client method span wrapping in gapic_v1.method (D) by chalmerlowe · Pull Request #18274 · googleapis/google-cloud-python

chalmerlowe · 2026-09-03T13:50:42Z

Summary

This PR implements Tier 3 (T3) Client Method Spans in google-api-core:

In google.api_core.gapic_v1.method._GapicCallable.__call__:
- Starts an OpenTelemetry SpanKind.CLIENT span representing the high-level GAPIC SDK method call (e.g. google.cloud.secretmanager.v1.SecretManagerService/ListSecrets).
- Sets standard T3 attributes (rpc.system = "grpc", rpc.service, rpc.method).
- Encompasses client preparation, retry loops, timeouts, and error handling.
- Automatically establishes context propagation so underlying wire-level transport (T4) spans attach as children under this parent span.
- Records exceptions and sets span error status if the method terminates with an unhandled exception.
Adds comprehensive unit tests in packages/google-api-core/tests/unit/gapic/test_method.py.

Reference:

  [Parent T3 Span] google.cloud.secretmanager.v1.SecretManagerService/ListSecrets (330.37 ms)
     └── [Child T4 Wire Span] /google.cloud.secretmanager.v1.SecretManagerService/ListSecrets (330.08 ms)

gemini-code-assist

Code Review

This pull request adds OpenTelemetry tracing support to _GapicCallable in google/api_core/gapic_v1/method.py and includes unit tests to verify the tracing behavior. The review feedback identifies a critical reliability issue: catching only ImportError during OTel setup could allow other exceptions to crash the API call, violating the self-contained fallback rule. Additionally, broadening the exception handling introduces a double execution risk for the wrapped function if it fails. A robust solution using a state flag is proposed to safely handle setup errors while ensuring wrapped function exceptions propagate correctly.

gemini-code-assist · 2026-09-03T13:52:40Z

+        if _observability.is_otel_capabilities_enabled():
+            try:
+                from opentelemetry import trace
+
+                tracer = trace.get_tracer("google.api_core")
+                raw_method = getattr(self._target, "_method", None)
+                if raw_method and isinstance(raw_method, (str, bytes)):
+                    if isinstance(raw_method, bytes):
+                        raw_method = raw_method.decode("utf-8")
+                    method_str = raw_method.lstrip("/")
+                    service, _, method = method_str.rpartition("/")
+                    span_name = method_str
+                else:
+                    service = "google.api_core"
+                    method = getattr(self._target, "__name__", "call")
+                    span_name = f"{service}/{method}"
+
+                with tracer.start_as_current_span(
+                    span_name,
+                    kind=trace.SpanKind.CLIENT,
+                    attributes={
+                        "rpc.system": "grpc",
+                        "rpc.service": service,
+                        "rpc.method": method,
+                    },
+                ) as span:
+                    try:
+                        return wrapped_func(*args, **kwargs)
+                    except Exception as exc:
+                        span.record_exception(exc)
+                        span.set_status(trace.StatusCode.ERROR, str(exc))
+                        raise
+            except ImportError:
+                pass


Critical Reliability & Correctness Issues

Resilience of Fallback Logic: Currently, only ImportError is caught. If any other exception occurs during OpenTelemetry setup, tracer retrieval, or span creation (e.g., AttributeError, TypeError, UnicodeDecodeError when decoding _method, or OTel initialization/configuration errors), it will propagate and crash the user's API call. Per the Repository Style Guide (Rule 3: Self-Contained Fallbacks), fallback logic must be resilient and self-contained, bypassing failures gracefully.

Double Execution Risk: If we simply broaden the exception handler to except Exception:, any exception raised by wrapped_func (which is caught and re-raised by the inner except Exception as exc) would be caught by the outer except Exception and trigger a second execution of wrapped_func(*args, **kwargs). This is a critical bug that could cause non-idempotent RPCs to be executed twice.

Solution

We can use a state flag (func_called) to track whether wrapped_func has been invoked. This allows us to catch all exceptions during OTel setup/span creation and fallback gracefully, while ensuring that any exception raised by wrapped_func itself is propagated immediately without triggering a double execution.

if _observability.is_otel_capabilities_enabled(): func_called = False try: from opentelemetry import trace tracer = trace.get_tracer("google.api_core") raw_method = getattr(self._target, "_method", None) if raw_method and isinstance(raw_method, (str, bytes)): if isinstance(raw_method, bytes): raw_method = raw_method.decode("utf-8") method_str = raw_method.lstrip("/") service, _, method = method_str.rpartition("/") span_name = method_str else: service = "google.api_core" method = getattr(self._target, "__name__", "call") span_name = f"{service}/{method}" with tracer.start_as_current_span( span_name, kind=trace.SpanKind.CLIENT, attributes={ "rpc.system": "grpc", "rpc.service": service, "rpc.method": method, }, ) as span: try: func_called = True return wrapped_func(*args, **kwargs) except Exception as exc: span.record_exception(exc) span.set_status(trace.StatusCode.ERROR, str(exc)) raise except Exception: if func_called: raise

References

Rule 3: Self-Contained Fallbacks - Fallback logic must be resilient and self-contained. Always wrap fallback configuration loading in try-except blocks to catch expected exceptions and bypass failures gracefully. ^(link)

chalmerlowe · 2026-09-03T14:57:09Z

+                        span.record_exception(exc)
+                        span.set_status(trace.StatusCode.ERROR, str(exc))
+                        raise
+            except ImportError:


Note
This is purposeful minimal error handling to allow us to discuss the overall approach, without getting bogged down in the minutiae.
With approval of the approach, I will update the error handling and tests.

daniel-sanche · 2026-09-04T16:51:28Z

+                else:
+                    service = "google.api_core"
+                    method = getattr(self._target, "__name__", "call")
+                    span_name = f"{service}/{method}"


This is the hot path that is called on every rpc. It seems like some of this would be doing the same (possibly slow) calculation on each invocation, right? Can we move that logic into the one-time init call?

daniel-sanche · 2026-09-04T16:51:45Z

+                    service = "google.api_core"
+                    method = getattr(self._target, "__name__", "call")
+                    span_name = f"{service}/{method}"
+


helper methods would be useful here

daniel-sanche · 2026-09-04T16:55:15Z

+                from opentelemetry import trace
+
+                tracer = trace.get_tracer("google.api_core")
+                raw_method = getattr(self._target, "_method", None)


have you tested this against a real client yet? IIRC, there are multiple layers of wrapping, so this may not be exposed the way you expect

I wonder if it would be possible to pass down a method name, instead of trying to extract it? The generator already knows it when calling _prep_wrapped_messages

daniel-sanche · 2026-09-04T17:02:00Z

+                    },
+                ) as span:
+                    try:
+                        return wrapped_func(*args, **kwargs)


There are two places this function can call into wrapped_function. If errors line up the wrong way, it could hit both. We need to be extra careful to avoid double invocation here, because that would be a very serious bug

It might be better to call wrapped_func a single time at the end of the method, but use a no-op context manager instead of the tracer if we can't get one

daniel-sanche · 2026-09-04T17:02:32Z

+            try:
+                from opentelemetry import trace
+
+                tracer = trace.get_tracer("google.api_core")


Do we need to create this on every invocation? Can we cache it for each request? Or even use a singleton shared across all instances?

daniel-sanche · 2026-09-04T17:04:01Z

+                    span_name,
+                    kind=trace.SpanKind.CLIENT,
+                    attributes={
+                        "rpc.system": "grpc",


IIRC, this wrapper is also used by HTTP. So we should try to gate this for now

daniel-sanche · 2026-09-04T17:05:21Z

+            try:
+                from opentelemetry import trace
+
+                tracer = trace.get_tracer("google.api_core")


how does this connect with the client's tracer provider? Is that coming later?

daniel-sanche · 2026-09-04T17:10:52Z

+                    },
+                ) as span:
+                    try:
+                        return wrapped_func(*args, **kwargs)


Note that this span won't be very meaningful for streaming rpcs, because it just tracks the stream set-up, not any of the data flow. I remember asking Wes about streaming, and he said it's out of scope.

We should check with Blake if he wants to track stream init like this, or if we should avoid recording any data for streaming rpcs

…ethod tracing

gemini-code-assist Bot reviewed Sep 3, 2026

View reviewed changes

chalmerlowe added this to the [o11y] Tracing milestone Sep 3, 2026

chalmerlowe changed the title ~~feat(gapic): add OpenTelemetry T3 client method span wrapping to gapic_v1.method~~ feat(gapic): add OpenTelemetry T3 client method span wrapping to gapic_v1.method (D) Sep 3, 2026

chalmerlowe self-assigned this Sep 3, 2026

chalmerlowe commented Sep 3, 2026

View reviewed changes

chalmerlowe changed the title ~~feat(gapic): add OpenTelemetry T3 client method span wrapping to gapic_v1.method (D)~~ feat(gapic): add OpenTelemetry T3 client method span wrapping in gapic_v1.method (D) Sep 3, 2026

chalmerlowe added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Sep 3, 2026

chalmerlowe force-pushed the feat/otel-tracing-t3-method-spans branch 3 times, most recently from bf82825 to bc94977 Compare September 4, 2026 09:13

chalmerlowe force-pushed the feat/otel-tracing-t3-method-spans branch from bc94977 to 67e3879 Compare September 4, 2026 11:42

chalmerlowe removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Sep 4, 2026

chalmerlowe force-pushed the feat/otel-tracing-t3-method-spans branch from 67e3879 to 6b9bfab Compare September 4, 2026 14:55

chalmerlowe marked this pull request as ready for review September 4, 2026 14:58

chalmerlowe requested a review from a team as a code owner September 4, 2026 14:58

daniel-sanche requested changes Sep 4, 2026

View reviewed changes

daniel-sanche reviewed Sep 4, 2026

View reviewed changes

chalmerlowe added 3 commits September 4, 2026 13:41

feat(gapic): add OpenTelemetry T3 client method span wrapping

95ca42a

chore(gapic): remove _is_tracing_supported dummy class variable

4cd28f0

test(gapic): achieve 100% branch and statement coverage for OTel T3 m…

9b46ef7

…ethod tracing

chalmerlowe force-pushed the feat/otel-tracing-t3-method-spans branch from fb16695 to 9b46ef7 Compare September 4, 2026 17:41

Navigation Menu

Uh oh!

feat(gapic): add OpenTelemetry T3 client method span wrapping in gapic_v1.method (D) - #18274

feat(gapic): add OpenTelemetry T3 client method span wrapping in gapic_v1.method (D)#18274
chalmerlowe wants to merge 3 commits into
feat/otel-tracing-transport-logicfrom
feat/otel-tracing-t3-method-spans

chalmerlowe commented Sep 3, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Sep 3, 2026

Uh oh!

chalmerlowe Sep 3, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026

Uh oh!

daniel-sanche Sep 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

chalmerlowe commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reference:

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Sep 3, 2026

Choose a reason for hiding this comment

Critical Reliability & Correctness Issues

Solution

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chalmerlowe commented Sep 3, 2026 •

edited

Loading

daniel-sanche Sep 4, 2026 •

edited

Loading