HOME/Technical Articles/Distributed tracing not working when using Google Cloud Load Balancer

Distributed tracing not working when using Google Cloud Load Balancer

2025/4/22 20:552025/4/22 21:14

This is Hasegawa (@rarirureluis) from the Service Reliability Group (SRG) of the Media Headquarters.

#SRG(Service Reliability Group) is a group that mainly provides cross-sectional support for the infrastructure of our media services, improving existing services, launching new ones, and contributing to OSS.

This article explains the issue where distributed tracing does not work as expected in an environment using Google Cloud Load Balancer.

Introduction Symptom: The flag in the traceparent header changes Cause: Known Load Balancer bug Workaround: Rewrite headers at your CDN Conclusion

Introduction

In environments using Google Cloud Load Balancer, you may encounter issues where distributed tracing does not work as expected.

traceparent

This article explains the cause of this issue and provides specific workarounds.

Symptom: The flag in the traceparent header changes

traceparent

This header is defined by the W3C Trace Context standard and contains information such as the trace ID, span ID, and sampling flag.

traceparent

-00

This causes problems as requests that you want to trace are not sampled and cannot be tracked by the distributed tracing system.

Cause: Known Load Balancer bug

This issue is caused by a known bug in Google Cloud Load Balancer.

This issue has been filed in Google's issue tracker and is currently being worked on to resolve.

Google Issue Tracker

https://issuetracker.google.com/issues/401141843

X-Cloud-Trace-Context

o=0

-00

o=1

-01

Workaround: Rewrite headers at your CDN

An effective workaround for this issue is to use the CDN's functionality to rewrite the HTTP headers in environments where a CDN is placed in front of the Google Cloud Load Balancer.

Specifically, add the following rules to your CDN settings: