About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Distributed tracing is not possible when using Google Cloud Load Balancer.

Distributed tracing is not possible when using Google Cloud Load Balancer.

This is Hasegawa (@rarirureluis) from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article explains an issue where distributed tracing does not function as expected in environments using Google Cloud's Load Balancer.
 

Introduction

In environments using Google Cloud's Load Balancer, you may encounter issues where distributed tracing does not function as expected.
traceparent
This article explains the cause of this problem and provides specific solutions to avoid it.

Problem description: The flags in the traceparent header change.

traceparent
This header is defined in the W3C Trace Context standard and contains information such as the trace ID, span ID, and sampling flag.
traceparent
-00
This results in a problem where requests that were originally intended to be traced are not sampled, making them untrackable by the distributed tracing system.

Cause: A known bug in Load Balancer

This phenomenon is caused by a known bug in Google Cloud Load Balancer.
This issue has been registered in Google's Issue Tracker, and efforts are currently underway to resolve it.
X-Cloud-Trace-Context
o=0
-00
o=1
-01

Workaround: Rewrite headers using a CDN.

An effective workaround for this problem is to use the CDN's functionality to rewrite HTTP headers in environments where a CDN is deployed in front of Google Cloud Load Balancer.
Specifically, you will add the following rule to your CDN settings:
  1. traceparent
  1. o=1
X-Cloud-Trace-Context
  • traceparent
  • X-Cloud-Trace-Context
  • 1
X-Cloud-Trace-Context;o=1
 

In conclusion

traceparent
X-Cloud-Trace-Context
If you are having trouble implementing distributed tracing in a system using Google Cloud Load Balancer, consider implementing this workaround.
SRG is looking for new team members. If you are interested, please contact us here.