gRPC

Send distributed traces to Last9 from a Golang gRPC using OpenTelemetry

Introduction

gRPC is a high-performance, open-source universal RPC framework. This comprehensive guide will help you instrument your gRPC application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on GitHub↗.

Pre-requisites

1. You have a gRPC application. For this document, we will use a simple gRPC server and client application. You can use any gRPC application and follow the same steps.
2. You have signed up for Last9, created a cluster, and obtained the following OTLP credentials from the Integrations page:
   • endpoint
   • auth_header

Install OpenTelemetry packages

To install the required packages, run the following command:

  go get -u go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
  go get -u go.opentelemetry.io/otel
  go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  go get -u go.opentelemetry.io/otel/sdk

Setup the OpenTelemetry SDK

To setup the OpenTelemetry SDK, you need to add the following code to your application:

package instrumentation // this can be changed to any name

import (
	"context"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
)

// InitTracer initializes the OpenTelemetry tracer
func InitTracer(serviceName string) func(context.Context) error {
    // Initialize the exporter
	exporter, err := otlptracegrpc.New(context.Background())
	if err != nil {
		panic(err)
	}

	attr := resource.WithAttributes(
        // Set the deployment environment semantic attribute
		semconv.DeploymentEnvironmentKey.String("production"), // You can change this value to "development" or "staging" or you can get the value from the environment variables
		// You can add more attributes here
	)

    // Create a new resource with the attributes
	resources, err := resource.New(context.Background(),
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithProcess(),
		resource.WithOS(),
		resource.WithContainer(),
		resource.WithHost(),
		attr)

	if err != nil {
		panic(err)
	}

    // Create a new tracer provider with the exporter and resource
	tp := sdktrace.NewTracerProvider(
        // Batch the traces to the exporter using BatchSpanProcessor
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resources),
	)

	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

	return tp.Shutdown
}

Instrument the gRPC server application

To instrument the gRPC server application, you need to add the following code to your application:

func main() {
	// Initialize the tracer
	shutdown := instrumentation.InitTracer("grpc-server")
	defer shutdown(context.Background())

	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer(
		grpc.StatsHandler(otelgrpc.NewServerHandler()),
	)

	pb.RegisterGreeterServer(s, &server{})
	log.Printf("server listening at %v", lis.Addr())
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

Instrument the gRPC client application

To instrument the gRPC client application, you need to add the following code to your application:

func main() {
	// Initialize the tracer
	shutdown := instrumentation.InitTracer("grpc-client")
	defer shutdown(context.Background())

    conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithStatsHandler(otelgrpc.NewClientHandler()))

    if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()

	c := pb.NewGreeterClient(conn)

	name := "World"
	if len(os.Args) > 1 {
		name = os.Args[1]
	}
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()
	r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name})
	if err != nil {
		log.Fatalf("could not greet: %v", err)
	}
	log.Printf("Greeting: %s", r.GetMessage())
}

Run the application

Set the endpoint and auth_header environment variables with the values you obtained from the Integrations page.

export OTEL_SERVICE_NAME=grpc-server-app // this is name of the server side application
export OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint>
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-auth-token>"

Run the server application:

go run server/main.go

Run the client application:

export OTEL_SERVICE_NAME=grpc-client-app // this is name of the client side application
export OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint>
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-auth-token>"

go run client/main.go

note

Note that the service name should be different for the server and client applications to distinguish between them.

View traces in Last9

After running the gRPC apps, you can visualize the traces in Last9's APM dashboard.

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.