Gorilla Mux

Send distributed traces and metrics to Last9 from Golang Gorilla Mux application using OpenTelemetry

Introduction

Gorilla Mux is a powerful HTTP router and dispatcher for Go. This comprehensive guide will help you instrument your Gorilla Mux application with OpenTelemetry and smoothly send the traces to Last9.

tip

You can also check out the example application on GitHub↗.

Pre-requisites

1. You have a Gorilla Mux application.
2. You have signed up for Last9, and obtained the following OTLP credentials from the Integrations page:
   • endpoint
   • auth_header

Install OpenTelemetry packages

To install the required packages, run the following command:

  go get -u go.opentelemetry.io/otel
  go get -u go.opentelemetry.io/otel/sdk
  go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  go get -u go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux
  go get -u go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc
  go get -u go.opentelemetry.io/otel/sdk/metric

Redis instrumentation

For setting up Redis instrumentation, first verify which go-redis version you are using.

If you are using go-redis v8 then

  go get -u github.com/go-redis/redis/extra/redisotel/v8

If you are using go-redis v9 then

  go get github.com/redis/go-redis/extra/redisotel/v9

Database instrumentation

If you are using sql.DB then use otelsql to instrument the database queries.

  go get -u go.nhat.io/otelsql

If you are using pgx then use otelpgx to instrument the database queries.

  go get -u github.com/exaring/otelpgx

Traces

This application generates traces for the following:

• HTTP requests using otelmux
• Database queries using otelsql
• redis operations using redisotel

Metrics

The application also generates metrics for the following:

• Database queries using otelsql

Environment variables

Set the following environment variables to send traces and metrics to Last9:

export OTEL_SERVICE_NAME=gorilla-mux-app-service // Change this value to the name of your service
export OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> // Last9 OTLP endpoint
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <auth_header>" // Last9 OTLP auth header
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<environment>,component=api // Add more attributes as needed as comma separated key=value pairs.

Instrument HTTP requests

In your Gorilla Mux application, create a new file last9/instrumentation.go and add the following code:

package last9

import (
	"context"
	"fmt"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
	"go.opentelemetry.io/otel/trace"
)

type Instrumentation struct {
	TracerProvider *sdktrace.TracerProvider
	Tracer         trace.Tracer
}

func initMetrics() (*metric.MeterProvider, error) {
	exporter, err := otlpmetricgrpc.New(context.Background())
	if err != nil {
		fmt.Println("Error creating metrics exporter:", err)
		return nil, err
	}

	resources, err := resource.New(context.Background(),
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithProcess(),
		resource.WithOS(),
		resource.WithContainer(),
		resource.WithHost())

	if err != nil {
		return nil, err
	}

	// Set up the meter provider with the exporter and resource and a periodic reader that flushes every minute
	mp := metric.NewMeterProvider(metric.WithResource(resources),
		metric.WithReader(metric.NewPeriodicReader(exporter, metric.WithInterval(1*time.Minute))))
	return mp, nil
}

func initTracerProvider() *sdktrace.TracerProvider {
	exporter, err := otlptracehttp.New(context.Background())
	if err != nil {
		panic(err)
	}

	resources, err := resource.New(context.Background(),
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithProcess(),
		resource.WithOS(),
		resource.WithContainer(),
		resource.WithHost())

	if err != nil {
		panic(err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resources),
	)

	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

	return tp
}

func NewInstrumentation() *Instrumentation {
	tp := initTracerProvider()

	return &Instrumentation{
		TracerProvider: tp,
		Tracer:         tp.Tracer("gorilla-mux-server"), // Change this value to the name of your service
	}
}

The above code configures the OpenTelemetry SDK to use the OTLP exporter and initializes the TracerProvider.

Next, at the entry point of your application, add the following code to instrument your application:

import (
    // other imports
    "github.com/gorilla/mux"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"
)

func main() {
    i := last9NewInstrumentation()

    defer func() {
        if err := i.TracerProvider.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer provider: %v", err)
        }
    }()

    r := mux.NewRouter()
    // Add the middleware before adding any routes
    r.Use(otelmux.Middleware("gorilla-mux-server")) // Change this value to the name of your service

    // Add your routes here
    r.HandleFunc("/users/{id}", UpdateUser).Methods("PUT")

    // Start the server
    log.Fatal(http.ListenAndServe(":8080", r))
}

Instrument database operations

Instrumenting with sql.DB

Add the following code to instrument the database queries. It uses the otelsql package to wrap the sql.DB object and emit traces and metrics for database queries and connections.

func initDB() (*sql.DB, error) {
	// Register the driver with tracing
	driverName, err := otelsql.Register("postgres",
		// Read more about the options here: https://github.com/nhatthm/otelsql?tab=readme-ov-file#options
		otelsql.AllowRoot(),
		otelsql.TraceQueryWithoutArgs(),
		otelsql.TraceRowsClose(),
		otelsql.TraceRowsAffected(),
		otelsql.WithDatabaseName("otel_demo"), // your database name
		otelsql.WithSystem(semconv.DBSystemPostgreSQL),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to register driver: %v", err)
	}

	db, err := sql.Open(driverName, dsnName)
	if err != nil {
		return nil, fmt.Errorf("failed to connect to database: %v", err)
	}

	// Record stats to expose metrics
	if err := otelsql.RecordStats(db); err != nil {
		return nil, err
	}

	return db, nil
}

Instrumenting with pgx

For database instrumentation where pgx is used, use otelpgx to wrap the pgx connection pool. Add the following code to instrument the database queries.

	var connString = os.Getenv("DATABASE_URL")
	cfg, err := pgxpool.ParseConfig(connString)
	if err != nil {
		fmt.Fprintf(os.Stderr, "create connection pool: %w", err)
		os.Exit(1)
	}

    // Add the tracer to the connection pool configuration
	cfg.ConnConfig.Tracer = otelpgx.NewTracer()
    // Create a new connection pool with the tracer
	conn, err = pgxpool.NewWithConfig(context.Background(), cfg)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Unable to connection to database: %v\n", err)
		os.Exit(1)
	}

info

Refer to the complete example using pgx adapter and Otel instrumentation here.

Instrument Redis operations

tip

Note that between redis-go versions v8 and v9 the import paths are changed.

Add the following code to instrument the Redis operations. It uses the redisotel package to wrap the redis.Client object and emit traces for redis operations.

go-redis v9

If you are using go-redis v9 then use the following code.

func initRedis() *redis.Client {
	rdb := redis.NewClient(&redis.Options{
		Addr: "localhost:6379", // Update this with your Redis server address
	})

	// Setup traces for redis instrumentation
	if err := redisotel.InstrumentTracing(rdb); err != nil {
		log.Fatalf("failed to instrument traces for Redis client: %v", err)
		return nil
	}
	return rdb
}

go-redis v8

If you are using go-redis v8 then use the following code.

func initRedis() *redis.Client {
	rdb := redis.NewClient(&redis.Options{
		Addr: "localhost:6379", // Update this with your Redis server address
	})

	// Setup traces for redis instrumentation
    rdb.AddHook(redisotel.NewTracingHook())
	return rdb
}

Run the application

Start your Gorilla Mux application by running the following command:

go run main.go

This will start the Gorilla Mux application and the OpenTelemetry SDK will automatically collect traces and metrics from the application. These traces and metrics will be sent to Last9 automatically based on the environment variables set.

View traces and metrics in Last9

After running the Gorilla Mux app, you can visualize the traces and metrics in Trace Explorer.

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.