fasthttp

Send distributed traces and metrics to Last9 from Golang fasthttp application using OpenTelemetry

Introduction

fasthttp is a fast, flexible HTTP server framework written in Go (Golang). This comprehensive guide will help you instrument your fasthttp application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on GitHub↗.

Pre-requisites

1. You have a fasthttp application.
2. You have signed up for Last9, created a cluster, and obtained the following OTLP credentials from the Integrations page:
   • endpoint
   • auth_header

Install OpenTelemetry packages

To install the required packages, run the following command:

  go get -u go.opentelemetry.io/otel
  go get -u go.opentelemetry.io/otel/sdk
  go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  go get -u go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc
  go get -u go.opentelemetry.io/otel/sdk/metric
  go get -u go.nhat.io/otelsql

For setting up Redis instrumentation, first verify which go-redis version you are using.

If you are using go-redis v8 then

  go get -u github.com/go-redis/redis/extra/redisotel

If you are using go-redis v9 then

  go get -u github.com/redis/go-redis/extra/redisotel

Using these packages, you can instrument your fasthttp application to send traces to Last9.

Traces

This application generates traces for the following:

• HTTP requests using last9/otelMiddleware
• Database queries using otelsql
• Redis commands using redisotel

For HTTP requests, wrap the fasthttp router with the last9.otelMiddleware middleware.

Refer to main.go for more details.

For database queries, use the otelsql package to wrap the sql.DB object.

Refer to initDB() in users/controller.go for more details.

For Redis commands, use the redisotel package to wrap the redis.Client object.

Refer to initRedis() in users/controller.go for more details.

Set the environment variables

Set the following environment variables:

export OTEL_SERVICE_NAME=fasthttp-app-service
export OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint>
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <auth_header>"

These environment variables are used to configure the OpenTelemetry SDK to send traces and metrics to Last9.

Instrument HTTP requests

Setup instrumentation for fasthttp application using last9/instrumentation.

package last9

import (
	"context"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
	"go.opentelemetry.io/otel/trace"
)

type Instrumentation struct {
	TracerProvider *sdktrace.TracerProvider
	Tracer         trace.Tracer
}

func initTracerProvider() *sdktrace.TracerProvider {
	exporter, err := otlptracehttp.New(context.Background())

	if err != nil {
		panic(err)
	}

	attr := resource.WithAttributes(
		semconv.DeploymentEnvironmentKey.String("production"), // replace with actual environment
		semconv.ServiceNameKey.String("fasthttp-server"), // replace with service name
	)

	resources, err := resource.New(context.Background(),
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithProcess(),
		resource.WithOS(),
		resource.WithContainer(),
		resource.WithHost(),
		attr)

	if err != nil {
		panic(err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resources),
	)

	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

	return tp
}

func NewInstrumentation() *Instrumentation {
	tp := initTracerProvider()

	return &Instrumentation{
		TracerProvider: tp,
		Tracer:         tp.Tracer("fasthttp-server"),
	}
}

The above code configures the OpenTelemetry SDK to use the OTLP exporter and initializes the TracerProvider. Next, at the entry point of your application, add the following code to instrument your application:

// main.go
func main() {
	i := NewInstrumentation()

	defer func() {
		if err := i.TracerProvider.Shutdown(context.Background()); err != nil {
			log.Printf("Error shutting down tracer provider: %v", err)
		}
	}()
	// other code
}

Now, add the otel fasthttp middleware to your application:

// main.go
func main() {
	i := last9.NewInstrumentation()

	defer func() {
		if err := i.TracerProvider.Shutdown(context.Background()); err != nil {
			log.Printf("Error shutting down tracer provider: %v", err)
		}
	}()
    // other code
  	// Routes
	r.GET("/users", h.GetUsers)
	r.GET("/users/{id}", h.GetUser)
	r.POST("/users", h.CreateUser)
	r.PUT("/users/{id}", h.UpdateUser)
	r.DELETE("/users/{id}", h.DeleteUser)
	r.GET("/joke", func(ctx *fasthttp.RequestCtx) {
		getRandomJoke(ctx, i)
	})

	handler := last9.OtelMiddleware("fasthttp-server") // replace with name of the service
	// other code
}

Instrument database operations

Instrumenting with sql.DB

Add the following code to instrument the database queries. It uses the otelsql package to wrap the sql.DB object and emit traces and metrics for database queries and connections. Refer to users/controller.go for more details.

func initDB() (*sql.DB, error) {
	driverName, err := otelsql.Register("postgres",
		// Read more about the options here: https://github.com/nhatthm/otelsql?tab=readme-ov-file#options
		otelsql.AllowRoot(),
		otelsql.TraceQueryWithoutArgs(),
		otelsql.TraceRowsClose(),
		otelsql.TraceRowsAffected(),
		otelsql.WithDatabaseName("otel_demo"), // database name
		otelsql.WithSystem(semconv.DBSystemPostgreSQL),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to register driver: %v", err)
	}

	db, err := sql.Open(driverName, dsnName)
	if err != nil {
		return nil, fmt.Errorf("failed to connect to database: %v", err)
	}

	// Record stats to expose metrics
	if err := otelsql.RecordStats(db); err != nil {
		return nil, err
	}

	return db, nil
}

Instrumenting with pgx

For database instrumentation where pgx is used, we use otelpgx to wrap the pgx connection pool. Add the following code to instrument the database queries.

	var connString = os.Getenv("DATABASE_URL")
	cfg, err := pgxpool.ParseConfig(connString)
	if err != nil {
		fmt.Fprintf(os.Stderr, "create connection pool: %w", err)
		os.Exit(1)
	}

    // Add the tracer to the connection pool configuration
	cfg.ConnConfig.Tracer = otelpgx.NewTracer()
    // Create a new connection pool with the tracer
	conn, err = pgxpool.NewWithConfig(context.Background(), cfg)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Unable to connection to database: %v\n", err)
		os.Exit(1)
	}

Refer to the complete example using pgx adapter and Otel instrumentation here.

Instrument Redis operations

tip

Note that between redis-go versions v8 and v9 the import paths are changed.

Add the following code to instrument the Redis operations. It uses the redisotel package to wrap the redis.Client object and emit traces for Redis commands.

go-redis v9

If you are using go-redis v9 then use the following code.

Refer to users/controller.go for more details.

func initRedis() *redis.Client {
	rdb := redis.NewClient(&redis.Options{
		Addr: "localhost:6379", // Update this with your Redis server address
	})

	// Setup traces for redis instrumentation
	if err := redisotel.InstrumentTracing(rdb); err != nil {
		log.Fatalf("failed to instrument traces for Redis client: %v", err)
		return nil
	}
	return rdb
}

go-redis v8

If you are using go-redis v8 then use the following code.

func initRedis() *redis.Client {
	rdb := redis.NewClient(&redis.Options{
		Addr: "localhost:6379", // Update this with your Redis server address
	})

	// Setup traces for redis instrumentation
    rdb.AddHook(redisotel.NewTracingHook())
	return rdb
}

Refer to main.go for more details.

Run the application

Start your fasthttp application by running the following command:

go run main.go

This will start the fasthttp application and the OpenTelemetry SDK will automatically collect traces and metrics from the fasthttp application. These traces will be sent to Last9 automatically based on the environment variables set.

View traces in Last9

After running the fasthttp app, you can visualize the traces in Last9's APM dashboard.

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.