Skip to main content

Prometheus

Send data to Last9 using Prometheus Remote Write

Introduction​

We make use of the Prometheus remote-write feature to allow Prometheus to write to an external remote endpoint i.e., Last9's remote-write endpoint.

Prerequisites​

Create a Last9 cluster by following Getting Started.

Keep the following information handy after creating the cluster:

  • $levitate_remote_write_url - Last9's Remote write endpoint
  • $levitate_remote_write_username - Cluster ID
  • $levitate_remote_write_password - Write token created for the cluster

Writing data to Last9​

The information flow is as follows:

Remote Write data to Last9 in addition to writing to your Prometheus

Just update the Prometheus configuration with the following changes for theremote_write:

remote_write:
  - url: "$remote_write_url"
    basic_auth:
      username: "$remote_write_username"
      password: "$remote_write_password"
    remote_timeout: 60s

After that, restart Prometheus, and data will be written to Last9.

Using Prometheus Agent mode​

With Last9 as Long-term storage, you don't need to store time series in your Prometheus. You can run Prometheus only in agent mode and stream data directly to Last9.

The agent mode is limited to discovery, scrape, and remote write, which fits our needs nicely.

Remote Write to Last9 using Prometheus Agent

You don't need to have the overhead of running full-fledged Prometheus anymore.

Here is a sample docker-compose.yaml that runs Prometheus in agent mode.

services:
  prometheus:
    container_name: prometheus-01
    image: prom/prometheus:v2.39.1
    ports:
      - 9091:9090/tcp
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml
    command:
      - "--config.file=/etc/prometheus/prometheus.yaml"
      - "--enable-feature=agent"
    network_mode: host
    restart: always

The prometheus.yaml will need to have remote_writesection as follows:

global:
  scrape_interval: 1m
  scrape_timeout: 30s

scrape_configs:
  - job_name: prometheus-self
    static_configs:
      - targets: ["127.0.0.1:9090"]

remote_write:
  - url: "$remote_write_url"
    basic_auth:
      username: "$remote_write_username"
      password: "$remote_write_password"
    remote_timeout: 60s

Verification​

The Last9 Cluster will start showing a summary of ingested samples.

Last9 Cluster's Performance metrics

Next steps​

Create a read token for your Last9 Cluster and follow our guide to Configure Grafana to visualize the time series data getting sent to Last9.

Scaling Prometheus Remote Write​

Read https://last9.io/blog/how-to-scale-prometheus-remote-write/, where we talk about how to scale Prometheus Remote write for Federated clusters, considering High Availability requirements.

Troubleshooting​

Please get in touch with us on Discord or Email if you have any questions.