Prometheus
Send data to Levitate using Prometheus Remote Write
Introduction
We make use of the Prometheus remote-write feature to allow Prometheus to write to an external remote endpoint i.e., Levitate's remote-write endpoint.
Prerequisites
Create a Levitate cluster by following Getting Started.
Keep the following information handy after creating the cluster:
$levitate_remote_write_url
- Levitate's Remote write endpoint$levitate_remote_write_username
- Cluster ID$levitate_remote_write_password
- Write token created for the cluster
Writing data to Levitate
The information flow is as follows:
Just update the Prometheus configuration with the following changes for
theremote_write
:
remote_write:
- url: "$remote_write_url"
basic_auth:
username: "$remote_write_username"
password: "$remote_write_password"
remote_timeout: 60s
After that, restart Prometheus, and data will be written to Levitate.
Using Prometheus Agent mode
With Levitate as Long-term storage, you don't need to store time series in your Prometheus. You can run Prometheus only in agent mode and stream data directly to Levitate.
The agent mode is limited to discovery, scrape, and remote write, which fits our needs nicely.
You don't need to have the overhead of running full-fledged Prometheus anymore.
Here is a sample docker-compose.yaml
that runs Prometheus in agent mode.
services:
prometheus:
container_name: prometheus-01
image: prom/prometheus:v2.39.1
ports:
- 9091:9090/tcp
volumes:
- ./prometheus.yaml:/etc/prometheus/prometheus.yaml
command:
- "--config.file=/etc/prometheus/prometheus.yaml"
- "--enable-feature=agent"
network_mode: host
restart: always
The prometheus.yaml
will need to have remote_write
section as follows:
global:
scrape_interval: 1m
scrape_timeout: 30s
scrape_configs:
- job_name: prometheus-self
static_configs:
- targets: ["127.0.0.1:9090"]
remote_write:
- url: "$remote_write_url"
basic_auth:
username: "$remote_write_username"
password: "$remote_write_password"
remote_timeout: 60s
Verification
The Levitate Cluster will start showing a summary of ingested samples.
Next steps
Create a read token for your Levitate Cluster and follow our guide to Configure Grafana to visualize the time series data getting sent to Levitate.
Scaling Prometheus Remote Write
Read https://last9.io/blog/how-to-scale-prometheus-remote-write/, where we talk about how to scale Prometheus Remote write for Federated clusters, considering High Availability requirements.
Troubleshooting
Please get in touch with us on Discord or Email if you have any questions.