๐ช๐ป Managing High Cardinality
This document explains how Levitate provides visibility, superior defaults, and control levers to tame high cardinality
Levitate Limits
| Quota Type | Limit | Reset Period | Possible Actions |
|---|---|---|---|
| Writes | |||
| Per Time Series Cardinality | 1M | Per Hour | Can be raised on request |
| Per Time Series Cardinality | 20M | Per Day | Can be raised on request |
| Streaming Aggregation Cardinality | 3M | Per Hour | Can be raised on request |
| Ingestion Concurrency | 20K | Per Second | Can be raised on Request |
| Number of Metrics Aggregated in one Pipeline | 1 Metric | Per Query | Cannot be changed for now |
| Reads | |||
| Time Series Scanned Per Query โ Blaze Tier | 5M | Per Query | Cannot be changed |
| Time Series Scanned Per Query โ Hot Tier | 10M | Per Query | Cannot be changed |
| Samples Scanned Per Query | 100M | Per Query | Cannot be changed |
| Query Time Range | 35 Days | Per Query | Can be raised on request |
Some metrics with very high cardinality will overflow beyond the above limits.
This data is to be found in the Overflow data source. Selecting an overflow lake is the same as Selecting any other Data source.
Almost all products have these hard limits and drop the data after it overflows or blocks the ingestion altogether. Either way, we have found this to result in extremely poor actionability. Customers are unaware or have too much toil to discover whatโs overflowing. We have introduced the Overflow lake to overcome this challenge and massively improve the actionability once you hit the Limits.
By the way, our Limits are seriously more than almost every other Prometheus provider.
Overflow Data Source
Start by landing on the Explore tab of Grafana and selecting the right Data source.
Using the metric browser, you can explore the overflow cardinality of the metric.
You cannot use the Overflow lake to create dashboards and extract data. In addition to cardinality exploration via the Metrics Browser, It only supports additive promQL queries that can form Streaming Aggregations.
Please refer to the later part of the document for details on PromQL-powered Streaming Aggregations.
An attempt to make non-streaming explore queries results in the following error:
422: cannot parse query "rate(http_requests_duration_seconds_bucket{accountId=\"1444038914\"}[1m])": function "rate" is not supported. Please check streaming aggregation docs to understand the supported functions 422: cannot parse query "rate(http_requests_duration_seconds_bucket{accountId=\"1444038914\"}[1m])": function "rate" is not supported. Please check streaming aggregation docs to understand the supported functions
Once you find data that is present in Overflowing Lake, you can take one of the following actions:
- Edit the instrumentation
- Edit the source to emit the right labelSet combinations.
- Drop labelSet combination.
- Edit the source Prometheus Agent to drop labels.
- Use Streaming Aggregation to reduce cardinality.
- Refer to our feature. PromQL-powered Streaming Aggregations
Updated 15 days ago