To make a PromQL query faster, first limit the time series it selects, then aggregate only to the level the dashboard needs. Measure the change with query statistics; if the same expensive expression is run repeatedly, consider a recording rule. A small result is not necessarily a cheap query: Prometheus may still need to load and process many input series to produce it.
Why PromQL queries get slow
Query cost depends on more than how many lines appear in a graph. A bare metric selector can expand across jobs, instances, paths, status codes, and other labels. Aggregating those series may shrink the output while leaving the work of reading and processing the inputs intact. Range, step, scrape interval, retention, joins, storage, and server limits also affect cost, so there is no universal speedup percentage for a particular rewrite.
Prometheus guidance puts particular emphasis on label cardinality: each unique combination of a metric name and label values is a distinct time series. The Prometheus community describes performance as almost always coming down to label cardinality in The Zen of Prometheus. Cardinality is therefore both an instrumentation concern and a query concern: broad selectors over many distinct series can make otherwise simple expressions expensive.
How to optimize a query
1. Bound the selector before exploring broadly
Start with explicit matchers for dimensions you know are relevant, such as a job, service, or cluster. For example, instead of querying every series for a metric:
Free tools Windows power users keep installed
One-click scans. No signup required.
#1 Best Overall
rate(http_requests_total[5m])try constraining it to the relevant application, if those labels exist in your installation:
rate(http_requests_total{job="api", service="checkout"}[5m])Prometheus recommends using the table view for broad or unfamiliar queries and keeping results to hundreds rather than thousands of time series before switching to graph view. If the instant result is unexpectedly large, inspect which labels cause the fan-out and whether the question actually requires those dimensions.
2. Filter before costly range work and joins
When the intended result is unchanged, narrow the vector before applying functions such as rate or other range calculations, or before a join or high-cardinality aggregation. A join can multiply intermediate series. Match on the smallest valid set of labels; use explicit on(...) or ignoring(...) and grouping modifiers only when the data model requires them. Do not remove a label merely to make a query smaller if that label distinguishes values the reader needs.
3. Aggregate away dimensions the result does not need
If a panel needs service-level totals rather than separate values for every instance or path, aggregate to service level. For example:
sum by (service) (rate(http_requests_total{job="api"}[5m]))Choose grouping labels to match the question. Aggregation reduces the number of output series; it does not by itself eliminate the work required to read the input series. Check that the resulting labels still distinguish the values the panel, alert, or downstream expression expects.
4. Aggregate ratios as separate totals
For a ratio such as errors divided by requests, aggregate the numerator and denominator separately, then divide. Do not average per-series ratios: that gives equal weight to series with very different request volumes. Prometheus recording-rule guidance illustrates preserving useful labels while removing unnecessary ones with without:
sum without (instance, path) (http_request_errors:rate5m)
/
sum without (instance, path) (http_requests:rate5m)Here, instance and path are removed while other labels are retained. Adapt the removed labels and metric names to the series in your environment.
When to use ad-hoc queries, recording rules, or subqueries
| Option | Best fit | Trade-off |
|---|---|---|
| Ad-hoc PromQL | One-off exploration or a query whose cost is acceptable at request time. | Prometheus evaluates the expression when requested; repeated panels repeat that work. |
| Recording rule | An expensive expression reused by dashboards or needed frequently. | The expression is precomputed on the rule group’s evaluation interval, adding configuration and monitoring responsibilities. The stored result is only as fresh as that interval. |
| Subquery | Composing a range calculation from an instant expression when that composition is needed. | Resolution and nested range work can multiply samples; a subquery is not automatically cheaper than the expression it replaces. |
Promote repeated expensive work to a recording rule
Prometheus defines recording rules as a way to precompute frequently needed or computationally expensive expressions and save their results as new time series. They can be particularly useful when several panels repeatedly evaluate the same expression. A rule evaluates the expression on its schedule; dashboards can then query the recorded series rather than redo the full calculation on every refresh.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallFor example, a service-level request-rate rule could look like this:
Rank #4
groups:
- name: service-sli
rules:
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))The metric and label names must match your installation. The name follows Prometheus guidance’s level:metric:operations convention: here, service level, HTTP requests, and a five-minute rate. Choose an evaluation interval that balances freshness with the workload you can support. Monitor rule evaluation: if a group has not finished before its next scheduled evaluation, Prometheus skips that iteration, which can leave a gap in the recorded series.
Prometheus’s Querying basics documentation advises pre-recording an expression if it still takes too long to graph ad hoc. A recording rule is not a universal fix: it shifts repeated query work into scheduled evaluation and introduces a freshness interval and rule operations to manage.
Use subqueries with a specific purpose
A subquery provides a range vector from an instant query, with an optional resolution. That makes it useful when one time-window calculation must be composed with another. But nested ranges and a fine resolution can increase the number of samples evaluated. If a slow subquery is repeatedly reused, consider whether an equivalent recording rule can represent the required calculation without changing its meaning.
Best Value
Reduce cardinality at instrumentation time
Query rewrites cannot fully compensate for metrics that create too many distinct series. Prometheus instrumentation guidance gives a general goal of keeping a metric’s cardinality below 10; for metrics above that, it advises limiting them to a handful across the whole system. It recommends investigating metrics over 100, or with potential to grow beyond 100, for alternate designs. These are guidelines for deciding what to inspect, not a guarantee that a particular metric count will be safe in every deployment.
Look especially for labels whose possible values are numerous or continually changing. A label with a distinct value for every user, request, or other unbounded entity can create a cardinality explosion. Prefer bounded dimensions that answer operational questions, and avoid adding a dimension simply because it is available in the application.
Cardinality needs context. Prometheus instrumentation documentation describes roughly 100,000 node_filesystem_avail time series for 10,000 nodes as manageable in its example, while adding per-user quota dimensions could raise the count into the millions. The example illustrates why a single threshold cannot replace reviewing the metric’s label design and growth.
Measure query cost before and after a change
Inspect the query log for slow or high-load work
Prometheus can log queries at runtime to help investigate slow requests or high load. When reviewing entries, compare the expression with its duration, time range, and step. Logging is useful for finding which queries deserve attention; it does not by itself explain which expression component consumed the most samples.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Use per-step statistics for sample counts
For more detailed engine counters, start Prometheus with --enable-feature=promql-per-step-stats and request query statistics with stats=all. Prometheus can then expose values including total queryable samples, samples read, peak samples, and related engine statistics. The feature flag must be enabled for the statistics to be available.
Compare the same query and time range before and after changing its selector, aggregation, or use of a recording rule. Read sample counts alongside duration: fewer samples read can indicate less work, but the result still needs to preserve the dashboard or alert’s intended meaning. Prometheus documents these statistics as observability tools, not as a benchmark that predicts a fixed speedup on other installations.
Quick Recap
A practical optimization sequence
- Inspect the result in table view. Begin with an instant query and identify whether a selector returns hundreds, thousands, or more series.
- Add valid label matchers. Restrict the query to the relevant job, service, cluster, or other known dimension.
- Choose the necessary output labels. Aggregate away dimensions the reader does not need, while preserving labels required to interpret or join the result.
- Check joins and range work. Confirm that match keys and grouping modifiers do not create unnecessary intermediate series.
- Measure the revised query. Use query logging and, when enabled, per-step statistics to compare work over the same range and step.
- Record repeated expensive expressions. If the same costly calculation is run frequently, evaluate whether a recording rule’s freshness and operational trade-offs fit the use case.
- Review instrumentation if series counts remain excessive. Find the label combinations driving growth and redesign unbounded dimensions where possible.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
