iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Agent metrics become expensive and less useful when every agent instance, conversation, or tool call creates a distinct time series. The root cause is metric cardinality: the number of unique combinations of a metric’s attribute values. Keep metrics focused on bounded operational categories, and use traces or logs for the high-detail context needed to investigate individual agent runs.
What cardinality means for agent metrics
A metric’s cardinality is determined by its distinct attribute combinations—not simply by how many requests the system receives. If an instrument records a measurement with attributes such as model, tool, and conversation ID, each different combination can require a separate aggregation set. The OpenTelemetry Metrics SDK specification defines cardinality as “the number of unique combinations of attributes.” OpenTelemetry Metrics SDK specification
For example, a request counter grouped by a small set of model names and tool categories may remain manageable. Add a conversation ID that is unique for every run, however, and the combinations can grow with traffic. The SDK may need more in-process aggregation state, while the metrics backend receives more time series to store and query. OpenTelemetry’s 2026 guidance discusses both effects. OpenTelemetry’s cardinality-limit guidance
Free tools Windows power users keep installed
One-click scans. No signup required.
Agent telemetry makes this easy to do accidentally. The GenAI semantic-conventions registry includes attributes for agents and conversations alongside provider, model, tool, and workflow context. Their presence in a convention does not mean every attribute belongs on every metric: an identifier may be useful for correlating one execution while being a poor dimension for an aggregate metric. OpenTelemetry GenAI attribute registry
#1 Best Overall
- Real-time detection: Capture voltage signals in real time and accurately measure the operating voltage of devices, systems or batteries.
- High stability: stable and reliable circuit design, suitable for harsh environments, high anti-interference ability and safety.
- High accuracy: Provides high-precision voltage measurement data with high resolution and accuracy for precision measurement requirements.
- [Comfortable to carry] Small and lightweight for easy transport and storage, easily take it anywhere you need it.
- Easy to install: Simple structure, easy installation, intuitive operation for fast voltage data acquisition and processing.
Why high cardinality is an operational problem
More aggregation state and time series
Each distinct attribute set can represent another metric stream or aggregation set. A unique request, session, or conversation ID can therefore turn a seemingly modest metric into a rapidly growing set of series. The consequences can include higher memory use in the instrumenting process and more storage and query work in the backend. These risks depend on the SDK, configuration, metric, and backend; cardinality figures are not interchangeable capacity guarantees.
SDK overflow can erase useful groupings
The OpenTelemetry Metrics SDK specification sets a default cardinality limit of 2,000 combinations per metric stream when no matching View or reader default supplies another limit. This is an SDK default, not a universal backend limit or a promise that every implementation uses the same configuration. The limit is applied after attribute filtering. OpenTelemetry Metrics SDK specification
Rank #2
When a stream exceeds its configured limit, additional combinations are folded into an overflow data point marked otel.metric.overflow=true, and their original attributes are removed. An overall total may still be correct, but a query grouped or filtered by a removed attribute can undercount. For example, if a success-status dimension is absent from overflow data, a status-filtered dashboard, SLO, or alert may no longer reflect all measurements. OpenTelemetry’s cardinality-limit guidance
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchCardinality is not the same as total system scale
Prometheus offers a different kind of guidance: its instrumentation page says the vast majority of metrics should have no labels, suggests below 10 as a general cardinality guideline, and recommends investigating metrics over 100 or that could reach that level. Those are Prometheus instrumentation rules of thumb, not a direct comparison with OpenTelemetry’s SDK limit of 2,000 combinations per stream. Prometheus also gives an example in which 10,000 nodes produce roughly 100,000 node_filesystem_avail time series and describes that scale as manageable. Total series across a fleet and cardinality for an individual metric are distinct considerations. Prometheus instrumentation guidance
Which agent attributes belong on metrics?
Start with the operational question a metric should answer. If the question is “How often do tool calls fail by tool type?” a bounded tool category and a bounded outcome are likely more useful than a raw call ID. If the question is “What happened in this particular conversation?”, a metric is usually the wrong place to retain every detail; correlate an individual trace or log instead.
Prefer dimensions with a deliberately bounded set of values, such as a controlled model family, tool category, workflow type, outcome category, HTTP method, or status code. The exact set should match the metric’s purpose and the values your instrumentation actually emits. Avoid adding raw URLs, user input, request IDs, session IDs, or unbounded error messages as metric attributes by default. OpenTelemetry calls out these types of values in its cardinality guidance. OpenTelemetry’s cardinality-limit guidance
For HTTP metrics, use route templates rather than concrete paths containing changing identifiers. The HTTP semantic conventions require low-cardinality route values and represent dynamic path segments with placeholders. A route such as /agents/{agent_id}/runs describes the operation without creating a new route value for every agent ID. OpenTelemetry HTTP metrics conventions
Recommended Free Tools
High cardinality is not automatically wrong. A per-tenant SLO may justify a tenant dimension when the need is explicit and the active tenant set is bounded. OpenTelemetry’s guidance notes delta temporality as potentially practical for a bounded active set; that is an example from the guide, not a universal recommendation. Cumulative temporality retains aggregation state across collection cycles and can accumulate more combinations. Choose based on the workload and SDK behavior rather than treating a temporality setting as a substitute for controlling dimensions. OpenTelemetry’s cardinality-limit guidance
Best Value
- Automatic Probe recognition
- Front panel touch pad: Real Time data view, Battery backup (CR4), Field replaceable probes
- Field calibration of probes
- Independent Channel Alarms (CR4)
- 48 Hours continuous battery life
How to find and reduce accidental growth
- Inspect the attributes on each growing metric. Identify which combinations are being produced and look for values that change per request, session, conversation, agent instance, tool call, or user. Confirm whether the metric’s intended questions actually require each dimension.
- Replace raw values with bounded classifications. Use route templates instead of concrete URLs, controlled categories instead of free-form error text, and an outcome such as success or failure instead of a unique execution identifier. Keep identifiers in traces or logs when they are needed to investigate a particular run.
- Remove attributes at the correct layer. Correct instrumentation upstream when an attribute does not belong on the metric. Where appropriate, configure an OpenTelemetry View to filter attributes from that metric stream. Attribute filtering reduces the combinations retained by the SDK; the specification applies the cardinality limit after this filtering. OpenTelemetry Metrics SDK specification
- Watch for overflow and trace it back to instrumentation. The
otel.metric.overflow=truemarker indicates that combinations exceeded the configured limit. Find the metric and attribute set responsible, then decide whether to bound, remove, or explicitly retain the dimension. Raising the limit alone does not correct an unbounded label and can increase memory exposure. OpenTelemetry’s cardinality-limit guidance - Raise a limit only for a deliberate, bounded need. If a dimension is necessary and its active set is understood, select a limit for that use case and account for the SDK and backend costs. Do not treat the SDK’s default of 2,000 as a target for every metric or as evidence that a backend can accommodate any resulting series volume.
Keep metrics aggregate and traces or logs specific
Metrics are for questions answered across many executions: request volume, latency distributions, error rates, or tool usage by bounded category. Traces and logs can carry more execution-specific context, such as a conversation or request identifier, when that detail is needed to follow a run. Use correlation identifiers consistently so an aggregate metric can direct an investigation to the relevant trace or log without making every identifier a metric dimension.
This separation does not make traces or logs unlimited or automatically safe. Retain only detail that has a clear diagnostic purpose, and consider privacy and data-handling requirements before recording user content or other sensitive values. The practical goal is to preserve per-run diagnostic value without forcing metric aggregation to maintain a new series for every run.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
