A metric time series is identified by its name and complete label set. Every distinct combination creates another series. Labels with unbounded values, such as request IDs or customer email addresses, can grow storage, memory, and query work with traffic volume. Histograms multiply that effect across bucket, sum, and count series, so a label that looks small on one counter can be expensive when copied everywhere.
Make this comfortable
Metric cardinality: keep observability usable during a surge
Operational decision
A claims gateway emits a request counter. Keep stable route templates, outcome class, and service revision as metric labels; place individual request IDs in trace or log records under the appropriate retention policy. The PromQL expression groups current request series by route so an operator can spot a route explosion. It is an inspection query, not a cardinality limit. Review the number of active series before and after each instrumentation change and set a collection budget per service. If a route label contains raw account numbers, replace it with the registered route template at the instrumentation point; dropping it later in the pipeline may already have burdened the agent. Run a load test with many distinct account IDs and verify series growth stays bounded while the error-rate alert still separates useful route classes.
count by (route) (claims_requests_total)Cost and verification
Fewer labels reduce ingestion and query cost but remove dimensions from aggregate metrics. Use traces for individual requests and logs for carefully controlled detail. A query that scans every series can itself be costly, so measure head-series count and ingestion rate with the monitoring system's own metrics rather than repeatedly issuing broad exploratory queries during an incident. Keep the remaining labels stable enough for dashboards and alert rules to survive releases.
Common Mistakes
- Do not label a counter with request ID, email, or raw path parameters.
- Do not assume a cheap counter label stays cheap on a histogram.
- Do not remove every dimension and leave operators unable to locate a failing route.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- Observability: join metrics, logs, and traces
- Alert design: page on impact and include a first action
- Distributed traces: preserve context without leaking data
- Log pipelines: preserve incident evidence without ingesting secrets
Practice and check
devops
operations