How do you capture high‑cardinality metrics without exploding cost?

High cardinality metrics capture signals with many unique label combinations such as user id or device id. They are invaluable for root cause analysis but they can overwhelm a time series backend and your budget. The goal is not to drop visibility but to collect the right detail at the right place with storage that matches query needs and cost constraints.

Why It Matters

In interviews and in production you will be judged on how you balance insight with efficiency. Pure metrics that include every user or every request can create millions of series which increases memory pressure index size and query latency. On call teams need a design that keeps p ninety five and p ninety nine visibility while avoiding runaway cost. Showing you can control cardinality while preserving debuggability is a strong system design interview signal.

How It Works Step by Step

Step 1. Set a cardinality budget Decide the maximum series you will allow per service per region and per team. Treat it like an SLO. For example a budget of one hundred thousand active series per service keeps head memory predictable. Exceeding the budget triggers guardrails in the next steps.

Step 2. Allowlist dimensions that matter Pick a small set of labels that explain most incidents such as region az instance class api endpoint experiment group. Block or rename high entropy labels such as request id user id and build sha. When product needs per user analysis route those events to a log or trace store instead of inflating the metrics space.

Step 3. Pre aggregate at the edge Use a sidecar or client library that batches and aggregates counters timers and gauges before sending to the metrics backend. Pre aggregation turns many granular updates into a single time series per minute per label set which shrinks writes and index churn.

Step 4. Use histograms and sketches for latency and size Quantiles from raw series are costly. Prefer HDR Histogram or DDSketch to encode latency distributions with tight memory bounds. One sketch per allowlisted dimension gives p fifty p ninety p ninety nine and max without creating a new series per request. For object sizes request bytes or payload samples store histograms rather than raw buckets.

Step 5. Apply dynamic sampling for spiky or unbounded keys If you must observe keys with unbounded space such as user or tenant apply head sampling during normal operation and raise the sampling rate during error spikes. Tail based sampling is great for traces and for metrics you can approximate it by gating on error rate so that failing requests are always recorded with more detail.

Step 6. Heavy hitter detection instead of per key metrics Use streaming heavy hitter algorithms to track the top K users endpoints or queries during each window. Emit only the top keys as labeled metrics and export the full key list as a log blob to object storage for offline inspection. This preserves fast dashboards while keeping long tail cost near zero.

Step 7. Tiered retention and downsampling Keep detailed minute level series for a short hot window such as seven days. Downsample to five minute and one hour rollups for warm weeks and months. For any dimension with very high cardinality shorten its hot retention further while keeping service level aggregates longer.

Step 8. Separate paths for metrics logs and traces Metrics serve fast numeric queries and alerting. Logs and traces carry the high cardinality detail. Route intense cardinality to a columnar or object store backed system and link from metric panels to the corresponding trace or log view. This keeps graphs fast and bills stable.

Step 9. Enforce guardrails in the write path Introduce a label policy that rejects or rewrites metrics with banned keys and caps the number of series per metric name. Emit audit metrics when drops happen so developers can fix abusive emitters. Make the policy visible in documentation and code review templates.

Step 10. Design for multi tenant fairness Allocate separate series budgets per tenant or team. If one tenant exceeds their share slow or drop only their high cardinality labels rather than impacting everyone. This avoids noisy neighbor issues inside the monitoring platform.

Real World Example

Consider a large social feed service that previously labeled latency with user id and post id to track p ninety nine outliers. This created tens of millions of series and frequent cardinality explosions during traffic spikes. The team introduced a plan.

• They set a series budget per service and added a label allowlist focused on region endpoint and experiment.
• They moved per user and per post detail to a trace store with aggressive tail sampling during errors.
• They replaced raw quantiles with HDR Histogram based timers.
• They added a streaming top K pipeline that emits the ten most expensive endpoints per region each minute as labeled metrics and sends a full key list to object storage.
• They changed retention to seven days for minute data and ninety days for hourly rollups.

Dashboards still show p fifty p ninety and p ninety nine for every important dimension. When an outage hits engineers click from a high level panel into traces that carry user level and post level context. Ingest volume and memory footprint dropped by more than half without losing the signals that matter.