How would you build a metrics store (Prometheus‑like) with remote storage?

A metrics store that feels like Prometheus but keeps data in remote storage gives you the best of both worlds. You keep the simple pull based scrape model and familiar query language while unlocking long retention, elastic scale, and team level isolation. Think of it as three clear paths that work together. an ingestion path that accepts time series samples, a storage path that persists and compacts them in a cheap durable system, and a query path that fans out, merges, and down samples results for fast dashboards.

Why It Matters

Prometheus on a single node is perfect for cluster level monitoring but it struggles with long retention, multi tenant isolation, and large query fanout. A remote storage backed design solves those limits.

• Long retention without local disks filling up

• Horizontal ingestion capacity for bursty services

• Multi region durability and simple disaster recovery

• Centralized observability across many teams while keeping per tenant quotas

• Lower total cost by using object storage and compute that can scale independently

How It Works step by step

Below is a concrete path you can implement. It mirrors how mature metric systems work while staying compatible with Prometheus scrape and query patterns.

1. Data model and naming

• Each time series is a metric name with a set of labels. Example. http_requests_total with labels job, instance, method, status.
• Store samples as time, value, and series id.
• Enforce cardinality budgets per tenant. set limits on unique series, labels per series, and samples per second.
• Prefer counters and histograms for aggregation friendly metrics. gauge is fine for point in time values.

2. Ingestion from Prometheus via remote write

• Prometheus scrapes targets and batches samples into remote write requests.
• A stateless distributor receives requests, authenticates the tenant, validates label rules, and assigns each series to a partition using a consistent hash of labels.
• Before acknowledging, the distributor appends to a write ahead log so back pressure or restarts do not lose samples.
• Per partition queues buffer and retry with exponential backoff. Keep idempotency by using sequence numbers or a de duplicate key of series id and timestamp.

3. Short term in memory buffering and chunk building

• Ingesters keep recent samples for a series in memory and build compressed chunks. Common encodings include delta of delta for timestamps and XOR based float encoding.
• When a chunk is sealed by size or time, the ingester writes it to object storage and updates the index store.
• Seal policy matters. longer windows shrink index writes but can increase memory pressure and recovery time.

4. Index design for fast label queries

• Use an inverted index that maps label name and value to series ids.
• Store the index in a key value database that scales reads and writes independently from object storage. Options include column family stores and cloud scale key value services.
• Keep a small hot cache. label postings lists, frequent series metadata, and last value for staleness checks.

5. Compaction and retention

• A background compactor merges many small chunks into larger blocks, reorders samples, removes duplicates, and rewrites index entries.
• Apply retention rules. for example keep raw ten second resolution for seven days, keep one minute rollups for ninety days, and keep one hour rollups for one year.
• Move older blocks to colder storage classes to save cost.

6. Downsampling and rollups

• Precompute rollups during compaction. sum, min, max, average for gauges and proper rate and increase for counters.
• For histograms, pre merge buckets so long range percentiles can read fewer blocks.
• The query planner chooses downsampled data when ranges are large and step is coarse.

7. Query path with remote read

• A stateless gateway accepts PromQL queries. It parses, plans, and splits the request into subranges and label filters.
• A query coordinator fans out to storage nodes using the inverted index to find series quickly.
• Results stream back as typed vectors, then the gateway applies PromQL operators, joins, binary math, and aggregations.
• Response caching stores final results and partial range results keyed by query plus time window.

8. High availability and duplicate handling

• Run Prometheus in an HA pair with different external labels such as replica A and replica B.
• Ingestion deduplicates by series fingerprint and timestamp, preferring one replica if values collide within a small window.
• Replicate chunks across zones asynchronously and verify with periodic scans.

9. Multitenancy and limits

• Partition by tenant id up front and propagate it through every pipeline stage.
• Enforce budgets. samples per second, active series, max label names, query time range, query complexity.
• Offer soft limits with admission control and hard limits with request rejection and clear errors.

10. Backfill and repair

• Provide a backfill API that accepts historical blocks produced by a tool. Useful for import from another system or manual fixes.
• A repair job scans for missing or corrupt blocks, rebuilds indexes, and replays the write ahead log when needed.

11. Security and governance

• TLS in transit and encryption at rest.
• Per tenant API tokens, optional mTLS for Prometheus servers.
• Audit logs for configuration and data access.