How do you design centralized logging (schema, retention, access)?

Centralized logging is the flight recorder for your services. It lets you search, trace, and explain what happened across many components in one place, which is vital during incidents and during a system design interview.

Introduction

Centralized logging collects logs from many services into a single platform for ingest, storage, search, and analysis. The core design choices are a clean schema for consistency, a smart retention plan that controls cost, and a secure access layer that protects sensitive data. You will also plan for scale, reliability, and performance so engineers can query quickly during an outage.

Why It Matters

Logs are the first draft of truth in distributed systems. A unified schema unlocks correlation across services and regions. Tiered retention keeps hot data ready for fast search while pushing older data to cheaper storage to control spend. Strong access controls and audit trails protect privacy and support compliance. Together these choices enable scalable architecture, faster mean time to detect and resolve, and interview ready explanations of observability trade offs.

How It Works Step by Step

Step 1. Clarify goals and constraints Write down search latency targets, ingest rate, peak traffic patterns, retention windows for hot warm and cold data, privacy and regulatory rules, and an allowed cost envelope.

Step 2. Collect logs Use standard emitters in services that write structured JSON to stdout, then ship with a lightweight agent on each node or as a sidecar. Support system logs, application logs, access logs, and audit logs. Batch, compress, and send over TLS to a collector that can buffer through spikes.

Step 3. Ingest and transform Push events through a queue for backpressure safety. Parse into a canonical schema, enrich with metadata like tenant, environment, region, and service version, then validate types. Drop or sample noisy categories and redact secrets before storage.

Step 4. Unify the schema Pick a clear set of top level fields for correlation and indexing, then nest optional details. At minimum include time, severity, service identity, request correlation, and user or tenant where applicable. Favor consistent names across teams so dashboards and alerts are reusable.

Example schema in JSON

{
  "timestamp": "2025-11-12T15:04:05.123Z",
  "level": "info",
  "service": {
    "name": "checkout",
    "version": "1.42.0",
    "env": "prod",
    "region": "ap-south-1",
    "host": "ip-10-0-1-23"
  },
  "correlation": {
    "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
    "span_id": "00f067aa0ba902b7",
    "request_id": "rk_8VfG3"
  },
  "actor": {
    "user_id": "u_12345",
    "tenant_id": "t_987",
    "ip": "203.0.113.42"
  },
  "http": {
    "method": "POST",
    "path": "/api/pay",
    "status": 200,
    "latency_ms": 143
  },
  "event": {
    "name": "payment_authorized",
    "code": "PAY_200",
    "message": "Authorization ok"
  },
  "tags": ["payments", "success"],
  "pii_flags": ["none"]
}

Step 5. Storage and retention strategy Use a hot tier for the most recent data where interactive search must be sub second. Keep a warm tier for thirty to ninety days with cheaper storage and slightly slower search. Archive to a cold tier in object storage for six months to several years, with query on read for audits. Define deletion rules for personal data and apply immutable vaults for regulated audit logs.

Step 6. Indexing and query patterns Index a small set of high value fields such as timestamp, level, service.name, correlation.trace_id, actor.tenant_id, http.status, and event.code. Avoid indexing free text. For time series workloads keep time partitioned shards that roll daily or hourly depending on volume.

Step 7. Access and governance Use single sign on and role based access control. Create roles like reader, incident responder, and admin. Restrict sensitive fields with field level and document level filters. Log every query for audit. Provide saved searches, dashboards, and alerts with per team permissions. Rate limit expensive queries to protect the cluster.

Step 8. Reliability and scale Isolate collectors per region for local durability, then replicate to a central view. Size queues for hours of peak traffic. Use multi writer storage clusters with replication, snapshot backups, and disaster recovery procedures that are tested. Track ingest lag and search p ninety five as service level indicators with error budgets.

Step 9. Cost management Control cardinality by bounding tag values, avoid unique ids in tag sets, and down sample verbose categories in high traffic paths. Compress at rest, keep small shards, and compact cold data. Educate teams with a log level policy that discourages debug in production.

Step 10. Operate and evolve Publish a schema registry and lint rules in CI to reject invalid fields. Maintain runbooks for agent rollout and rollback. Review top queries, orphaned dashboards, and costly indexes each quarter.