Datadog System Design Interview Questions and How to Answer Them

A Worked Outline: Design a Metrics Monitoring System

Use this order. It fits the 60 minutes and covers what the interviewer scores.

• Set the numbers first. Assume 10 million hosts sending 100 metrics each minute. That is about 16 million writes per second. State the number, then design for it.
• Define a data point. A metric name, a timestamp, a value, and a set of tags. Tags are labels such as host or region.
• Name the cardinality problem. Cardinality is the number of distinct tag combinations. One bad tag, such as a user id, can create millions of series and break storage.
• Write path. Agents batch and compress. Intake servers validate and push to a partitioned log. Partition by metric name and tag hash so one series always lands in the same place.
• Storage. Use a time series store with column layout and delta encoding. Recent data stays on fast disks. Older data moves to object storage.
• Rollups. Precompute one minute, one hour, and one day summaries. A dashboard showing a month of data reads the hour rollup, not raw points.
• Query path. Push filters down to storage. Cache dashboard results for a few seconds, since many users watch the same charts.
• Alerting. Evaluate rules on the stream, not by querying storage. Group alerts so one incident does not send a thousand messages.

The Tradeoffs to Say Out Loud

Interviewers score the tradeoffs more than the diagram.

• Accuracy against cost. Sampling traces at one percent saves most of the storage bill and loses rare events. Say which you would choose and why.
• Freshness against load. A five second delay on dashboards lets you batch writes and cut cost.
• Retention. Raw data for a short window, rollups for a long one. Say the numbers you picked.
• Durability. Losing one metric point is acceptable. Losing an alert is not. Treat the two paths differently.

What Gets Candidates Rejected

The common failure is designing a normal web application. A relational table of data points does not survive 16 million writes per second. Say the write rate early and let it force your choices.

The second failure is silence about cardinality. It is the first thing a monitoring engineer thinks about.

How to Prepare

• Build the base first. Grokking System Design Fundamentals covers partitioning, caching, and queues.
• Practice full designs under time. Grokking the System Design Interview includes the metrics and logging problems this round uses.
• Read the shorter Datadog list. Datadog system design interview questions covers extra variants.
• Compare with other companies. The Uber design questions and Snowflake design questions use the same data heavy patterns.
• Do one live practice round. Mock interviews with a working engineer show where you run out of time.
• Know the other rounds too. See Datadog coding interview questions.