Grokking DevOps and SRE Interviews
Vote

0% completed

​

How DNS Resolves a Name

The Servers in a Lookup

Record Types

TTL and Caching

Negative Caching

The Resolver Configuration

Reading dig Output

Common Failures

Using This in an Interview

Key Takeaways

Practice Questions

Title image: a name goes to a resolver, which asks the root, top-level, and authoritative servers and returns an address.
Title image: a name goes to a resolver, which asks the root, top-level, and authoritative servers and returns an address.

The Domain Name System (DNS) is a distributed database that maps names, such as shop.example.com, to addresses and other records. No single server holds the whole database. Each part of a name is managed by different servers, and caches at several points keep copies of recent answers.

Almost every connection begins with a DNS lookup. When the lookup fails or returns an old answer, the error often looks like a network or application failure instead.

The sections below follow one lookup from an application to the server that holds the answer, and then show how to read the result and recognize its common failures.

The Servers in a Lookup

A lookup passes through up to five kinds of DNS software, and each one either answers from its cache or asks the next.

  • Stub resolver. This small library inside the operating system reads /etc/resolv.conf and sends the question to a recursive resolver.
  • Recursive resolver. This server, run by the company, the cloud provider, or a public service, finds the answer for the client and caches it.
  • Root servers. They do not know the address, but they name the servers for each top-level domain, such as .com.
  • Top-level domain (TLD) servers. The .com servers name the authoritative servers for example.com.
  • Authoritative servers. They hold the zone, the set of records for a domain, and return the final answer.
One lookup with an empty cache: the stub resolver asks the recursive resolver, which asks the root, the .com servers, and the authoritative server, for a total of about 61 ms; a cached answer takes about 1 ms.
One lookup with an empty cache: the stub resolver asks the recursive resolver, which asks the root, the .com servers, and the authoritative server, for a total of about 61 ms; a cached answer takes about 1 ms.

With an empty cache, the lookup might take 1 ms to the resolver, 12 ms to a root server, 18 ms to a .com server, and 30 ms to the authoritative server. That is 1 + 12 + 18 + 30, or about 61 ms. The resolver then caches every answer it received, so the next client's lookup takes about 1 ms. In practice, a busy resolver already holds the root and .com answers, so most uncached lookups need only the authoritative step.

Record Types

Each DNS record has a name, a type, a time to live, and a value, and the type decides what the value means.

TypeValueExample use
AAn IPv4 addressshop.example.com to 203.0.113.10
AAAAAn IPv6 addressThe same name over IPv6
CNAMEAnother name, which the resolver then looks upshop.example.com to a load balancer's name
MXA mail server and its priorityWhere email for example.com is delivered
TXTFree textProving domain ownership to a provider
NSThe authoritative servers for a zoneDelegating example.com to its DNS provider

A CNAME cannot share its name with other records. A zone's top name, such as example.com itself, needs NS and other records, so it cannot be a CNAME.

TTL and Caching

The time to live (TTL) is the number of seconds for which a resolver may cache a record before it must ask again. A long TTL reduces lookups and keeps answers available during short outages of the authoritative servers. A short TTL makes changes take effect quickly.

Suppose that shop.example.com has an A record with a TTL of 3,600 seconds, and you change its address at 09:00. A resolver that cached the old answer at 08:59 serves it until 09:59. To shorten that window, lower the TTL to 60 seconds at least 3,600 seconds before the change, so that every cached copy of the long TTL has expired. After the change, clients then see the new address within about a minute.

Changing an address at 09:00: with a 3,600-second TTL, a resolver that cached the old answer at 08:59 serves it until 09:59; with the TTL lowered to 60 seconds in advance, the old answer expires by 09:01.
Changing an address at 09:00: with a 3,600-second TTL, a resolver that cached the old answer at 08:59 serves it until 09:59; with the TTL lowered to 60 seconds in advance, the old answer expires by 09:01.

Some programs keep their own cache, so a client can still use an old address after the resolver has the new one. Check the program's own DNS cache settings, or restart the process.

Negative Caching

A resolver also caches the answer that a name does not exist, called NXDOMAIN, for a time taken from the zone's SOA record. The SOA (start of authority) record holds settings for the zone, and the negative cache time is the smaller of the SOA record's own TTL and its last field.

dig new.example.com ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5121 ;; AUTHORITY SECTION: example.com. 900 IN SOA ns1.example.com. admin.example.com. 42 7200 900 1209600 900

Here, both values are 900, so the resolver remembers for 15 minutes that new.example.com does not exist. If a health check queried the name before you created its record, clients may receive NXDOMAIN for up to 15 minutes after you create it.

The Resolver Configuration

The stub resolver reads /etc/resolv.conf to learn which recursive resolvers to ask and how long to wait.

nameserver 10.0.0.2 nameserver 10.0.0.3 search internal.example.com options timeout:2 attempts:2

The resolver asks the first nameserver and moves to the next only after a timeout, which is 5 seconds by default; this file shortens it to 2. The search line adds a domain to short names, so db is tried as db.internal.example.com. The ndots option sets how many dots a name needs before it is tried as written first. Kubernetes pods commonly use ndots:5, so a name such as api.example.com is first tried with each search domain, which adds extra queries.

Reading dig Output

The dig command sends one DNS question and prints the full answer, so it shows what the resolver returned rather than what the application assumed.

dig shop.example.com ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41872 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; ANSWER SECTION: shop.example.com. 300 IN CNAME shop-lb.example.net. shop-lb.example.net. 60 IN A 203.0.113.10 ;; Query time: 1 msec ;; SERVER: 10.0.0.2#53(10.0.0.2) (UDP)

The status NOERROR means that the name exists. The CNAME leads to a load balancer's name, whose A record gives the address. The second column is the remaining TTL, which counts down while the answer stays in the cache. The 1 ms query time suggests a cached answer, and the SERVER line shows which resolver answered. The flags lack aa, so the answer did not come directly from an authoritative server. Running dig @ns1.example.com shop.example.com asks the authoritative server instead, and dig +trace repeats the whole path from the root.

Common Failures

SymptomMeaningFirst check
NXDOMAINThe name does not existA spelling mistake, a missing record, or a wrong search domain
NOERROR with no answerThe name exists, but not with this record typeAn AAAA query for a name with only an A record
SERVFAILThe resolver could not obtain a valid answerWhether the authoritative servers respond, and DNSSEC validation
Old answer after a changeA cache still holds the earlier recordCompare dig at the resolver with dig at the authoritative server
Slow lookupsA resolver times out before the next one is triedThe Query time in dig, and the order in resolv.conf

DNSSEC adds signatures to records so that resolvers can verify them, and a broken signature appears as SERVFAIL. A dead first nameserver adds the full timeout to every lookup, which Drill: A Slow Server That Is Not Busy investigates step by step.

Using This in an Interview

"We changed the DNS record an hour ago, but some users still reach the old server. Why?" tests caching. Explain the cache, then show how you would prove it.

"Resolvers cache the old record for its TTL, and some applications cache it longer. I would compare dig against our authoritative server with dig against the users' resolver, and check the remaining TTL. For the next change, I would lower the TTL in advance, at least one old TTL before the change."

Expect two follow-up questions:

  • "What is the difference between NXDOMAIN and SERVFAIL?" NXDOMAIN is an answer that the name does not exist, while SERVFAIL means that the resolver could not obtain any valid answer.
  • "Why not set every TTL to 5 seconds?" Resolvers would query the authoritative servers far more often, more lookups would miss the cache and wait longer, and a short authoritative outage would affect clients at once.

Key Takeaways

  • A recursive resolver asks the root, TLD, and authoritative servers and caches each answer.
  • The TTL limits how long an old answer survives, so lower it before a planned change.
  • NXDOMAIN answers are cached too, for a time set by the zone's SOA record.
  • The order and timeout in resolv.conf decide how long a failed resolver delays every lookup.
  • dig shows the status, the TTL, the answering server, and the query time.

With the name resolved to an address, the client next opens a connection. The next lesson, TCP Connections and Ports, explains how that connection is made and why it can be refused or time out.

Practice Questions

1. A record has a TTL of 86,400 seconds, and you must move the service tomorrow. What should you do today?

<details><summary>Show answer</summary>

Lower the TTL now, at least 86,400 seconds before the move. Resolvers that cached the record may keep it for up to a day, so the lower TTL takes effect only after those copies expire. Raise the TTL again after the move.

</details>

2. dig at your resolver returns the old address with a TTL of 2,400, while dig at the authoritative server returns the new address. What is happening?

<details><summary>Show answer</summary>

The resolver is serving a cached copy of the old record. It will ask again in 2,400 seconds, or 40 minutes. The record itself is correct, so the only choices are to wait or to clear that resolver's cache.

</details>

3. Every lookup on a server takes just over 5 seconds, but dig against the second nameserver is fast. What is the likely cause?

<details><summary>Show answer</summary>

The first nameserver in resolv.conf does not respond. The stub resolver waits for the 5-second default timeout before asking the second one. Remove or repair the first nameserver, or shorten the timeout.

</details>

Reading Progress

0%


Vote for new content

On This Page

The Servers in a Lookup

Record Types

TTL and Caching

Negative Caching

The Resolver Configuration

Reading dig Output

Common Failures

Using This in an Interview

Key Takeaways

Practice Questions