Grokking DevOps and SRE Interviews
Vote

0% completed

​

TLS and Certificates

The Handshake, Simplified

Certificates and Chains

Reading openssl s_client Output

Expiry, SNI, and Name Mismatch

Mutual TLS

Automated Renewal with ACME

Using This in an Interview

Key Takeaways

Practice Questions

Title image: a server certificate linked through an intermediate certificate to a trusted root.
Title image: a server certificate linked through an intermediate certificate to a trusted root.

TLS (Transport Layer Security) protects a connection between a client and a server, such as a browser and shop.example.com. It promises three things:

  • Confidentiality: no one on the network path can read the data.
  • Integrity: no one on the path can change the data without detection.
  • Authentication: the server proves that it holds the private key, a secret that only the server knows, for a certificate that names shop.example.com.

It does not hide which IP address the client contacts, and the hostname is usually visible too. It does not protect data after decryption, on the server, in logs, or at a proxy that ends TLS. It also does not show that the site is honest. This lesson explains how the third promise works and why it causes most TLS incidents in production.

The Handshake, Simplified

Before any application data moves, the two sides agree on keys, and the server proves its identity. In TLS 1.3, the client sends a ClientHello with the TLS versions and cipher suites (sets of encryption algorithms) it supports, its share of a key exchange, and the server's name. The server replies with its choice, its own key share, its certificate, a signature, and a Finished message.

Both sides compute the same secret from the two key shares, although the secret never crosses the network. The signature proves that the server holds the private key that matches its certificate. The client checks the certificate, sends its own Finished message, and then sends the request.

A simplified TLS 1.3 handshake after the TCP handshake: one round trip for TCP and one for TLS before the first request, 80 ms at a 40 ms round trip.
A simplified TLS 1.3 handshake after the TCP handshake: one round trip for TCP and one for TLS before the first request, 80 ms at a 40 ms round trip.

TLS 1.3 needs one round trip and TLS 1.2 needs two. At a 40 ms round trip, a new connection spends 40 ms on TCP and 40 ms on TLS 1.3, so the first request leaves after 80 ms. With TLS 1.2, it leaves after 120 ms.

Certificates and Chains

A certificate links a public key to one or more names, and a certificate authority signs that link. The public key is the half of the key pair that anyone may see, and it checks signatures made with the private key. A certificate authority (CA) is an organization that clients trust to check who controls a name. The certificate lists its names in the subject alternative name (SAN) field, a start and end of validity, the public key, the issuer, and the issuer's signature.

Clients do not trust the server's certificate directly. The server certificate, often called the leaf, is signed by an intermediate certificate, and the intermediate is signed by a root certificate. Roots are stored in the client's trust store, the list of root certificates shipped with the operating system, the browser, or a language runtime.

The certificate chain: the server sends its leaf and the intermediate, and the client checks each signature up to a root in its own trust store.
The certificate chain: the server sends its leaf and the intermediate, and the client checks each signature up to a root in its own trust store.

The server must send the leaf and every intermediate, but not the root. A missing intermediate causes a common incident. Some browsers complete the chain from a cache or by downloading the intermediate, so the site works in a browser. Programs such as curl and Python fail with an error about the local issuer certificate.

Reading openssl s_client Output

openssl s_client performs the handshake and prints the chain that the server sent.

openssl s_client -connect shop.example.com:443 -servername shop.example.com </dev/null
CONNECTED(00000003) depth=0 CN = shop.example.com verify error:num=20:unable to get local issuer certificate verify error:num=21:unable to verify the first certificate Certificate chain 0 s:CN = shop.example.com i:C = US, O = Example Trust, CN = Example Issuing CA New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Verify return code: 21 (unable to verify the first certificate)

The chain has one entry, numbered 0. Its subject (s:) is the site, and its issuer (i:) is an intermediate that the server did not send. The client cannot connect the leaf to a root, so verification fails with code 21. A healthy result prints depth lines for 2, 1, and 0 and ends with Verify return code: 0 (ok).

Note that s_client reports verification errors but still completes the connection. It also skips the hostname check unless you add -verify_hostname shop.example.com.

Expiry, SNI, and Name Mismatch

Three more errors are common.

  • Expiry. An expired certificate fails with certificate has expired (code 10). Public certificates may now last at most 200 days, and that limit falls in two more steps to 47 days. Let's Encrypt, a free CA, issues 90-day certificates by default and plans to move to 45 days. To test a saved certificate, openssl x509 -in cert.pem -noout -checkend 1209600 prints Certificate will expire if it ends within 14 days (14 x 86,400 seconds).
  • SNI. Server Name Indication places the hostname in the ClientHello, so a server that hosts many sites on one IP address can return the right certificate. A client that omits it receives the server's default certificate. s_client sends SNI from the -connect name when it is a DNS name, so pass -servername when you connect to one backend by IP address.
  • Name mismatch. The requested name must appear in the SAN list. A wildcard such as *.example.com stands for exactly one part of a name, so it covers shop.example.com, but not example.com or api.shop.example.com. With -verify_hostname, a mismatch fails with code 62.

Mutual TLS

In mutual TLS (mTLS), the client also presents a certificate, and the server verifies it. The server checks the client certificate against a CA that it trusts, often an internal one. Services use mTLS to prove their identity to each other without passwords. A service mesh, a layer of proxies that carries traffic between services, can issue these certificates and replace them automatically before they expire. A client certificate that expires breaks the connection in the same way as a server certificate.

Automated Renewal with ACME

ACME (Automatic Certificate Management Environment) is the protocol that lets a program prove control of a name and receive a certificate without a person. Certbot and cert-manager, a Kubernetes add-on, are common ACME clients. The CA issues a challenge, and the client answers it in one of two ways:

ChallengeHow the client proves controlTypical failure
HTTP-01Serves a token at http://<name>/.well-known/acme-challenge/ on port 80Port 80 blocked, or a redirect sends the request elsewhere
DNS-01Creates a TXT record at _acme-challenge.<name>The DNS provider credential has expired

Let's Encrypt requires DNS-01 for wildcard certificates. Let's Encrypt recommends renewal at about two-thirds of a certificate's lifetime, so a 90-day certificate renews near day 60, with 30 days left.

Renewal can still fail without anyone noticing. The client may fail every challenge, or it may write a new file that the server never loads because nobody reloaded it. Therefore, monitor the certificate that the server actually returns, from outside. If the renewal on day 60 fails, an alert at 14 days remaining fires on day 76 and still leaves 14 days to fix the problem.

Using This in an Interview

A candidate report for a systems development role describes a request to explain the TLS handshake step by step. Explain the steps, then show that you can debug them.

"After the TCP handshake, the client sends supported versions, cipher suites, a key share, and SNI. The server returns its key share, its certificate chain, and a signature that proves it holds the private key. The client checks the chain to a trusted root, the expiry dates, and the hostname. In production I test this with openssl s_client and check the chain, the verify code, and the names."

Expect two follow-up questions:

  • "The site works in a browser but fails from curl. Why?" The server probably omits an intermediate certificate, and the browser completes the chain from its cache or by downloading the intermediate. Check the chain with s_client and fix the server configuration.
  • "How do you prevent certificate expiry incidents?" Automate renewal with ACME, reload the server after renewal, and monitor the served certificate from outside.

Key Takeaways

  • TLS provides confidentiality, integrity, and server authentication, but it does not hide the destination or protect decrypted data.
  • TLS 1.3 adds one round trip after the TCP handshake.
  • The server must send its full chain except the root.
  • Most certificate failures are expiry, a missing intermediate, a missing SNI, or a hostname that is not in the SAN list.
  • Monitor the certificate that clients receive, not only the renewal job.

The next lesson, What Happens When You Type a URL, places DNS, TCP, and TLS in the full path of one web request.

Practice Questions

1. A server returns only its leaf certificate. Why might Python clients fail while a browser succeeds?

<details><summary>Show answer</summary>

Python cannot build a chain to a trusted root without the intermediate. Python uses only the certificates that the server sent and its trust store, so it reports an unknown local issuer.

</details>

2. One IP address hosts two sites, and a client that does not send SNI receives the wrong certificate. Why?

<details><summary>Show answer</summary>

Without SNI, the server cannot know which site the client wants. It returns its default certificate, which names the other site, so the hostname check fails.

</details>

3. ACME renewed a certificate on day 60, but users saw it expire on day 90. What probably happened?

<details><summary>Show answer</summary>

The server kept the old certificate in memory because it was never reloaded. The new file existed on disk, but the running process did not read it. Add a reload after renewal, and monitor the certificate that clients actually receive.

</details>

Reading Progress

0%


Vote for new content

On This Page

The Handshake, Simplified

Certificates and Chains

Reading openssl s_client Output

Expiry, SNI, and Name Mismatch

Mutual TLS

Automated Renewal with ACME

Using This in an Interview

Key Takeaways

Practice Questions