Service Discovery Patterns: DNS, Consul, and Kubernetes-Native Approaches

• Kubernetes-only environment, single cluster: Use Kubernetes-native discovery. It is built-in, requires no additional infrastructure, and integrates with health checking natively. This covers the majority of use cases for teams running entirely on Kubernetes.

• Multi-platform (Kubernetes + VMs + bare metal): Use Consul. Its agent-based model works on any platform, and its DNS interface provides backward compatibility with applications that cannot use Consul's API directly.

• Multi-datacenter or multi-cloud: Use Consul with WAN federation. Kubernetes DNS does not span clusters. Consul's multi-datacenter federation enables cross-region service discovery without custom solutions.

• Simple, small-scale deployment: DNS-based discovery with low TTLs and external health checking may be sufficient. It requires no additional infrastructure beyond your existing DNS server.

• Service mesh requirements: Consul Connect or Istio/Linkerd on Kubernetes. Service meshes add mTLS, traffic policies, and observability on top of discovery. Choose based on your platform.

Service Registration Patterns

How services get into the registry is as important as how they are discovered.

Self-Registration

The service instance registers itself with the registry on startup and deregisters on shutdown. This is the pattern Consul uses: each service calls the Consul agent's registration API during initialization.

The service is responsible for its own lifecycle in the registry.

Advantage: No external dependency for registration. The service knows when it is ready.

Disadvantage: Registration logic is coupled to the application. Every service in every language must implement registration.

Third-Party Registration

An external component (the platform) handles registration automatically.

The service does not know about the registry. This is how Kubernetes works: when a pod is created, the Kubernetes control plane automatically adds its IP to the corresponding Service's endpoint list. When the pod is deleted, Kubernetes removes it.

Advantage: Zero registration code in the application. Services are unaware of the discovery mechanism.

Disadvantage: Depends on the platform. Services running outside Kubernetes (VMs, serverless) cannot use this pattern without additional tooling.

Sidecar Registration

A sidecar proxy (Envoy, Consul Connect proxy) handles registration, health checking, and discovery on behalf of the application. The application communicates only with localhost. The sidecar handles all service mesh concerns. This is the pattern used by Istio, Linkerd, and Consul Connect.

Advantage: Application is completely decoupled from discovery. Works in polyglot environments.

Disadvantage: Adds a sidecar container to every pod, consuming additional CPU and memory.

DNS Record Types for Service Discovery

Understanding DNS record types is important for implementing DNS-based discovery correctly.

• A records map a hostname to an IP address. The simplest form of service discovery: payment-service.internal → 10.0.1.5. Multiple A records for the same name enable basic round-robin load balancing. The DNS client receives all IPs and typically connects to the first one (or a random one, depending on the resolver).
• SRV records provide richer information: priority, weight, port, and target hostname. SRV records enable weighted load balancing (send 70% of traffic to one instance, 30% to another) and priority-based failover (prefer primary instances, fall back to secondary). Consul and Kubernetes CoreDNS both support SRV queries.
• CNAME records alias one domain name to another. Useful for pointing a friendly service name to a load balancer's hostname: payment.api.internal → payment-lb.us-east.elb.amazonaws.com. The CNAME is resolved to the load balancer's A records, which resolve to the actual IPs.

Service Discovery and Service Mesh

Service meshes build on service discovery by adding security (mTLS between services), traffic management (canary deployments, circuit breaking, retries), and observability (distributed tracing, metrics) to the communication layer.

Istio on Kubernetes uses the Kubernetes API server as its service registry. Envoy sidecar proxies intercept all traffic and apply routing rules defined by Istio's control plane. Discovery is Kubernetes-native; the mesh adds the security and traffic management layer.

Consul Connect uses Consul's service catalog as the registry and deploys sidecar proxies for mTLS and access control. Because Consul works across platforms (Kubernetes, VMs, serverless), Consul Connect provides service mesh capabilities in heterogeneous environments where Istio (Kubernetes-only) cannot reach.

Linkerd is a lightweight Kubernetes-native service mesh that uses Kubernetes Services for discovery and adds mTLS, retries, and observability with minimal configuration overhead.

For teams already running Kubernetes-native discovery, adding a service mesh is an incremental step that preserves the existing discovery model while adding security and traffic management.

Real-World Patterns

Single Kubernetes Cluster (Most Common)

Use Kubernetes Services with CoreDNS.

Define readiness probes for health checking.

Use headless Services for gRPC.

No additional infrastructure needed. This covers 80% of microservices architectures.

Kubernetes + Legacy VMs

Deploy Consul alongside Kubernetes.

Kubernetes services register with Consul via the Consul-Kubernetes sync feature.

VM services register directly with Consul agents.

Both Kubernetes and VM services discover each other through Consul's DNS or HTTP API.

Multi-Region Active-Active

Deploy Consul clusters in each region with WAN federation. Services register locally. Cross-region discovery enables failover: if all instances in Region A fail, clients discover instances in Region B through Consul's federated catalog.

Health checks run locally in each region, and only healthy instances are returned.

Hybrid Cloud (AWS + On-Premises)

Consul agents run in both environments.

On-premises services register with the on-premises Consul cluster.

AWS services register with the AWS Consul cluster.

WAN federation connects both clusters.

Services discover each other regardless of where they run.

This pattern is common in enterprises migrating from on-premises to cloud, where some services have moved to Kubernetes on AWS while others remain on legacy VMs in the data center.

Consul provides a unified discovery layer that bridges both worlds during the transition.

How This Shows Up in System Design Interviews

Service discovery comes up when interviewers ask about microservices communication, scaling, or deployment architecture.

Here is how to present it:

"For service-to-service communication, I would use Kubernetes Services for discovery within the cluster. Each microservice is exposed via a ClusterIP Service with a stable DNS name. Readiness probes ensure only healthy pods receive traffic. For the gRPC service that needs client-side load balancing across multiple pods, I would use a headless Service so the client receives all pod IPs. If the architecture spans multiple clusters or includes non-Kubernetes workloads, I would add Consul for cross-platform discovery with health checking and WAN federation for multi-region failover."

For structured practice on system design problems involving service communication patterns, the Grokking the System Design Interview course covers microservices architecture and infrastructure patterns.

Common Mistakes

1. Hardcoding IP addresses or hostnames in application code. Even in Kubernetes, hardcoding pod IPs breaks when pods restart. Always use service names and let the discovery mechanism resolve them.

2. Using DNS with high TTLs for dynamic environments. A TTL of 300 seconds means a dead instance receives traffic for up to 5 minutes. In environments where instances scale frequently, use TTLs of 5 to 30 seconds or switch to a registry-based approach like Consul or Kubernetes Services.

3. Ignoring the gRPC load balancing problem. gRPC uses long-lived HTTP/2 connections. A Kubernetes ClusterIP Service load-balances at connection time, not per-request. Once a gRPC client establishes a connection to one pod, all requests go to that pod. Use headless Services with client-side load balancing or a service mesh for gRPC.

4. Deploying Consul when Kubernetes-native discovery suffices. If all your services run in a single Kubernetes cluster, Consul adds operational complexity without significant benefit. Kubernetes Services handle discovery, health checking, and load balancing natively.

5. Not implementing graceful shutdown. When an instance shuts down, it should deregister from the service registry before stopping. Without graceful shutdown, the registry continues routing traffic to the dying instance until health checks detect the failure. In Kubernetes, the pre-stop hook and readiness probe handle this automatically if configured correctly.

6. No fallback when the registry is unavailable. If Consul's servers are temporarily unreachable, client applications should cache the last known good set of endpoints and continue operating. A service discovery outage should not cascade into a full system outage.

Conclusion: Key Takeaways

• Service discovery replaces hardcoded addresses with dynamic lookups. Services register their instances, and clients query the registry to find healthy endpoints. This is essential for any microservices architecture.

• Two patterns: client-side and server-side. Client-side (client queries registry directly) gives more control. Server-side (router handles discovery) is simpler for applications. Kubernetes Services are server-side. Consul supports both.

• DNS is universal but limited. No health checking, TTL caching delays, limited metadata. Works for simple, small-scale deployments.

• Consul is the multi-platform, multi-datacenter choice. Agent-based, distributed health checking, DNS and HTTP interfaces, WAN federation, key-value store. Use when spanning Kubernetes, VMs, and multiple regions.

• Kubernetes-native discovery is the default for single-cluster Kubernetes. Built-in, zero additional infrastructure, integrated health checking. Use headless Services for gRPC client-side load balancing.

• In interviews, match the approach to the infrastructure. Kubernetes-only? Native Services. Multi-platform? Consul. gRPC? Headless Services. Multi-region? Consul federation. Name the specific pattern and tool.