Step 5: High-Level Design

The system has one job: whenever an application creates a record, it hands back a unique 64-bit ID, either sortable by time or opaque. We follow one request through it: Priya signs up in US-East, and the accounts service needs an ID for her new row before it can insert it.

First, a decision: a library or a service?

The Snowflake format from earlier steps needs no coordination per ID. A process that knows its own machine number, the slot in the ID that no other generator uses, can produce IDs without contacting anything. That leaves a real choice about where the code runs when the accounts service needs Priya's ID.

As a library. The accounts service imports a package, receives a machine number at startup, and makes Priya's ID inside its own process, at a cost of a few processor instructions and no network.

As a service. The accounts service calls an ID service over the network and waits for the answer before inserting the row. Twitter's original Snowflake worked this way.

(A third option, generating IDs inside the database, as Instagram does with a stored function in PostgreSQL, ties ID generation to one storage engine, so we leave it aside.) On every machine-level measure, the library wins:

A service puts a network hop in front of every insert, and it makes ID generation a shared dependency whose outage stops writes everywhere.

The reasons to build one anyway concern organizations, not machines. First, the ID has room for only 1,024 machine numbers, and every generating process needs its own. Handing them out across deploys and autoscaling is hard when there are thousands of processes, while a service needs only a few dozen long-lived nodes. Second, a library must be kept correct in every client language, so one clock bug can exist in several places at once. Third, a format change, like moving the epoch (the date the time count starts from), forces every library client to redeploy, while a service needs one rollout.

The practical answer is both. A few high-volume services that can be trusted with a machine number use the library. Everything else calls the service and asks for up to a thousand IDs per call. Both draw machine numbers from the same pool of 1,024, through the same coordinator, because a library ID must never collide with a service ID. The diagram below compares the two ways of shipping the algorithm.

Following Priya's request through the service

The service is the harder of the two to design, so the rest of this lesson follows Priya's request through it. It runs in many regions with many nodes in each, and its parts need very little from each other.

Before any request: a node gets its machine number. When a generator node starts, it asks the coordinator, a small ZooKeeper or etcd cluster, for a free machine number in its region's range. The coordinator holds that number for the node while the node stays alive, and no other live node can have it. This is the only time a node talks to the coordinator. If the coordinator goes down, existing nodes keep generating, and only growth stops, because no new node can get a number.

Then Priya's request runs in five steps:

1. The accounts service calls the API. It sends POST /v1/ids with type set to sortable.
2. The gateway picks the nearest node. DNS, an anycast address, or a cloud load balancer routes the call to a healthy generator node in the closest region, here US-East. Any node can answer, and almost all of the caller's wait is network distance. If US-East is overloaded or down, the call goes to another region at a latency cost, and a client should retry a failed call once before reporting an error.
3. The node builds the ID. The node, which holds region 1, host 3 (the next section explains these numbers), reads the current time as a 41-bit count of milliseconds since the epoch. It combines that with its region and host numbers and with sequence 5, because it has already issued five IDs in this millisecond. It asks no other node for anything. Its machine number, last timestamp, and counter are private, and nothing about the request is kept afterward, so the node is stateless in the sense that matters: losing it costs only its capacity.
4. The node returns the ID as a string, usually within a few milliseconds, because JavaScript would lose the low digits of a 64-bit number.
5. A failed call is simply retried, possibly on another node. The retry makes a new ID, and an ID that was made but never delivered is lost without harm.

An opaque ID takes the same path, except that in step 3 the node fills a UUIDv4 from a secure random number generator.

How regions avoid collisions without talking

At almost the same moment, a node in Europe issues its own ID. Regions never consult each other, so the ID itself must keep them apart. The 10-bit machine field is split into a 5-bit region number and a 5-bit host number, so each region owns 32 machine numbers: region 0 holds 0 to 31, region 1 holds 32 to 63, and so on. Europe is region 2.

Suppose the Europe node had the same timestamp and the same sequence as Priya's ID. The region bits still differ, so the two 64-bit values still differ, and no message passed between the regions. This holds only if no two regions ever share a range, which is why the ranges are fixed by configuration rather than agreed at runtime.

Sorted together, the two IDs fall in time order. When the timestamps match to the millisecond, the region bits decide, so strict time order breaks only for IDs made in the same millisecond. Opaque IDs need no region bits, because their randomness alone makes them unique.

The design deliberately has no database of counters. The segment mode from Step 6, where nodes fetch blocks of IDs from a counter store, would put that store on the path of every refill. It remains a fallback for places where clocks cannot be trusted.