Unique ID Generator: Scalability and Performance

The three numbers that decide capacity

Capacity here is decided by three numbers and nothing else: nodes, machine numbers, and sequence bits. Everything in this step is about one of the three.

Why is there nothing to shard? Sharding means splitting work across machines so each one handles a separate part. Snowflake already does that inside the id. Each node holds its own machine number, so the ids it makes are ones no other node can make.

That is why any request can go to any node. For sortable ids we prefer a node in the caller's region, but only for latency.

To add capacity, add a node. The coordinator hands it a free machine number, and that number is its slice of the id space. Capacity grows in a straight line with the number of nodes, up to the number of machine numbers.

Regions work the same way. Each region owns its own range of machine numbers, so a new region is one more range.

The segment approach from Step 6 would have needed real sharding: separate counters, or several sequence tables. We keep the database out of generation, so that problem never appears.

Spreading the load. Every node does the same work and shares no state. So plain round-robin or least-connections spreads load well enough. A cleverer policy would gain nothing.

Give each node a health check, a ping or a test id. A node that fails or slows down is marked unhealthy, and the load balancer stops sending to it.

CPU is the main cost per request, so watch CPU and add nodes when it stays high. If one region carries more load than it should, give it more nodes. Or send some of its calls to a quieter region, at a small latency cost. That is safe because region choice is about speed, not correctness.

Is there anything to cache? Almost nothing. An id is computed, not fetched, so there is no earlier answer worth keeping. The segment approach prefetches its next block to hide the database wait, and we do not use that approach.

What helps instead is batching. A client that needs many ids asks for a batch, say 100 in one call, and uses them locally. Making 100 ids on a node takes about a millisecond.

So the client pays one round trip instead of 100. This is a client-side option, and the service already supports it.

What if a database counter cannot be avoided? Then split its key space rather than run one counter faster. One database gives out odd ids and another gives out even ids. Throughput doubles and nothing collides. The general form is mod N sharding: N counters, and id = sequence_value * N + shard_index.

The price is global order. An id from counter 0 can be larger than an id from counter 1 that was made later. Snowflake needs none of this, because it keeps the database out of the path of every id. There is no multi-master database problem to solve.

Counters have a second answer for heavy clients. Use one counter per kind of id, one for users, one for orders, and so on. Each then scales on its own. Snowflake and UUID need none of that. One service serves every kind of id, because every id is unique everywhere.

A caller that wants an id to show its type can add a prefix like "user_" or "order_" in its own code. Cronofy does this. It sits outside this service.

Does the coordinator scale with us? ZooKeeper stores only tiny entries, so it handles 1000+ nodes with ease. Spread its members across racks and zones, so one failure does not take it down.

Tens of thousands of generator nodes would be too many sessions and heartbeats for one cluster. We are unlikely to get there. If we did, we would run one small coordinator per region.

Regions already own separate machine numbers, so ids stay unique under that split. Each regional coordinator then tracks a few dozen machines, say 64 with a 6-bit machine field. Several small clusters replace one large one, and each has a light load.

Spare capacity is just as cheap. Extra nodes can sit idle and take over on a failure or a spike. Since adding a node is so easy, it is simpler to always run N+1 nodes above the expected need. An unassigned machine number costs nothing.

Single points of failure. A single point of failure is one part whose loss stops the whole system. Most of them are already gone from this design:

• No database is on the path of an id. Snowflake removes it.
• The coordinator is replicated, and it is not called per request.
• The load balancer can be redundant, with several instances or a cloud service that already is.
• Regions are independent. Losing one does not stop the others.
• If a whole region is down, the other regions keep serving, at higher latency for its clients.
• The epoch, the start time the timestamp counts from, must be the same on every node. A node with a different epoch makes ids that can clash or sort wrong. Put it in code or one shared config, and check it on deploy.

The epoch is the last value every node must agree on. Everything else on that list is already independent.

What about threads inside one node? If several threads compete for the sequence counter, give each thread half the range. Each thread then acts as its own small generator, with a hidden thread bit.

At our target this is not needed. An AtomicInteger in Java increments millions of times a second. Revisit only if profiling shows contention.

Opaque ids need none of this. Random ids need no sharding, because the collision odds are already too small to matter. Each node works alone.

If we wanted extra certainty, we could make 120 random bits and append 8 bits of machine number. Two machines could then never make the same value, and 120 random bits keep almost the same collision odds.

We do not do it, because that is not standard UUID and it is not needed. UUIDv1 and v2 include a MAC address, and v4 is pure random. Machine bits would also make the id a little less opaque. Someone who knew the layout could read the machine from it, so we keep to standard random.

Latency. The work per request is a local computation plus one network hop, so the target from Step 2 is easy to meet. Inside a region expect 1 to 5 ms.

The p99, the time that 99 of every 100 requests beat, rises only when something pauses. A node that fills its sequence and waits a millisecond is one pause. A garbage collection stop in a managed language is another. Any pause halts generation.

Step 2 treats a p99 above 10ms as a fault. So fix a recurring pause rather than accept it. Pick languages or settings that keep those pauses short, and monitor latency.

Cross-region failover can cost 100 ms or more, but it is rare. There is no heavy I/O per request, so meeting the target is mostly a matter of keeping nodes healthy under load.

What to log at this rate. Logging every request is impossible, so sample the logs or keep counts instead. Each node counts the ids it makes and exports metrics like ids per second and sequence use.

Those counts are what show when a limit is near. Sequence values that keep reaching 4090+ inside a millisecond mean that node is at capacity. Add nodes.

Clock drift can be watched by comparing timestamps in ids from different nodes now and then. Something simpler is enough. Monitor NTP and alert on any large adjustment.

What we traded away. The design takes on some coordination and clock management. In return it gets throughput and scale. What we gave up is strict global order. That would need one sequence, or a global agreement for every id, and neither fits the throughput target.

We get approximate order by timestamp instead. Sorting by id gives almost creation order, with rare inversions when two clocks differ by a few milliseconds. That is enough for most uses.

We also accept the tiny chance of a UUID collision in exchange for no central coordination. The odds make that a good trade.

Two id types give callers a choice. Use Snowflake ids when sorting or a small key matters. Use UUIDs when the id must not be guessable. The cost is two code paths, which is minor.

Using the wrong type has real costs. A Snowflake id in a public URL lets someone guess a neighboring id and fetch data. A UUID as a primary key doubles the key size in every index. So each service picks the type that fits its use.

The shape to remember is that this service scales by addition and nothing else. There is no data to rebalance, no cache to warm, and no leader to elect. Adding a node means asking the coordinator for a free machine number, and that is the whole operation.

Next: Step 8, which says what happens when the clock goes backwards.