Step 7: Scalability and Performance

Problem 1: How many IDs one node can make

The 12 sequence bits give each node 4,096 IDs per millisecond, about 4 million a second. That is roughly forty times the whole system target of 100,000 a second. When a node does use all 4,096 values inside one millisecond, it waits for the next tick, a pause well under a millisecond.

That wait is also the measurement that matters. Logging every request is impractical at this rate, so nodes export counts: IDs per second and sequence use per millisecond. Nodes that repeatedly reach 4,090 or more within a millisecond are at capacity, and the region needs more nodes.

Callers can reduce their own cost with batching. An ID is computed, not fetched, so a cache has nothing useful to keep. Instead, a client that needs many IDs asks for up to 1,000 in one call. The client then pays one network round trip instead of a thousand.

Problem 2: How many nodes the format allows

Every node needs a machine number that no other live node holds, and the ten machine bits allow 1,024 of them. In this design they are split five and five: 32 regions, each with 32 node numbers. Because the region is part of the machine number, two regions can never produce the same ID, even in the same millisecond with the same sequence.

Adding a node means asking the coordinator for a free number. There is no data to move and no leader to elect, so capacity grows linearly with the node count, up to the 32 numbers a region owns. A new region is one more range of numbers.

This is the ceiling the format reaches first. A node runs out of CPU long before it runs out of sequence values, so the answer to more load is always more nodes, and every node uses up a machine number. Widening the machine field later changes the ID format, so the split between region and node bits should be planned early.

The coordinator itself scales easily. ZooKeeper stores only tiny entries and handles a thousand or more generator sessions. It is not called per request, so if it becomes unreachable, running nodes keep generating IDs. Only a new node cannot get a number.

Problem 3: How many nodes a region needs

Back to US-East and its 40,000 requests a second. Load tests show one node serving 25,000 a second at 60 percent CPU, the level at which latency stays flat. Two nodes cover the peak, a third covers one failure, and a fourth covers a deploy. The region runs four nodes and uses four of its 32 machine numbers. CPU is the limit it will meet first, and more nodes solve it.

Spreading the load is simple, because every node does the same work and none depends on another. Round-robin balancing is enough, and a node that fails its health check stops receiving traffic. Any node can serve any request, and callers use the closest region only because the network round trip is nearly all of what they wait for. A region short on capacity can send some calls to a quieter one, because region choice affects speed, never correctness.

The in-region latency target of 1 to 5 ms is easy to meet, because each request is a local computation plus one network hop. The p99 rises only when something pauses, like a node waiting for the next millisecond or a garbage collection stop. Step 2 treats a p99 above 10 ms as a fault, so a recurring pause is fixed rather than tolerated.

Keeping it available

No database sits on the path of an ID, the coordinator is replicated, and the load balancer runs as several instances or as a managed service. Regions are independent, so losing one leaves the others serving, at higher latency for its clients. The one setting every node must share is the epoch. A node with a different epoch produces IDs that can collide or sort incorrectly, so the epoch belongs in code and is checked at deploy time.

Opaque IDs need even less: no machine numbers and no coordinator at all.

What the design traded away

The design accepts machine number coordination and clock management in exchange for throughput. It gives up strict global order, which would need one sequence or a global agreement for every ID, and neither fits 100,000 IDs a second. On the opaque path, it accepts a negligible chance of a UUID collision in exchange for no coordination at all.

Offering two ID types puts the choice with each caller, and the wrong choice has real costs. A Snowflake ID in a public URL lets someone guess neighboring IDs. A UUID as a primary key doubles the key size in every index and slows inserts on a large table.