Unique ID Generator: Scalability and Performance

The three numbers that decide capacity

Capacity here is decided by three numbers and nothing else: nodes, machine numbers, and sequence bits.

• Why there is nothing to shard: The advantage of the Snowflake approach is that it inherently shards the ID generation by machine. Each node is a shard (identified by its machine ID bits) that can work in parallel. We don't have to manually partition requests; any incoming request can go to any node (for opaque IDs) or any specific region's node (for time-sortable, we prefer local region node for latency). The assignment of unique node IDs via coordinator is essentially partitioning the ID space - each node handles a disjoint subset of IDs (distinguished by its machine bits). If we need to add more capacity, we add more nodes (up to the limit of machine ID space) and each new node takes up its slice of the ID space. This is linear scalability with respect to number of nodes. We also partition by region: each region's cluster uses its region ID space. That way, adding a new region (if the service expands to new datacenters) is also just an extension of the partition scheme. For the segment DB approach (if we had used it), scaling might involve sharding the database (e.g., having separate counters for different "business tags" or using multiple sequence tables). But since we're favoring no central DB, we avoid that complexity.

• Load Balancing: Ensuring even distribution of requests prevents any single node from becoming a bottleneck. The load balancer (or a smart client library if we had one) should distribute calls roughly evenly among the healthy nodes in a region. Since all nodes do the same work and are stateless, a simple round-robin or least-connections strategy works. If one node starts lagging (maybe local issue), the LB should mark it unhealthy and bypass it. We'll also use health checks on each node (a simple ping or test ID generation) to detect failures quickly. Because request processing is lightweight, typically CPU is the main factor - we might monitor CPU usage on nodes and adjust the number of nodes if CPU stays high. If one region has disproportionate load, we can either overprovision nodes there or even route some requests to a less loaded region as a fallback (with a slight latency cost). The system is flexible: any node can theoretically serve any request, so geographic affinity is only for performance, not correctness.

• Caching and Prefetching: For our chosen design, there isn't much to cache - generation is computation, not a data fetch. One area of "prefetch" is in the segment allocation strategy: nodes would prefetch the next ID block before the current one runs out, to hide the DB latency. Since we aren't primarily using that approach, this isn't needed. However, one analogous idea: if a client knows it will need many IDs, it could request a batch (say 100 IDs in one call) and cache them client-side. This reduces calls per ID. Our service can support a batch API easily. Batching 100 IDs still only takes a millisecond or so to generate on a node (just loop or use vectorized operations), and reduces network roundtrips for the client by 100x. This is an optional performance optimization at the application level.

• Sharded ID Ranges (if using DB): In a scenario where a centralized DB counter was unavoidable, one method is to shard by key-space. For example, odd IDs could be generated by one DB and even IDs by another - doubling throughput without collisions. Or more generally, mod N sharding, where each of N sequences feeds a portion of IDs (like ID = sequence_value * N + shard_index). However, combining results from such sharded sequences loses global sorting (an ID from shard 0 might be higher than one from shard 1 even if created earlier, depending on the multiplier). Since we have Snowflake, we don't need this, but it's worth noting as a theoretical scaling method. Our design avoids multi-master DB issues entirely by not using the DB for generation in the critical path.

• Separating the heaviest clients: If some services use far more IDs than others, another approach (within a segment strategy) is to have separate sequences for separate contexts. For instance, one could maintain different counters for "user IDs", "order IDs", etc., each scaling independently. But again, with Snowflake/UUID, we don't need separate counters - one unified service can generate for all contexts without risk of collision. If needed, we could prefix IDs by type, but that's not required since all IDs are globally unique anyway. Instead, we might rely on the client to know what an ID refers to (or use an opaque prefix like the Cronofy example where they prefix "user_" or "order_" to an ID to indicate type - that's an application-layer detail outside our core system).

• Coordinator capacity: ZooKeeper itself can handle 1000+ nodes easily, as it's only storing small data. But for safety, we distribute ZK's ensemble nodes across different racks/zones. If we reached tens of thousands of generator nodes (which as discussed is unlikely), ZK might become a bottleneck (in maintaining that many ephemeral nodes and heartbeats). In such a scenario, we could partition machine ID assignment by region (e.g., each region runs its own small coordinator for its machines, since regions are already disjoint in ID space). This localizes the coordination traffic. The global uniqueness still holds because region IDs differ. So effectively each region's ZK manages up to, say, 64 machines (for 6-bit machine id example) - trivial for ZK. This way we wouldn't have one giant ZK cluster for all 1000+ nodes, but multiple smaller ones.

• Shadowing and Redundancy: We can run extra standby nodes that aren't actively serving but can quickly take over if load increases or a node fails. However, given the ease of horizontal scaling, it might be simpler to just always run N+1 nodes beyond expected capacity. The coordinator doesn't mind some unused IDs; they're just not assigned.

• Avoiding Single Points of Failure (SPOFs): We have addressed many SPOFs:

  • No single database in the critical path (Snowflake eliminates that).
  • The coordinator (ZK) is replicated and not needed per request, so it's not a runtime SPOF.
  • The load balancer can be redundant (multiple HA proxy instances or cloud LB service which is inherently redundant).
  • Each region is independent, so a failure in one region doesn't bring down others.
  • If an entire region is down, the system can still serve from other regions (global availability, though clients might have higher latency).
  • We should ensure the ID epoch (start time) is consistent across all nodes - this is a configuration setting. If one node had a different epoch, its IDs could clash or be out of order. This is just a deployment detail (use same config everywhere, and perhaps include epoch in the code). That's a "consistency of configuration" SPOF to watch out for - solved by automation and verification in deployment.

• Splitting the sequence inside one node: As mentioned, if one node has multiple threads frequently contending on the sequence counter, we could assign half the sequence range to one thread, half to another, to reduce lock contention. This effectively treats each thread as a sub-generator (with implicit extra bit for thread ID). However, given our throughput target and that languages like Java can atomically increment an AtomicInteger millions of times per second, this complexity likely isn't necessary. If profiling shows lock contention at some extreme load, we can revisit this optimization.

• Opaque IDs need none of this: For random IDs, no sharding needed because collision probability is already negligible. Each node works independently. If we were extremely paranoid, we could incorporate node ID into the random (e.g., generate 120 random bits and then append 8 bits of machine ID). This would guarantee no two machines ever output the same value (because their last 8 bits differ) and still leaves 120 random bits - practically the same collision probability. But this isn't standard UUID and not necessary. Standard UUIDv1/v2 include MAC address for uniqueness; in v4 it's purely random. We trust the randomness, but adding machine ID would do no harm except making the IDs slightly less opaque (someone could figure out which machine from those bits if they knew how we structured it). For simplicity, we'll stick to standard random.

• Latency considerations: The latency target from Step 2 is easy to meet for what is basically a local computation plus a network hop. In a single region, we expect ~1-5 ms responses. The 99th percentile might rise if there is a pause (e.g., if a node hits a sequence rollover and waits a millisecond, or if a GC pause in a managed language stalls a thread briefly). Step 2 treats a p99 above 10ms as a fault, so a recurring pause is something to fix rather than accept. Cross-region failovers might approach 100+ ms due to network distance, but those would be rare. We will monitor latency and ensure garbage collection or other system overhead doesn't degrade it (possibly by using languages or settings that minimize stop-the-world pauses, since any pause halts ID generation temporarily). The design inherently is real-time; we're not doing heavy I/O per request, so meeting latency is mostly about keeping the system healthy under load.

• Sampling the logs: At high QPS, logging every request is impossible. We will sample logs or aggregate counts. For monitoring throughput, each node can maintain a counter of IDs generated and perhaps export metrics (like IDs/sec, sequence utilization, etc.). This helps ensure we know if we approach any limits (for example, if we start seeing sequence values consistently hitting 4090+ in some milliseconds, it indicates that node is at capacity and we should add nodes). Similarly, monitoring the drift of clocks could be done by comparing timestamps in IDs from different nodes occasionally (but that's advanced; more simply NTP monitoring and alerts on any big adjustments suffice).

• Trade-offs and Choices Recap: Our chosen design trades off a bit of complexity in coordination and clock management for a huge gain in performance and scalability. We deliberately do not provide absolute global ordering of IDs - strict ordering would require funneling through a single sequence or a global consensus each time, which is incompatible with the throughput needs. Instead, we get approximate ordering (by timestamp) which is sufficient for most use-cases (e.g., sorting by ID gives you almost-sorted by creation time, with rare inversions if clocks skew by a bit). We also trade the theoretical possibility of UUID collisions for the practicality of no central coordination - which is a good trade given the probabilities involved. By offering two types of IDs, we give flexibility: Snowflake IDs for when sorting or smaller size is needed, and UUIDs for when non-guessability is paramount. The cost is maintaining two code paths, but that's minor. Each ID type has its domain of optimal use, and using the wrong one can have downsides (e.g., using Snowflake IDs in URLs where someone could guess an ID and fetch data might be a security risk, or using UUIDs as DB keys can bloat indexes). The system design covers both, so the architects of various services can choose accordingly.

The shape to remember is that this service scales by addition and nothing else. There is no data to rebalance, no cache to warm, and no leader to elect. Adding a node means asking the coordinator for a free machine number, and that is the whole operation.