Unique ID Generator: Capacity Estimation

On This Page

Step 3: Back-of-the-Envelope Capacity Estimation

Before designing, let's estimate the load and storage implications:

Capacity estimation

Request Volume: Assume ~100k ID generation requests per second as a baseline, which is 8.6×10^9 IDs per day (100,000 * 86,400). If we plan for peaks of 200-300k/sec, that's on the order of 1.7×10^10 to 2.6×10^10 IDs/day. Over a year, this could be trillions of IDs. The system must handle this volume continuously. Each ID itself is small, but the sheer count is large.
ID Size and Format: A time-sortable ID will be a 64-bit number (8 bytes). In decimal string form, it might be up to 19-20 digits. An opaque ID we might implement as a 128-bit UUID (16 bytes), typically represented as a 36-character string (including hyphens). These sizes are modest per ID. For example, 64 bits can uniquely represent up to 9.22×10^18 values, which is plenty for the foreseeable number of IDs (even 10^10 per day for 10 years is 3.65×10^12 total). A 128-bit space (~3.4×10^38 possibilities) is astronomically large, effectively ensuring uniqueness.
Data Storage: The ID generation service does not need to permanently store each ID it generates - it only computes and returns it. We avoid any requirement to log every ID (which would be massive data). The only persistent data might be small metadata: for example, if using a database in segment mode (discussed later), we'd store the current max ID for each segment. Alternately, if using a coordinator for machine IDs, it might store assigned IDs. These are tiny (bytes or a few integers). Thus, storage costs are minimal. We trade off storage for compute; that is acceptable because generating an ID takes almost no work.
Memory and State: Each generator instance might hold some in-memory state, like the last timestamp and sequence number it used, and its assigned machine/region ID. This is on the order of a few bytes per instance. Even with 1000 instances, that's trivial memory. No large in-memory data structures are required, since we're not caching large datasets - just computing on the fly.
Network Throughput: At 100k IDs/sec, if each response is, say, ~50 bytes (ID plus protocol overhead), that's ~5 MB/sec outbound from the service cluster - easily handled by modern network interfaces. Even 500k/sec would be ~25 MB/sec, which is high but can be distributed across nodes and regions. Internally, a few coordination messages (heartbeats to a coordinator or DB queries) will be much lower volume than the ID traffic.
Read/Write Ratio: The workload is overwhelmingly write-like (generating new IDs). There is almost no concept of "reading" existing IDs from storage - an ID is generated and delivered, not stored for lookup. We might occasionally validate an ID format (e.g., if a client asks "is this a valid ID string?") but that's just computation, not reading from a database. If using a DB for segments, those involve writes (incrementing counters). Overall, it's a write-heavy, insert-only scenario. This simplifies consistency concerns: we don't have to manage complex read queries or multi-row transactions, just ensure each generation action is unique and fast.

Next: Step 4, where the API is specified.

Mark as Completed

On This Page

Step 3: Back-of-the-Envelope Capacity Estimation