Unique ID Generator: Capacity Estimation

Step 3: Back-of-the-Envelope Capacity Estimation

Step 2 set the targets in words. Now we turn them into numbers, and see what load and storage they imply. The diagram sums up the result, and each figure is worked out below it.

Request volume. Assume about 100,000 ID requests per second as a baseline. That is 8.6x10^9 IDs per day (100,000 x 86,400). If we plan for peaks of 200,000 to 300,000 per second, that is 1.7x10^10 to 2.6x10^10 IDs per day. Over a year it reaches trillions. Each ID is small, but the count is large, and the system must handle it without pause.

ID size and format. A time-sortable ID is a 64-bit number, 8 bytes. As a decimal string it is up to 19 or 20 digits. An opaque ID is a 128-bit UUID, 16 bytes, usually shown as a 36-character string with hyphens. Both are modest per ID.

The two widths are worth comparing against the volume above. 64 bits can represent up to 9.22x10^18 values. That is plenty. Even 10^10 IDs per day for 10 years is only 3.65x10^13 in total. A 128-bit space, about 3.4x10^38 values, is so large that uniqueness is effectively assured.

Data storage. The service does not store each ID it makes. It only computes and returns it. Logging every ID would be a huge amount of data, so we do not require it.

That one choice removes the largest write workload the system could have had. A table recording every ID would take 100,000 inserts per second, and nothing would ever read it.

The only persistent data is small metadata. In segment mode (discussed later), a database holds the current max ID per segment. With a coordinator for machine IDs, it holds the assigned IDs. These are bytes or a few integers, so storage cost is minimal. We trade storage for compute, and that is fine because making an ID takes almost no work.

Memory and state. Each generator instance holds a little in-memory state: its last timestamp, its sequence number, and its assigned machine or region ID. That is a few bytes per instance. Even 1,000 instances need trivial memory. There are no large in-memory structures, since we cache no data. We compute each ID as it is asked for.

Network throughput. At 100,000 IDs per second, with about 50 bytes per response (ID plus protocol overhead), the cluster sends about 5 MB per second. A modern network interface handles that easily. Even 500,000 per second is about 25 MB per second. That is high but spread across nodes and regions. Internal coordination traffic, like heartbeats to a coordinator or database queries, is far smaller than the ID traffic.

Read/write ratio. The workload is almost entirely write-like: making new IDs. There is almost no reading of existing IDs from storage. An ID is made and delivered, not stored for lookup. We might validate an ID format if a client asks, but that is computation, not a database read. If we use a database for segments, those calls are writes that increment counters.

Overall this is a write-heavy, insert-only scenario. That simplifies consistency. There are no complex read queries and no multi-row transactions. Each generation action just has to be unique and fast.

Notice what these figures have in common. None of them is large. The scale in this system is the request rate, not the data, which makes it a compute problem rather than a storage one.

Next: Step 4, where the API is specified.