On this page

The End of the Infinite Budget

Understanding the Basics of FinOps

The Hardware Reality: Why AI Costs So Much

The Scarcity Premium

Power and Cooling

The Token Economy: Measuring Consumption

Input vs. Output Tokens

Doing the Math: Envelope Estimation

Step 1: Establish the Volume

Step 2: Estimate the Payload

Step 3: Calculate Total Tokens

Step 4: Apply Unit Costs

Step 5: Annualize

Strategies for Cost Optimization

1. Model Selection and Routing

2. Semantic Caching

3. Context Window Management

4. Quantization

The Build vs. Buy Decision

Conclusion

Why Infinite Budget Designs Fail: Mastering AI Cost Estimation

Arslan Ahmad

March 21st, 2026

Don't let hidden costs ruin your design. Explore strategies like model routing and caching to build financially viable AI systems.

On This Page

The End of the Infinite Budget

Understanding the Basics of FinOps

The Hardware Reality: Why AI Costs So Much

The Scarcity Premium

Power and Cooling

The Token Economy: Measuring Consumption

Input vs. Output Tokens

Doing the Math: Envelope Estimation

Step 1: Establish the Volume

Step 2: Estimate the Payload

Step 3: Calculate Total Tokens

Step 4: Apply Unit Costs

Step 5: Annualize

Strategies for Cost Optimization

1. Model Selection and Routing

2. Semantic Caching

3. Context Window Management

4. Quantization

The Build vs. Buy Decision

Conclusion

System design interviews have long been the gatekeeper for senior engineering roles.

For years, the grading rubric focused on three main pillars: scalability, availability, and reliability. Candidates would stand at a whiteboard and draw complex diagrams featuring massive clusters of servers, redundant databases, and sophisticated load balancers.

If the system could theoretically handle millions of requests without crashing, the design was considered a success. However, a new constraint has entered the evaluation criteria, driven by the massive adoption of Artificial Intelligence (AI). That constraint is cost.

The era of "growth at all costs" has ended. In the current economic climate, engineering efficiency is just as valuable as engineering capability.

A candidate who proposes a technically sound architecture that costs significantly more than the revenue it generates will likely fail the interview.

This is especially true for systems involving Large Language Models (LLMs) and generative AI, where operational costs can spiral out of control if not managed with precision.

The practice of integrating financial accountability into the engineering process is known as FinOps. While it might sound like a topic for the finance department, it is now a critical skill for software architects.

Understanding how to estimate, monitor, and optimize the dollar cost of a distributed system is what separates a junior developer from a staff engineer.

This guide explores the financial mechanics of modern AI systems and provides a framework for navigating these discussions in a technical interview.

The End of the Infinite Budget

In traditional web development, the cost of serving a user is relatively low and predictable.

A standard web server can handle thousands of requests per second, and text-based data storage is cheap. Engineers rarely had to worry that a sudden spike in traffic would bankrupt the company.

The primary concern was keeping the site online, not paying the bill.

AI changes this dynamic fundamentally.

The computational resources required to run a deep learning model are orders of magnitude higher than those required to run a standard web application.

Processing a single query through a modern LLM requires massive matrix calculations that lock up high-performance hardware for seconds at a time.

When a candidate designs a system that routes every single user interaction through the most powerful available model, they are designing an "infinite budget" system. They are prioritizing theoretical performance over financial viability.

Interviewers are increasingly presenting scenarios where the business margins are thin, requiring the candidate to justify every dollar of infrastructure spend.

Understanding the Basics of FinOps

FinOps stands for Financial Operations. It is a cultural practice and a set of operating procedures that brings financial awareness to the variable spending model of the cloud.

In a traditional data center, hardware was a fixed cost (Capital Expenditure or CapEx). You bought a server once and used it for five years.

In the cloud, costs are variable (Operating Expenditure or OpEx). You pay for what you use, and you pay for it only while you use it.

For a system designer, FinOps involves three key phases:

Inform: Understanding where the money is going and allocating costs to specific features.
Optimize: Finding ways to reduce waste without sacrificing performance.
Operate: Building processes that keep costs in check continuously.

In the context of an interview, you are primarily focused on the "Inform" and "Optimize" phases. You must demonstrate that you know what the cost drivers are and that you have strategies to minimize them.

The Hardware Reality: Why AI Costs So Much

To understand the cost, one must understand the hardware.

Traditional applications run on CPUs (Central Processing Units). These are general-purpose chips designed to handle a wide variety of tasks, from running an operating system to querying a database.

Cloud providers have vast fleets of CPUs, and they are relatively inexpensive to rent.

AI models, particularly those based on the Transformer architecture (like GPT), run inefficiently on CPUs.

They require GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units).

These chips are designed for massive parallel processing, allowing them to perform the billions of calculations required for a single AI inference.

The Scarcity Premium

High-performance GPUs are in short supply.

Because demand outstrips supply, cloud providers charge a significant premium for access to them.

When you design a system that relies on real-time inference, you are effectively reserving a slice of this expensive hardware.

Unlike a web server that can "sleep" when idle, loading a massive AI model into GPU memory often requires keeping the instance active 24/7 to avoid "cold start" latency.

This leads to high idle costs, which means paying for expensive hardware even when no users are online.

Power and Cooling

There is also a physical cost component. GPUs consume significantly more electricity than CPUs and generate more heat.

Data centers require advanced liquid cooling and high-voltage power infrastructure to support them. These utility costs are passed down to the consumer. When you choose a larger model, you are directly increasing the energy consumption of your system.

The Token Economy: Measuring Consumption

In traditional system design, we measure traffic in Requests Per Second (RPS) or bandwidth in Gigabytes.

In the world of generative AI, the fundamental unit of economics is the token.

A token is a chunk of text. It can be a whole word, part of a word, or a punctuation mark. A common rule of thumb is that 1,000 tokens equal approximately 750 words.

Costs for AI services are almost always quoted in "Dollars per Million Tokens."

Input vs. Output Tokens

It is crucial to distinguish between the two types of traffic:

Input Tokens: This is the text sent to the model. It includes the user's question, any system instructions, and any background data provided to help the model answer.
Output Tokens: This is the text the model generates.

In most pricing models, Output Tokens are significantly more expensive than Input Tokens.

This is because generating new text is computationally harder than reading existing text. When designing a system, you must be careful not only about what the user asks but also about how verbose the model's answer is.

A design that allows the model to ramble endlessly will drive up costs linearly.

Doing the Math: Envelope Estimation

One of the most impressive skills a candidate can show is "envelope math," the ability to perform rough cost estimates on the whiteboard.

This proves that the design is grounded in reality.

Step 1: Establish the Volume

Start with the requirements. Let's assume a hypothetical feature: a writing assistant used by 100,000 people daily.

Daily Active Users (DAU): 100,000
Queries per User: 10
Total Daily Queries: 1,000,000

Step 2: Estimate the Payload

Next, determine the size of the data.

Average Input: The user pastes a paragraph to be rewritten (200 words) plus system instructions (50 words). Total: 250 words ≈ 330 tokens.
Average Output: The model provides a rewritten version (200 words). Total: 200 words ≈ 260 tokens.

Step 3: Calculate Total Tokens

Daily Input Tokens: 1,000,000 queries × 330 tokens = 330,000,000 tokens (330 million).
Daily Output Tokens: 1,000,000 queries × 260 tokens = 260,000,000 tokens (260 million).

Step 4: Apply Unit Costs

Using hypothetical market rates (e.g., $5.00 per million input tokens,$ 15.00 per million output tokens):

Input Cost: 330 million × $5.00 \=$ 1,650 per day.
Output Cost: 260 million × $15.00 \=$ 3,900 per day.
Total Daily Cost: $5,550.

Step 5: Annualize

Monthly Cost: ~$166,500.
Annual Cost: ~$2,000,000.

By performing this calculation, you transform an abstract architectural diagram into a concrete business case.

You can now look at the interviewer and discuss whether a $2 million annual bill is acceptable for this feature. If the feature is a core revenue driver, it might be. If it is a minor add-on, the design needs optimization.

Strategies for Cost Optimization

Once the baseline cost is established, the next step is to propose specific architectural changes to reduce it.

This is where FinOps meets engineering creativity.

1. Model Selection and Routing

The most powerful lever for cost control is model selection. Not every query requires the largest, smartest model available.

A user saying "Hello" or asking for a simple keyword extraction does not need a model with hundreds of billions of parameters.

A Model Router is a design pattern where a small, lightweight model sits at the entry point. It classifies the user's intent. Simple queries are routed to a cheap, fast model.

Only complex, reasoning-heavy queries are routed to the expensive "frontier" model.

This tiered approach can reduce the blended cost per query by huge margins without degrading the user experience.

2. Semantic Caching

In traditional systems, caching is used to speed up database reads.

In AI systems, caching is used to avoid expensive generation.

Since AI generation is non-deterministic (it can produce different answers each time), standard caching doesn't always work.

Semantic Caching solves this. It uses embeddings (vector representations of text) to understand the meaning of a user's question.

If User A asks "How do I reset my password?" and User B asks "I forgot my password, how to fix?", the system recognizes these are the same request.

Instead of paying the AI to generate the answer again, the system serves the cached response from the first interaction.

This reduces the cost of that query to near zero.

3. Context Window Management

Engineers often make the mistake of stuffing the entire conversation history into the context window to ensure the AI "remembers" everything.

However, because you pay for every input token on every turn of the conversation, this results in costs that grow exponentially as the chat gets longer.

An optimized design manages the context window aggressively. It might use a summarization step to compress the chat history into a few key bullet points, or it might use a retrieval mechanism to only fetch the specific lines of history that are relevant to the current question.

This keeps the input payload small and the cost flat.

4. Quantization

If the design involves self-hosting models rather than using an API, Quantization is a critical concept.

Standard models use high-precision numbers (32-bit or 16-bit) to represent their internal weights. This requires massive amounts of GPU memory (VRAM).

Quantization reduces the precision of these numbers to 8-bit or 4-bit integers.

While this can cause a very slight drop in model accuracy, it dramatically reduces the memory footprint.

A model that previously required four GPUs might fit on just one after quantization. This cuts the hardware rental cost by 75% instantly.

The Build vs. Buy Decision

A major strategic decision in system design is whether to use a managed API (like OpenAI or Anthropic) or to build and host open-source models (like Llama or Mistral).

The API Route (Buy):

Pros: Zero infrastructure management. Instant scalability. You pay only for tokens used.
Cons: Higher unit cost at scale. Data privacy concerns. Rate limits.
FinOps View: This is ideal for startups, low-traffic applications, or unpredictable workloads. The cost is purely variable (OpEx).

The Self-Hosted Route (Build):

Pros: Lower unit cost at high volume. Full control over data. No rate limits.
Cons: High operational overhead. You must pay for GPUs even when they are idle.
FinOps View: This becomes viable only when traffic is high and consistent enough to keep the GPUs fully utilized. If the GPUs sit idle at night, the "savings" are lost.

Conclusion

The role of the system architect is evolving. It is no longer enough to connect boxes and arrows to ensure data flows correctly. The modern architect must ensure that value flows correctly.

When preparing for a System Design Interview, remember that cost is a constraint just as rigid as latency or bandwidth.

By incorporating FinOps principles into your design process, you demonstrate a level of maturity that appeals to hiring managers. You show that you understand the business implications of your technical choices.

Key Takeaways:

Treat cost as a requirement: A system that costs more than it earns is a failed system.
Understand the unit economics: Know the difference between token-based billing and time-based billing.
Calculate the impact: Use simple math to estimate the monthly bill of your design.
Optimize inputs: Reduce the amount of data sent to the model to save on input costs.
Cache intelligently: Never pay to generate the same answer twice.
Right-size the resources: Use the smallest capable model and hardware for the task.

Mastering these concepts will not only help in passing the interview but will also provide a strong foundation for building sustainable, scalable systems in the real world.

What our users say

Matzuk

Algorithms can be daunting, but they're less so with the right guide. This course - https://www.designgurus.io/course/grokking-the-coding-interview, is a great starting point. It covers typical problems you might encounter in interviews.

Simon Barker

This is what I love about http://designgurus.io’s Grokking the coding interview course. They teach patterns rather than solutions.

Eric

I've completed my first pass of "grokking the System Design Interview" and I can say this was an excellent use of money and time. I've grown as a developer and now know the secrets of how to build these really giant internet systems.

Designgurus on Substack

Deep dives, systems design teardowns, and interview tactics delivered daily.

Read on Substack

Annual Subscription

Get instant access to all current and upcoming courses for one year.

Access to 50+ courses

New content added monthly

Certificate of completion

$29.08

/month

Billed Annually

Recommended Course

Grokking Prompt Engineering for Professional Portfolio and Job Search

453+ students

4.1

Elevate your career with Grokking Prompt Engineering for Professional Portfolio and Job Search - the ultimate AI-powered guide for crafting a standout portfolio, polishing resumes and cover letters, and nailing interviews in today’s competitive job market.

View Course