Grokking Generative AI Fundamentals
Vote

0% completed

​

Cost, Latency and Rate Limits

A support tool adds a "suggest a reply" button, and a model drafts each answer. Its cost builds up in four steps, at illustrative large-model prices:

  1. One request sends 3,000 tokens and gets back a 400-token reply.
  2. At 3 per million input tokens and 15 per million output tokens, that costs $0.015.
  3. Agents open 20,000 tickets a day, so the button costs $300 a day.
  4. A 30-day month costs 9,000, and twelve months cost 108,000.

The price of one request is easy to ignore, but the feature runs 600,000 times a month

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content