Prompt Caching and Cache Friendly Loop Design

Two Desk loops make the same 10 calls to the same model, send the same 141,000 tokens, and have caching turned on. One costs 39,630 units of input. The other costs 176,250 units, which is more than a loop with caching turned off.

The difference is one harmless-looking line of code. The second loop sorts its tools by how recently each was used, so the model sees the most useful tools first. That line is expensive because of how a provider decides that text can be reused.