LLM performance and cost¶
All topics · Alphabetical index
Concepts¶
- Input token cost; output token cost; reasoning budgets
- Prompt caching; context reuse; reusable prefixes
- Response caching; semantic caching; cache invalidation; tenant isolation
- Model routing; small-model first pass; escalation
- Batching; parallelization; streaming
- Latency budgets; retrieval latency; tool latency
- Provider quotas; backpressure; admission control; retry amplification
- Cost per completed task; quality-adjusted cost; budget enforcement
- Graceful degradation; cancellation; unnecessary tool calls
- Prefill; decoding; KV cache; continuous batching; quantization
Further reading¶
Reference links for this topic.