2026-10-19
Understanding TPM, RPM, TPS, and TTFT in LLM APIs 2026-10-19 ~ 2026-10-19
When working with Large Language Model (LLM) APIs, several metrics are commonly used to describe rate limits, throughput, and latency.
Understanding these metrics is essential for optimizing API performance, managing costs, and building scalable LLM applications, especially AI agents and large-scale inference systems.
1. Key Metrics at a Glance
| Metric | Full Name | Meaning |
|---|---|---|
| TPM | Tokens Per Minute | Maximum tokens processed per minute |
| RPM | Requests Per Minute | Maximum API requests per minute |
| TPS | Tokens Per Second | Typically, token generation speed or throughput |
| TTFT | Time to First Token | Latency before the first output token arrives |
These metrics describe different aspects of LLM API performance.
2026-05-07
Cache Hits in LLMs 2026-05-07 ~ 2026-10-15
The note is generated by Gemini.
Overview
In the context of Large Language Models (LLMs), a Cache Hit occurs when the inference engine identifies that specific data (tokens, KV pairs, or full responses) from a previous request is already stored in memory.
🚀 Key Types of Cache Hits
1. KV (Key-Value) Cache Hit
- Level: Inference Engine (e.g., vLLM, TensorRT-LLM).
- Mechanism: During the “Prefill” stage, the model stores the Key and Value tensors of the attention mechanism.
- The Hit: Reuses stored tensors for identical prefixes.
- Result: Drastically reduces Time to First Token (TTFT).
2. Semantic Cache Hit
- Level: Application Layer.
- Mechanism: Uses Vector Embeddings to compare queries.
- The Hit: Returns a saved answer if the meaning is the same.
- Result: Zero GPU cost and near-zero latency.
📊 Comparison Table
| Metric | Cache Hit | Cache Miss |
|---|---|---|
| Computation | Low (Retrieval) | High (Full Forward Pass) |
| Latency | Milliseconds | Seconds |
| Cost | Negligible | Full Token Usage |