Understanding TPM, RPM, TPS, and TTFT in LLM APIs

When working with Large Language Model (LLM) APIs, several metrics are commonly used to describe rate limits, throughput, and latency.

Understanding these metrics is essential for optimizing API performance, managing costs, and building scalable LLM applications, especially AI agents and large-scale inference systems.

1. Key Metrics at a Glance

These metrics describe different aspects of LLM API performance.

• TPM and RPM primarily measure API rate limits.

• TPS measures token processing or generation throughput.

• TTFT measures how quickly a model starts responding.

2. TPM — Tokens Per Minute

TPM represents the maximum number of tokens that can be processed within one minute.

For example, suppose an API provider sets:

• TPM = 100,000

• Average tokens per request = 2,000

The theoretical number of requests allowed by the token limit is:

$$\frac{100,000}{2,000}=50\text{ requests/min} $$

Token consumption generally includes both input and output tokens:

$$T_{\text{total}}=T_{\text{input}}+T_{\text{output}} $$

However, the exact accounting method varies across API providers. Some estimate token consumption based on the requested maximum output length rather than waiting for actual generation.

3. RPM — Requests Per Minute

RPM specifies the maximum number of API requests that can be submitted within one minute.

For example:

• RPM = 500

• TPM = 100,000

Even if your requests contain very few tokens, you cannot exceed the RPM limit.

Therefore, the effective request throughput is constrained by both TPM and RPM:

$$
R_{\text{effective}}
\leq
\min\left(
\text{RPM},
\frac{\text{TPM}}{\bar T_{\text{request}}}
\right)
$$

where $\bar T_{\text{request}}$ represents the average tokens consumed per request.

This is a theoretical rate-limit bound, assuming relatively stable request sizes.

4. TPS — Tokens Per Second

TPS measures how many tokens are processed or generated per second.

In LLM inference benchmarks, TPS commonly refers to output token generation speed.

For example, if an LLM generates 1,000 output tokens in 20 seconds:

$$
\text{TPS}=\frac{1,000}{20}=50
$$

The model achieves an average output generation speed of 50 tokens per second.

Importantly, TPS can describe either:

• Per-request TPS: Token generation speed for an individual request.

• Aggregate TPS: Total token throughput across concurrent requests.

These should not be confused.

A high aggregate TPS does not necessarily imply fast generation for individual requests.

5. TTFT — Time to First Token

TTFT measures the time between submitting a request and receiving the first generated output token.

It is particularly important for interactive applications such as chatbots and AI assistants.

For example:

• Request submitted at 0 ms.

• First output token received at 400 ms.

Then:

$$
\text{TTFT}=400\text{ ms}
$$

TTFT is influenced by several factors, including network latency, request queuing, input length, and model prefill computation.

Lower TTFT generally means the user sees the model begin responding sooner.

However, low TTFT does not necessarily imply high TPS. A model may start responding quickly but generate subsequent tokens slowly.

6. Why These Metrics Matter for LLM Agents

For agentic systems, API performance affects not only response speed but also the number of tasks that can be completed within a given time budget.

Consider an agent system where:

• Each trajectory consumes 20,000 tokens.

• The API provides 1,000,000 TPM.

• Each trajectory requires multiple sequential LLM calls.

The theoretical token-budget throughput is:

$$
\frac{1,000,000}{20,000}
=50\text{ trajectories/min}
$$

However, actual throughput may be lower because of:

• RPM limits.

• Request concurrency restrictions.

• Per-request token generation speed.

• TTFT and network latency.

• Sequential dependencies between agent actions.

• Tool execution time.

For example, even with a generous TPM limit, an agent that requires many sequential reasoning steps may be bottlenecked by inference latency rather than token quotas.

This distinction is especially important in large-scale agent evaluation, reinforcement learning rollouts, and test-time compute optimization.

7. Conclusion

Although TPM, RPM, TPS, and TTFT are closely related, they measure different aspects of LLM API performance.

TPM and RPM determine how much traffic an API can accept, while TPS and TTFT help characterize how quickly it responds.

When designing scalable LLM systems, these metrics should be considered together rather than optimized independently.