Time to first token (TTFT) is the time between sending a request to a large language model and receiving the first token of its answer. You measure it in milliseconds or seconds. It is the silent gap before streaming starts, and it is the biggest single factor in how responsive an AI product feels.
A token is a small chunk of text, usually a word or part of a word. TTFT covers everything that happens before that first chunk reaches you: network travel, queueing, reading your prompt, and producing one token. A model can stream 100 tokens per second and still feel slow if the first one takes four seconds.
This guide explains why TTFT matters, what creates the delay, and how to measure it with a short streaming script. It also separates TTFT from TPOT, ITL, tokens per second and total latency, then covers how to lower it. Geodd runs serverless and dedicated inference behind an OpenAI-compatible API, so TTFT is a metric worth testing on your own prompts before you pick a provider.
Why time to first token matters
TTFT matters because it decides how long a user stares at a blank screen. Once text starts streaming, people read at roughly 200 to 300 words per minute, which is slower than most models generate. After the first token lands, the wait is mostly over in the user's mind.
It sets perceived responsiveness
With streaming on, the user judges your app by one moment: when text first appears. Total generation time matters less, because reading happens while generation continues. A response that starts in 0.4 seconds and finishes in 8 feels faster than one that starts in 3 seconds and finishes in 5.
Users judge an LLM app by when the first word appears, not when the last one does.
Where TTFT matters most
Some workloads punish a slow first token more than others:
- Chat assistants. A visible pause reads as a broken app.
- Voice agents. TTFT adds directly to the silence before the agent speaks.
- Coding assistants. Inline suggestions are useless if they arrive after you have moved on.
- Agent loops. Each sequential model call pays TTFT again. Ten calls at 0.5 seconds each add 5 seconds before any generation time.
Batch jobs such as bulk summarization care more about throughput. As a rule of thumb, interactive products should aim for a first token in about a second or less. That is a practical target, not a formal standard.
What happens before the first token appears
TTFT is the sum of several stages, and only some of them are under your control. Knowing the stages tells you where to look when the number is bad.
The stages inside TTFT
- Network. Your request travels to the provider and the first chunk travels back.
- Queueing. The request waits for free capacity on the serving cluster.
- Tokenization. Your text is converted into tokens.
- Prefill. The model processes the entire prompt in one pass and builds its attention cache. This is usually the largest share.
- First decode step. The model samples the first output token.
After that comes the steady stream of later tokens, which TTFT does not cover.
What makes it grow
Prompt length is the main lever. Prefill work grows with the number of input tokens, so a 200K-token document takes far longer to start than a 500-token question, even on the same model. Load is the second. A busy shared endpoint queues requests, and cold capacity on serverless systems adds delay.
Reasoning models need a caveat. Many emit thinking tokens before the visible answer. Decide whether your TTFT counts the first reasoning token or the first answer token, and use the same definition across every test.
Prefill cost scales with prompt length, so long prompts raise TTFT before the model writes a word.
How to measure time to first token
To measure TTFT, send a streaming request, start a timer, and stop it when the first chunk with actual content arrives. A non-streaming request cannot show TTFT, because it only returns after the full answer is done.
A streaming script you can copy
This script works with any OpenAI-compatible endpoint. Replace the base URL with the one from your provider's docs. The model ID below is Geodd's GLM 5.3 Flash.
import time, statistics
from openai import OpenAI
client = OpenAI(base_url="https://YOUR_PROVIDER_BASE_URL/v1", api_key="YOUR_API_KEY")
def measure(prompt, model):
start = time.perf_counter()
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=200,
stream=True,
)
ttft = None
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
if ttft is None:
ttft = time.perf_counter() - start
return ttft, time.perf_counter() - start
runs = [measure("Explain TCP slow start in two sentences.", "zai-org/glm-5.3-flash") for _ in range(30)]
ttfts = sorted(r[0] for r in runs if r[0])
print("p50:", statistics.median(ttfts), "p95:", ttfts[int(len(ttfts) * 0.95) - 1])
The script skips the opening chunk, which often carries only a role and no text. If your model streams reasoning in a separate field, adjust the check to match your definition.
Benchmarking without fooling yourself
A useful time to first token benchmark controls the variables that move the number:
- Run at least 30 requests and report the median (p50) and the 95th percentile (p95). Averages hide the slow tail that users remember.
- Keep prompt length fixed per test, and repeat at short, medium and long sizes.
- Test from the region your users are in, not from your laptop on another continent.
- Discard the first warm-up call, then test again at a busier time of day.
Judge TTFT by p95 at your real prompt length, because the average hides the waits users notice.
TTFT vs TPOT, ITL, tokens per second, and latency
TTFT measures the wait for the first token. The other metrics describe what happens afterward or in total. They answer different questions, so you need to read them together.
The metrics side by side
| Metric | What it measures | Typical unit | Tells you about |
|---|---|---|---|
| TTFT | Request sent to first token received | seconds or ms | How fast the response starts |
| TPOT | Average time per output token after the first | ms per token | How smooth the stream is |
| ITL | Gap between two consecutive tokens | ms | Stutters and jitter |
| Tokens per second | Output rate, roughly 1 / TPOT | tokens/s | Streaming speed |
| End-to-end latency | Request sent to final token | seconds | Total task time |
On TPOT versus ITL: TPOT is one average over a whole response, while ITL looks at each individual gap. Some tools use the two terms loosely, so check the definition before comparing numbers. ITL percentiles are the better way to spot mid-stream stalls.
A worked example
End-to-end latency is approximately TTFT plus (output tokens minus 1) times TPOT. Take a 300-token answer from two models:
- Model A: TTFT 0.4 s, TPOT 25 ms (40 tokens/s). Latency = 0.4 + 299 × 0.025 = 7.9 s.
- Model B: TTFT 1.5 s, TPOT 12 ms (about 83 tokens/s). Latency = 1.5 + 299 × 0.012 = 5.1 s.
Model B finishes sooner, but Model A feels better in a chat window because text appears after 0.4 seconds. For a background job, Model B wins. The right metric depends on whether a human is watching.
Tokens per second tells you how fast a response flows, and TTFT tells you when it starts.
How to lower time to first token
You can cut TTFT on both sides of the API call. Start with the prompt, because it is the part you fully control, then look at where and how the model is served.
Changes you can make today
- Trim the prompt. Remove unused instructions, and retrieve only the passages you need instead of pasting whole documents.
- Put static content first. If your provider supports prefix caching, a stable system prompt at the start lets repeated requests skip part of the prefill.
- Stream every interactive response. It does not shorten TTFT, but it makes the wait visible and short.
- Limit reasoning effort where the task does not need deep thinking.
- Choose a smaller or "Flash" class model when quality allows. Smaller models prefill faster.
Questions to ask your provider
Ask for p95 TTFT at your prompt length, not a single best-case figure. Ask which region serves your traffic, since distance adds network time and EU teams may need EU-hosted inference. Geodd, for example, hosts API inference in EU data center infrastructure by default for EU customers.
Also compare deployment models. Serverless endpoints scale automatically but share capacity, so bursts can add queueing. Dedicated GPUs give single-tenant, fixed capacity that removes the sharing variable, at the price of managing utilization. Test both against your real traffic before deciding.
Ask any provider for p95 TTFT at your prompt length, and verify it with your own script.
Measure it before you choose
Time to first token is the delay before an LLM starts answering. It is made of network time, queueing, prefill and the first decode step, and it grows with prompt length and load. Report it as a median and a p95, and read it alongside TPOT, ITL and end-to-end latency, because each one answers a different question.
Published numbers rarely match your prompts, region or traffic, so the only figure that counts is the one you measure yourself. The script above takes a few minutes to run and gives you a real baseline. To try it on a current model, run it against GLM 5.3 Flash through Geodd's OpenAI-compatible API, then compare the p50 and p95 with any other provider you are considering.
