Inference latency describes how long a model request takes to respond. Time to first token measures how quickly streaming output begins, while end-to-end latency measures the time until the full response is complete. These numbers answer different user-experience questions and should not be treated as interchangeable.
For a chat interface, time to first token often matters most because users see text start to appear. For a background job that produces a report, total time and cost matter more. An AI agent that makes several model and tool calls in a row adds each delay together, so a fast single call can still produce a slow task.
Measure latency at realistic prompt lengths and traffic levels, and report percentiles such as p50 and p95 rather than only averages. Network time, queueing and tool calls may also contribute to the delay users experience. Related: LLM Inference, Tokens per Second.