LLM inference is the stage where a trained model is used, rather than trained. The application sends input tokens and settings to the model, which computes and generates output tokens. Inference may run through a hosted API or on hardware controlled by the developer.
When you call a hosted API, inference runs on the provider's servers. When you run an open-weight model yourself, you provide the hardware and also pay for idle time. Inference cost often becomes the main running cost of an AI feature, so estimate it with real prompts and expected daily volume before launch. Techniques such as model routing and semantic caching can lower it.
Inference performance depends on model size, hardware, input length, output length, batching and serving configuration. For production systems, track both latency and cost under realistic workloads. Related: Inference Latency, Tokens per Second.