Model quantization stores some model values with fewer bits than the original representation. Lower precision can make a model smaller and easier to run on limited hardware, and some runtimes can execute quantized models more efficiently. The benefit depends on the quantization method, hardware and software support.
A common example is running a model stored in lower precision on a laptop or single graphics card that could not hold the original version. Smaller files also load faster. Different methods trade quality for size in different ways, so a label such as a bit width does not tell you how the model will behave on your task. Quantization is most often used with an open-weight model.
Quantization can affect output quality, especially at aggressive settings or on sensitive tasks. Test the quantized model against representative examples and compare quality, memory consumption and latency before deploying it. Related: Small Language Model, LLM Inference.