Glossary
What is Inference?
A trained AI model producing predictions or responses on new inputs. It is the stage at which the model runs in real use after training.
Inference is the stage in which a ready model produces a response to the input it receives from a user. Chatting with an LLM is, in fact, running inference. It differs from training in that the model’s weights do not change.
The prominent metrics in inference are response latency, the number of requests that can be served at the same time, and cost per request. Model size, the numerical precision used and techniques such as batching affect these metrics. It requires less computation than training, but it must run continuously and at scale.
Serving a model in your own environment requires sufficient GPU memory. Fine-tuning can be used to make the model behave in an organization-specific way, and RAG for up-to-date information. As load grows, load balancing also comes into play.