GPU selection for LLM training and inference: key criteria
Nanda Teknoloji Engineering Team · · 8 min read
In brief
In choosing a GPU for an LLM, the deciding criterion is not raw compute power but GPU memory first; the model weights, the KV cache, and in training the optimizer state must fit in memory. Then come memory bandwidth, interconnect between GPUs, and operating cost.
When choosing a GPU for a large language model (LLM), the first question should not be “which card is the fastest?” but “how do my model and context sit in memory?” To run a model, its weights, the KV cache of the requests being processed at that moment, and working buffers must fit in GPU memory. In training and fine-tuning, gradients and the optimizer state are added to this. If memory is not enough, you have to split the model across multiple GPUs or quantize it. If memory is sufficient, the next criteria are memory bandwidth, interconnect between GPUs, software compatibility, and total operating cost. This article covers these criteria in LLM projects with a calculation method, a decision table, and common mistakes.
Define the workload first: training, fine-tuning, or inference?
GPU selection changes entirely depending on the type of workload. There are three separate scenarios, and each has a different bottleneck.
- Training from scratch (pre-training): Requires very large data and a long time. It needs many GPUs, a fast network, and an enterprise MLOps team. It is not a realistic goal for most organizations.
- Fine-tuning (fine-tuning): You adapt a ready-made model with your own data. Parameter-efficient methods (such as LoRA) reduce memory considerably; a few GPUs in a single server are often enough.
- Inference (inference): You serve the trained model to users. Latency, the number of concurrent users, and cost per request come to the fore.
The vast majority of enterprise projects fall into the second and third categories. So it is more realistic to think in terms of “right-sized inference and light fine-tuning” rather than “giant clusters for training.”
Memory calculation: will the model fit?
A simple formula is enough for a first estimate:
Weight memory ≈ parameter count × bytes per parameter
Bytes per parameter depend on the number format: 4 for 32-bit, 2 for 16-bit (FP16/BF16), 1 for 8-bit, and half a byte for 4-bit. For example, a 7-billion-parameter model needs about 14 GB of weights at 16 bits; a 70-billion-parameter model needs about 140 GB in the same format. This is the lower bound for the weights; the real requirement is higher.
What is added on top:
- KV cache: Grows with context length and layer count for each request. Long context and many concurrent users can consume more memory than the weights.
- Activations and working buffers: The inference framework’s own share.
- Gradients and optimizer state in training: Full training requires several times the weight memory. That is why full fine-tuning needs far more memory than inference of the same model.
A practical rule: calculate the model’s weight size, leave a serious buffer for the KV cache and working margin, then measure with your real request profile. The gap between estimate and production usually comes from context length.
Decision table: which criterion comes to the fore in which need?
| Workload | Primary bottleneck | Criterion that stands out in selection | Typical approach |
|---|---|---|---|
| Small model inference (for example single-department use) | Cost and latency | Memory sufficiency, unit cost | Single GPU, quantized model |
| Large model inference | Model does not fit in memory | Memory capacity, interconnect between GPUs | Splitting across multiple GPUs |
| Inference with high concurrency | KV cache and bandwidth | Memory bandwidth, batching | Continuous batching with an inference engine |
| Parameter-efficient fine-tuning | Memory | Capacity, software compatibility | Single server, a few GPUs |
| Full fine-tuning / training from scratch | Memory and communication | Interconnect speed, cluster network, storage | Multi-server cluster, expert operations |
Why can memory bandwidth matter more than compute power?
The generation phase of LLM inference (producing text token by token) is often limited not by compute but by memory bandwidth. For each new token, a large part of the model weights is read from memory. So even if two cards have similar compute power, the one with higher memory bandwidth responds faster in inference.
By contrast, the phase in which the prompt is first processed (prefill) is compute-intensive. In applications that summarize long documents or read many context chunks with RAG, prefill time can be decisive. Knowing which of these two phases your own workload leans on is half of choosing the right card. In systems using RAG, since the context will be long, also account separately for the KV cache requirement.
Multi-GPU: when is it needed, and what does it cost?
If the model does not fit on a single GPU, there are three common splitting methods: tensor parallelism (splits the operation within a layer), pipeline parallelism (distributes layers across GPUs), and data parallelism (replicates the same model and shares the requests). Tensor parallelism requires frequent and intensive communication between GPUs; that is why high-speed interconnect inside the same server is vital. Over a standard network, this kind of communication can largely erase the gain.
So ask this: if your model fits on the GPUs in a single server, do not go to splitting across servers. Evaluate the alternatives of quantization, shorter context, or a smaller model first. A multi-node solution also brings network, scheduler, and fault-handling complexity.
Quantization and model size: the way to shrink hardware
Storing weights with 8 or 4 bits instead of 16 reduces the memory requirement noticeably and permits smaller, cheaper hardware. The price is quality risk: the effect varies with the model, the quantization method, and the task. The difference may stay small in tolerant tasks such as summarization; it may be more pronounced in code generation or sensitive classification.
The recommended method is to compare different model sizes and quantization levels with your own evaluation dataset (real questions, expected answers). In this way, you arrive at the decision of “a good-enough model and a right-sized card” instead of “a bigger card.”
Software stack and compatibility
Hardware selection is a whole with the software. Check the following:
- Do the inference engine and training libraries you will use support the GPU architecture and driver version you choose?
- Are the precision formats you want to use (such as BF16, FP8) natively supported on the hardware?
- Do you need GPU sharing (partitioning, time-sharing) in the container and orchestration layer?
- Monitoring: do GPU utilization, memory, temperature, and error metrics flow into your existing observability tools?
Software incompatibility is one of the most common reasons a card that is strong on paper ends up inefficient in practice.
Power, cooling, and data center realities
GPU servers draw far more power and generate more heat than traditional servers. If you will run them in your own data center, rack power density, cooling capacity, and cabling must be verified beforehand. In the rental model this load is with the provider, but you need to clarify capacity and supply terms in the contract. For a detailed assessment, see the What is GPU as a Service article.
Common mistakes
- Looking only at parameter count: Sizing that does not account for the KV cache and context length ends in insufficient memory in production.
- Measuring with a single user: A system that looks fast in a pilot collapses under concurrent load.
- Buying training hardware for inference: The different profile of the inference workload may make more economical options possible.
- Skipping the network: Thinking about interconnect speed afterward in a multi-GPU plan.
- Assuming the utilization rate: Cost keeps running while the GPU sits idle; the real utilization rate changes the decision.
- Leaving security for last: Not including control points such as an AI Gateway in the design from the start, for inspecting data flowing to and from the model.
Buy or rent?
Ownership makes sense for workloads with high and predictable utilization; however, you also take on power, cooling, spare parts, drivers, and the refresh cycle. Renting provides flexibility in the trial phase, with variable load, or while the hardware generation is changing rapidly. Many organizations choose a hybrid path: they use rented capacity for initial trials and peaks, and separately planned capacity for the continuous base load. In evaluating the options, the in-house private LLM article also offers guidance for considering together the scope of the GPU as a Service model and in-house operation requirements.
Decision summary
- Clarify whether it is training, fine-tuning, or inference.
- Calculate the memory requirement from model size and context length; leave a buffer.
- Evaluate memory bandwidth and batching according to your latency and concurrency target.
- Where possible, stay on a single server; choose a multi-node structure only if necessary.
- Compare model size and quantization with your own data.
- Include power, cooling, software compatibility, and security in the plan from day one.
- Make the buy-or-rent decision on the basis of utilization rate.
Moving forward with Nanda Teknoloji
Nanda Teknoloji carries out the discovery, design, migration, and operation steps in projects like these: classifying the workload, sizing, setting up the environment, and continuous monitoring. If your need is for trials or variable capacity, the GPU as a Service solution is a good starting point; as load and usage become clear, the architecture can be re-evaluated together.