GPU and AI Infrastructure

GPU selection for LLM training and inference: key criteria

Nanda Teknoloji Engineering Team · · 8 min read

In brief

In choosing a GPU for an LLM, the deciding criterion is not raw compute power but GPU memory first; the model weights, the KV cache, and in training the optimizer state must fit in memory. Then come memory bandwidth, interconnect between GPUs, and operating cost.

When choosing a GPU for a large language model (LLM), the first question should not be “which card is the fastest?” but “how do my model and context sit in memory?” To run a model, its weights, the KV cache of the requests being processed at that moment, and working buffers must fit in GPU memory. In training and fine-tuning, gradients and the optimizer state are added to this. If memory is not enough, you have to split the model across multiple GPUs or quantize it. If memory is sufficient, the next criteria are memory bandwidth, interconnect between GPUs, software compatibility, and total operating cost. This article covers these criteria in LLM projects with a calculation method, a decision table, and common mistakes.

Define the workload first: training, fine-tuning, or inference?

GPU selection changes entirely depending on the type of workload. There are three separate scenarios, and each has a different bottleneck.

  • Training from scratch (pre-training): Requires very large data and a long time. It needs many GPUs, a fast network, and an enterprise MLOps team. It is not a realistic goal for most organizations.
  • Fine-tuning (fine-tuning): You adapt a ready-made model with your own data. Parameter-efficient methods (such as LoRA) reduce memory considerably; a few GPUs in a single server are often enough.
  • Inference (inference): You serve the trained model to users. Latency, the number of concurrent users, and cost per request come to the fore.

The vast majority of enterprise projects fall into the second and third categories. So it is more realistic to think in terms of “right-sized inference and light fine-tuning” rather than “giant clusters for training.”

Memory calculation: will the model fit?

A simple formula is enough for a first estimate:

Weight memory ≈ parameter count × bytes per parameter

Bytes per parameter depend on the number format: 4 for 32-bit, 2 for 16-bit (FP16/BF16), 1 for 8-bit, and half a byte for 4-bit. For example, a 7-billion-parameter model needs about 14 GB of weights at 16 bits; a 70-billion-parameter model needs about 140 GB in the same format. This is the lower bound for the weights; the real requirement is higher.

What is added on top:

  1. KV cache: Grows with context length and layer count for each request. Long context and many concurrent users can consume more memory than the weights.
  2. Activations and working buffers: The inference framework’s own share.
  3. Gradients and optimizer state in training: Full training requires several times the weight memory. That is why full fine-tuning needs far more memory than inference of the same model.

A practical rule: calculate the model’s weight size, leave a serious buffer for the KV cache and working margin, then measure with your real request profile. The gap between estimate and production usually comes from context length.

Decision table: which criterion comes to the fore in which need?

Workload Primary bottleneck Criterion that stands out in selection Typical approach
Small model inference (for example single-department use) Cost and latency Memory sufficiency, unit cost Single GPU, quantized model
Large model inference Model does not fit in memory Memory capacity, interconnect between GPUs Splitting across multiple GPUs
Inference with high concurrency KV cache and bandwidth Memory bandwidth, batching Continuous batching with an inference engine
Parameter-efficient fine-tuning Memory Capacity, software compatibility Single server, a few GPUs
Full fine-tuning / training from scratch Memory and communication Interconnect speed, cluster network, storage Multi-server cluster, expert operations

Why can memory bandwidth matter more than compute power?

The generation phase of LLM inference (producing text token by token) is often limited not by compute but by memory bandwidth. For each new token, a large part of the model weights is read from memory. So even if two cards have similar compute power, the one with higher memory bandwidth responds faster in inference.

By contrast, the phase in which the prompt is first processed (prefill) is compute-intensive. In applications that summarize long documents or read many context chunks with RAG, prefill time can be decisive. Knowing which of these two phases your own workload leans on is half of choosing the right card. In systems using RAG, since the context will be long, also account separately for the KV cache requirement.

Multi-GPU: when is it needed, and what does it cost?

If the model does not fit on a single GPU, there are three common splitting methods: tensor parallelism (splits the operation within a layer), pipeline parallelism (distributes layers across GPUs), and data parallelism (replicates the same model and shares the requests). Tensor parallelism requires frequent and intensive communication between GPUs; that is why high-speed interconnect inside the same server is vital. Over a standard network, this kind of communication can largely erase the gain.

So ask this: if your model fits on the GPUs in a single server, do not go to splitting across servers. Evaluate the alternatives of quantization, shorter context, or a smaller model first. A multi-node solution also brings network, scheduler, and fault-handling complexity.

Quantization and model size: the way to shrink hardware

Storing weights with 8 or 4 bits instead of 16 reduces the memory requirement noticeably and permits smaller, cheaper hardware. The price is quality risk: the effect varies with the model, the quantization method, and the task. The difference may stay small in tolerant tasks such as summarization; it may be more pronounced in code generation or sensitive classification.

The recommended method is to compare different model sizes and quantization levels with your own evaluation dataset (real questions, expected answers). In this way, you arrive at the decision of “a good-enough model and a right-sized card” instead of “a bigger card.”

Software stack and compatibility

Hardware selection is a whole with the software. Check the following:

  • Do the inference engine and training libraries you will use support the GPU architecture and driver version you choose?
  • Are the precision formats you want to use (such as BF16, FP8) natively supported on the hardware?
  • Do you need GPU sharing (partitioning, time-sharing) in the container and orchestration layer?
  • Monitoring: do GPU utilization, memory, temperature, and error metrics flow into your existing observability tools?

Software incompatibility is one of the most common reasons a card that is strong on paper ends up inefficient in practice.

Power, cooling, and data center realities

GPU servers draw far more power and generate more heat than traditional servers. If you will run them in your own data center, rack power density, cooling capacity, and cabling must be verified beforehand. In the rental model this load is with the provider, but you need to clarify capacity and supply terms in the contract. For a detailed assessment, see the What is GPU as a Service article.

Common mistakes

  • Looking only at parameter count: Sizing that does not account for the KV cache and context length ends in insufficient memory in production.
  • Measuring with a single user: A system that looks fast in a pilot collapses under concurrent load.
  • Buying training hardware for inference: The different profile of the inference workload may make more economical options possible.
  • Skipping the network: Thinking about interconnect speed afterward in a multi-GPU plan.
  • Assuming the utilization rate: Cost keeps running while the GPU sits idle; the real utilization rate changes the decision.
  • Leaving security for last: Not including control points such as an AI Gateway in the design from the start, for inspecting data flowing to and from the model.

Buy or rent?

Ownership makes sense for workloads with high and predictable utilization; however, you also take on power, cooling, spare parts, drivers, and the refresh cycle. Renting provides flexibility in the trial phase, with variable load, or while the hardware generation is changing rapidly. Many organizations choose a hybrid path: they use rented capacity for initial trials and peaks, and separately planned capacity for the continuous base load. In evaluating the options, the in-house private LLM article also offers guidance for considering together the scope of the GPU as a Service model and in-house operation requirements.

Decision summary

  1. Clarify whether it is training, fine-tuning, or inference.
  2. Calculate the memory requirement from model size and context length; leave a buffer.
  3. Evaluate memory bandwidth and batching according to your latency and concurrency target.
  4. Where possible, stay on a single server; choose a multi-node structure only if necessary.
  5. Compare model size and quantization with your own data.
  6. Include power, cooling, software compatibility, and security in the plan from day one.
  7. Make the buy-or-rent decision on the basis of utilization rate.

Moving forward with Nanda Teknoloji

Nanda Teknoloji carries out the discovery, design, migration, and operation steps in projects like these: classifying the workload, sizing, setting up the environment, and continuous monitoring. If your need is for trials or variable capacity, the GPU as a Service solution is a good starting point; as load and usage become clear, the architecture can be re-evaluated together.

Frequently asked questions

How many GB of GPU memory does LLM inference need?

As a first approximation, multiply the parameter count by the byte size of one parameter; 16-bit weights need 2 bytes per parameter. The KV cache and a working margin are added on top, so the number of concurrent users determines the total.

Can training and inference use the same GPU?

Technically yes, but the needs differ. Training is more sensitive to memory capacity and interconnect between GPUs, while inference is more sensitive to latency, memory bandwidth, and cost efficiency.

Does quantization reduce quality?

When weights are stored with fewer bits, the memory requirement drops and in most cases the quality loss stays limited; however, the effect varies with the model and the task. Do not put it into production without measuring with your own data.

Is it more correct to buy or rent GPUs?

Ownership makes sense if the workload is predictable and continuous; renting makes sense if it is variable or in the trial phase. Utilization rate, operating expertise, and the hardware refresh cycle together determine the decision.

Related solutions

Related posts

Let's plan your critical systems together

Tell us what you need; our team will work out the right architecture and migration path with you.