Ask ten engineers how much hardware a large language model needs, and you will get ten answers, most of them starting with “it depends.” That answer is honest, but it is not useful when you are about to sign a purchase order or commit to a cloud contract.
The good news is that the dependencies are predictable. Once you understand where memory goes, why bandwidth matters more than raw speed, and how training differs from serving, you can size a system with a spreadsheet instead of a guess.
Start With Memory, Not Compute
Most people shopping for AI hardware look at compute figures first. For language models, that is usually the wrong place to start.
The first question is whether the model fits. If the weights, the working data, and the overhead do not fit in GPU memory, the job either fails or slows to a crawl as data shuttles across slower links. Compute only matters once the model is resident.
That is why GPU memory capacity, measured in gigabytes of high-bandwidth memory (HBM), tends to drive LLM hardware decisions more than any other spec.
The Simple Math of Model Weights
Model weights follow straightforward arithmetic. Each parameter takes a fixed number of bytes depending on its numeric precision.
At 16-bit precision (FP16 or BF16), each parameter uses 2 bytes. A 7-billion-parameter model needs roughly 14 GB just for its weights, and a 70-billion-parameter model needs roughly 140 GB.
Quantization shrinks that footprint. At 8-bit precision the same 70B model drops to around 70 GB, and at 4-bit it lands near 35 GB. Quality trade-offs vary by model and method, so test on your own workload before committing to an aggressive format.
Keep in mind that weights are the floor, not the total. Every running model needs extra memory on top of them.
Why Context Length Changes Everything
The biggest source of that extra memory during inference is the key-value (KV) cache. As a model processes a prompt and generates text, it stores intermediate attention data for every token so it does not have to recompute it.
The KV cache grows with context length, with the number of simultaneous requests, and with the size of the model. A model that fits comfortably with short prompts can run out of memory once users start pasting in long documents or you raise the batch size to serve more people at once.
This is one reason serving frameworks have invested heavily in memory management. The Paged Attention technique introduced with vLLM treats KV cache like virtual memory pages, cutting waste so more requests fit on the same card. Smart software can buy real headroom, but it cannot make a model fit where the math says it will not.
Training Is a Different Problem Than Inference
Serving a model and training one place very different demands on hardware, and mixing them up is one of the most expensive sizing mistakes a team can make.
Training needs to hold far more than the weights. A common rule of thumb for mixed-precision training with the Adam optimizer is about 16 bytes per parameter once you count weights, gradients, a full-precision master copy, and optimizer states. By that math, fully fine-tuning a 7B model needs over 100 GB before activations are even counted.
If your roadmap includes full fine-tuning or pretraining, it pays to look at higher-memory accelerators early, and the secondary market has made options like H200s for training larger language models far easier to budget for than they were at launch. With 141 GB of HBM3e per GPU, versus 80 GB on the H100, each card holds more of the model and its training state, which means fewer GPUs to split the job across.
Parameter-efficient methods change the equation for smaller teams. Techniques such as LoRA train a small set of added weights while the base model stays frozen. The QLoRA paper showed a 65B-parameter model being fine-tuned on a single 48 GB GPU by combining this approach with 4-bit quantization.
When One GPU Stops Being Enough
Eventually most serious workloads outgrow a single card. At that point the model, the data, or both get split across several GPUs.
Tensor parallelism splits individual layers across GPUs. Pipeline parallelism assigns different layers to different GPUs. Data parallelism runs copies of the model on separate batches and syncs the results.
Each approach adds communication overhead. The more GPUs that need to talk to each other, the more the speed of the connections between them starts to matter.
Bandwidth Is the Hidden Bottleneck
For text generation, memory bandwidth often matters more than peak compute. Each new token requires reading a large share of the model’s weights from memory, so the speed of that memory sets a ceiling on how fast tokens come out.
This is why newer accelerators advertise bandwidth so prominently. The H200, for example, is rated at 4.8 TB/s of memory bandwidth, a meaningful jump over the H100’s 3.35 TB/s.
Between GPUs, the same logic applies. High-speed links like NVLink inside a server, and InfiniBand or fast Ethernet between servers, keep multi-GPU jobs from stalling while cards wait on each other. A cluster with fast GPUs and slow interconnects will underperform its spec sheet.
Power, Cooling, and the Rest of the Server
Data center GPUs in this class can each draw up to around 700 watts. An eight-GPU server can pull on the order of 10 kilowatts at full load, which is more than many standard office racks or older colocation contracts were designed to handle.
Before buying, confirm that your facility can deliver the power and remove the heat. Some newer configurations call for liquid cooling, which is a facilities project, not just a hardware purchase.
The supporting components matter too. Fast NVMe storage keeps training data flowing, enough system RAM prevents bottlenecks when loading checkpoints, and a capable CPU keeps data pipelines from starving the GPUs.
A Practical Way to Size Your Hardware
Start with the largest model you realistically plan to run in the next year or two, not the one you are testing today. Calculate its weight footprint at the precision you intend to use.
Next, add headroom for the KV cache based on your expected context length and concurrent users. For training, apply the per-parameter rule of thumb, or plan around parameter-efficient methods if full fine-tuning is out of reach.
Then decide how many GPUs that total implies, and check whether your interconnect, power, and cooling can support them. Run a pilot on rented capacity if you can, because real workloads rarely match back-of-envelope estimates exactly.
Hardware for LLMs is expensive, but it is not mysterious. Get the memory math right first, and most of the other decisions fall into place.
I’m Rajesh Kumar, a DevOps, SRE, DevSecOps, Cloud, and Platform Engineering expert passionate about sharing practical knowledge, real-world experiences, and industry best practices. I have worked at Cotocus and regularly write about technology, travel, investing, health, product reviews, and digital marketing through my various platforms.
I publish technical articles at DevOps School, travel stories at Holiday Landmark, stock market insights at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow, and SEO and digital marketing strategies at Wizbrand.
Find Trusted Cardiac Hospitals
Compare heart hospitals by city and services — all in one place.
Explore Hospitals