Best GPU for running LLMs at home, at each budget tier

How this list is made

Two numbers decide what a GPU can do with a language model at home:

1. Memory decides what runs. The model's weights, its KV cache and the software's buffers have to fit. If they do not, the model either does not load or runs partly from system RAM, which is much slower. The VRAM guide has the figure for each model.
2. Memory bandwidth decides how fast it can generate. Every generated token reads all the active weights once, so bandwidth divided by bytes per token is the most tokens per second any software can reach. Real engines stay below this ceiling.

So the tiers below are memory sizes, and within each tier the cards are ranked by bandwidth. Every speed is the ceiling from the Can I run it? calculator for one conversation at an 8,192-token context, in tokens per second: an upper bound, not a benchmark. "Offload, n" means the model only runs with part of it in system RAM (assumed 32 GB of DDR5-5600), and n is the ceiling then. "No" means it does not fit even with RAM.

There are no prices here, because they change weekly and differ between sellers. Within a generation, more memory and more bandwidth cost more, so the tiers run from the least to the most expensive kind of card; check current prices yourself before choosing between tiers.

Software matters as much as the chip. NVIDIA cards use CUDA, which llama.cpp, Ollama, LM Studio, vLLM and nearly every other engine support. AMD cards run llama.cpp through ROCm on the cards AMD lists for it, or through Vulkan, which works on any card with a Vulkan driver; AMD's ROCm 10.1 compatibility list covers the current Radeon RX 7000 and 9000 cards except the RX 7600 XT. Intel Arc cards use llama.cpp's SYCL backend (the Arc B580 is on its verified list) or Vulkan. Engines beyond llama.cpp, such as vLLM, support NVIDIA best. The software notes for every card are in the calculator's hardware table.

Entry tier: 8 to 12 GB

An 8 GB card holds 8-billion-parameter models at Q4_K_M with room for a long context (6.58 GiB for Llama 3.1 8B at 8k), and little more. A 12 GB card also holds Qwen3 14B at Q4_K_M (10.66 GiB at 8k). gpt-oss-20b needs 14.01 GiB, so on these cards part of it runs from system RAM; as a mixture-of-experts model it stays usable that way.

What the arithmetic says: if you can, choose 12 GB over 8 GB, because it moves you from 8B to 14B models. Among the 12 GB cards, the NVIDIA GeForce RTX 3080 Ti has the highest bandwidth in this list.

Mainstream tier: 16 GB

16 GB holds gpt-oss-20b in its published MXFP4 form entirely in VRAM, Qwen3 14B at up to Q6_K, and Mistral Small 3.2 24B Instruct (2506) at Q4_K_M with an 8k context (15.93 GiB, with very little to spare). 30B-class dense models do not fit at Q4_K_M.

What the arithmetic says: every card here runs the same models; bandwidth separates them by 4.3 times, from the NVIDIA GeForce RTX 5080 (960 GB/s) down to the Intel Arc Pro B50 Graphics (224 GB/s). Several 16 GB cards also come in an 8 GB version (RTX 4060 Ti, RTX 5060 Ti, RX 9060 XT); check the memory size, not just the name.

High-end tier: 20 to 32 GB

24 GB is where 30B-class dense models fit: Qwen3 32B at Q4_K_M takes 21.67 GiB at 8k and Gemma 4 31B 20.23 GiB. 32 GB adds headroom for longer context or Q6_K. A 70B model at Q4_K_M (43.7 GiB at 8k) does not fit on any single card in this tier.

What the arithmetic says: the NVIDIA GeForce RTX 5090 has 32 GB and the most bandwidth in this tier (1792 GB/s), which puts its ceilings well above the rest. Among 24 GB cards, the RTX 3090, RTX 3090 Ti, RTX 4090 and RX 7900 XTX are within 8% of each other on bandwidth, so the ceilings are close; the differences between them are software, power draw and price. Several 32 GB workstation cards (AMD Radeon AI PRO, Intel Arc Pro B65 and B70) trade bandwidth for memory.

Two cards instead of one

Two 24 GB cards hold a 70B model at Q4_K_M entirely in VRAM: 44.7 GiB in 48 GiB. The ceiling is then 20.7 tokens per second on two RTX 3090s and 22.3 on two RTX 4090s, against 3.9 with one card and the rest in system RAM. A second card adds memory, not bandwidth: llama.cpp's default layer split passes each token through the cards one after the other. Two cards also need a motherboard with two suitable slots, a power supply for both, and room for the heat.

Workstation tier: 48 to 96 GB

These are professional workstation cards. 48 GB holds a 70B model at Q4_K_M on one card. 96 GB holds gpt-oss-120b in its published MXFP4 form (62.05 GiB at 8k) with room for context.

Unified memory instead of a graphics card

A Mac, an AMD Ryzen AI Max mini-PC or an NVIDIA DGX Spark shares one large pool of memory between CPU and GPU. They hold models no single consumer card can, at lower bandwidth than high-end cards. Each row is the largest memory configuration the vendor offers, with the GPU's default share of it:

The Apple Silicon vs NVIDIA guide goes through the trade-off: capacity against bandwidth, and what each costs to run.

Where to buy

GPUCostLab does not list prices. The manufacturer pages linked from the hardware table give each card's full specifications; for current prices, check retailers such as Amazon and confirm the memory size on the listing. Cards from board partners use the same GPU and memory size; their clocks, power limits and coolers can differ.

What this guide does not tell you

• Measured speed. Every speed here is a bandwidth ceiling. Real speed depends on the engine, its version, the quantization kernels and the driver, and is lower. Measured results are planned and will be published with their scripts.
• Prompt processing. Reading a long prompt is limited by compute, not bandwidth, and is not ranked here.
• Quality. A smaller quantization fits more but loses some accuracy, by an amount that depends on the model.
• Noise, heat and power. The electricity calculator turns each card's rated power into a monthly cost.

Also available as Markdown.