Introduction

Qwen 3.8 27B in NVFP4 needs about 19 GB of VRAM for its weights. NVFP4 is a 4-bit format for model weights from NVIDIA, and it needs about half the memory of the 8-bit FP8 format. This makes it one of the few strong models that can run on a single 32 GB GPU. The question is how much memory is left for the KV cache, and so for the context, and which features cost the most.

We measured this on vLLM 0.30.0, a server that runs models on the GPU and serves many requests at the same time. We used the NVIDIA NVFP4 checkpoint and an FP8 KV cache. NVFP4 needs a modern NVIDIA GPU with the Blackwell architecture, for example a GeForce RTX 50 series card. vLLM itself also runs on older NVIDIA GPUs, but with other model formats. We changed one feature at a time:

• MTP (multi-token prediction) on or off;
• image support on or off;
• CUDA graphs on or off;
• the number of parallel requests and tokens per step;
• the memory budget of a 32 GB card, and in a separate experiment of a 24 GB card.

We also checked that the server stays stable under parallel load.

Running several sessions at once with vLLM

We use vLLM because it serves several sessions at the same time from one shared memory pool. For example, a coding agent can work on a task while you ask the same model another question in a second chat.

llama.cpp can also run several sessions, but each session gets its own fixed part of the memory when the server starts. With a small GPU, people therefore usually choose one large session instead of several small ones. vLLM does not split the memory in advance: the KV cache is shared, and each request takes as much as it needs at the moment. The same memory can hold one long session or several shorter ones.

Test setup

We ran all tests on one modern NVIDIA GPU with the Blackwell architecture. To test a 32 GB and a 24 GB card, we limited the memory that vLLM can use with --gpu-memory-utilization. The limit is the memory the system reports for the card, minus 1.5 GB: 30.34 GB for a 32 GB card such as the RTX 5090, which reports 31.84 GB, and 22.5 GB for a 24 GB card. A real card can differ slightly, for example when other programs use the GPU. All numbers come from the vLLM startup log, and each configuration was started once.

Results for a 32 GB GPU

MTP is the most expensive feature: with the same settings, the KV cache holds about 40% fewer tokens with MTP. Image support costs much less. The charts below show the results for the 32 GB budget; the subsections explain each feature, the stability under load and the startup time.

The model supports a context of 262,144 tokens. With --max-model-len auto, vLLM lowers the context to what fits into the KV cache and writes the new value into the log. The memory budget of 30.34 GB is shared by three parts: the model weights, the KV cache, and activations with CUDA graphs.

How much memory MTP costs

On the 32 GB budget, MTP takes about 2 GB of memory; with the same settings, the KV cache holds about 40% fewer tokens.

On a small budget, MTP has a clear price in three places:

• Weights: the MTP layer needs about 0.8 GB.
• Activations and CUDA graphs: about 1.3 GB more.
• KV cache per token: with MTP, one token needs about 20% more KV cache memory, so the same memory holds fewer tokens.

How much memory image support costs

Image support costs about 1 GB and 12% of the context. Qwen 3.8 27B can read images, for example screenshots or photos of documents. With image support, vLLM loads the image encoder of the model, which needs about 0.9 GB to 1 GB. The KV cache drops by about 12% with and without MTP in our measurements. We limited each request to one image and no video:

--limit-mm-per-prompt '{"image":1,"video":0}'

With this limit, the memory reserved for activations did not grow. One image per message is enough for many uses. If you do not need images at all, --language-model-only saves this memory.

Fewer parallel requests and a smaller batch

With 16 requests and 4,096 tokens per step, MTP and CUDA graphs fit with about 169,000 tokens in the KV cache, and this setup is also the fastest. We changed two flags of our first setup: --max-num-seqs 16 instead of 64 requests and --max-num-batched-tokens 4096 instead of 16,384 tokens. vLLM then records CUDA graphs only for up to 16 requests and needs less memory for activations: 4.35 GB instead of 5.59 GB. The KV cache grows from about 131,000 tokens to about 169,000 tokens. A smaller batch processes a long prompt in smaller parts, so the first token of the answer comes a little later.

Running without CUDA graphs

Without CUDA graphs, MTP and almost the full context of the model fit together: about 254,000 tokens in the KV cache instead of about 169,000 tokens. We added --enforce-eager to the recommended command; it turns off CUDA graphs and torch.compile. The memory for activations and CUDA graphs drops from 4.35 GB to 1.63 GB; most of what is left is fixed memory of CUDA and the libraries.

Without MTP as well, the KV cache is the largest: about 315,000 tokens. The full context of the model, 262,144 tokens, fits with room to spare.

The last two bars in the charts above show these variants. Both ran our 99 tasks with 16 tasks at the same time without a single server error or restart.

Speed: the price of a larger KV cache

MTP with CUDA graphs is by far the fastest: about 148 tokens per second (tok/s) for one session. Without MTP, the model writes at about 66 tok/s, without CUDA graphs at about 47 tok/s, and without both at only about 17 tok/s. CUDA graphs let vLLM run each step of the model with less work on the CPU, and MTP writes several tokens in one step. The chart below shows the speed of one session.

The variant with CUDA graphs and without MTP was measured on the 24 GB budget. The speed does not depend on the size of the budget: the variant without CUDA graphs and without MTP wrote at the same speed on 24 GB and on 32 GB. With 8 sessions at the same time, the totals were about 1,015 tok/s with MTP and CUDA graphs, 471 tok/s with CUDA graphs only, 369 tok/s with MTP only and 129 tok/s without both.

Reading long prompts

Reading the prompt is fast in every setup we tested, and this alone is a big strength of vLLM on a modern GPU. When the model reads a prompt, it processes thousands of tokens in one step, so the GPU can use all of its cores at the same time. The extra work on the CPU without CUDA graphs hardly matters here; only writing slows down.

During our runs with 16 tasks at the same time, vLLM read prompts at about 5,500 tok/s to 7,000 tok/s with MTP and about 3,100 tok/s to 5,100 tok/s without MTP, both without CUDA graphs. These are the middle half of the 10-second averages in the vLLM log; they show what our tasks needed, not the upper limit. At 5,000 tok/s, a prompt of 50,000 tokens, for example several source files or a long report, is read in about 10 seconds.

Stability with 16 parallel tasks

The server ran without a single error or restart in all our tests: with MTP on and off, with and without CUDA graphs, on the 32 GB and the 24 GB budget. We used a set of 99 tasks: agentic tasks with tool calls and coding tasks from HumanEval. 16 tasks ran at the same time, and the requests that did not fit into the KV cache waited in a queue.

A few tasks did not finish because the model itself looped or stopped in the middle of an answer. The server kept working. These runs test the server, not the quality of the model, so we do not compare their scores.

Startup time and the kernel cache

The first start took 7 minutes 14 seconds, and later starts took from 1.5 minutes to 4 minutes. Two steps make the first starts slow:

• Kernel compilation. At the very first start, FlashInfer compiles the FP4 kernels for the GPU, which needs the CUDA compiler nvcc. The compiled kernels are saved in ~/.cache/flashinfer.
• Autotuning. FlashInfer tests several kernel settings for the model and saves the best ones. In our log, this took 2 minutes at the first start of a new combination of features, here MTP with image support.

Both results are saved in the cache and used again at the next start.

This is a clear disadvantage compared with llama.cpp and Ollama, which usually load a model in seconds. vLLM is therefore set up with one model that you then work with for a long time. In llama.cpp and Ollama, switching between models is faster and easier, but they are made mainly for one session at a time.

Start command for a 32 GB GPU

The recommended start command for a 32 GB GPU: MTP with CUDA graphs, 16 parallel requests and 4,096 tokens per step, without images. It gives about 169,000 tokens of KV cache, or about 375 pages of text, at about 148 tok/s for one session.

vllm serve nvidia/Qwen3.8-27B-NVFP4 \
  --host 127.0.0.1 --port 8000 \
  --dtype auto \
  --language-model-only \
  --skip-mm-profiling \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --async-scheduling \
  --enable-chunked-prefill \
  --load-format fastsafetensors \
  --max-model-len auto \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.953 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

• --gpu-memory-utilization is a fraction of the whole GPU memory. An RTX 5090 reports 31.84 GB, and 0.953 gives vLLM 30.34 GB and leaves 1.5 GB for the system, as in our tests. If the desktop runs on the same GPU and needs more, lower the value.
• --kv-cache-dtype fp8 stores the KV cache in 8 bits. This practically doubles the number of tokens compared with the default 16 bits.
• --max-model-len auto picks the longest context that fits.
• --max-num-seqs 16 allows 16 requests at the same time, and --max-num-batched-tokens 4096 processes up to 4,096 tokens in one step. With 64 requests and 16,384 tokens, the KV cache is only about 131,000 tokens.
• For images, remove --language-model-only and --skip-mm-profiling and add --limit-mm-per-prompt.

To see how much fits, look for these lines in the vLLM log after the start:

GPU KV cache size: 169,400 tokens
Maximum concurrency for 168,000 tokens per request: 1.01x

Longer context on a 32 GB GPU

About 254,000 tokens of KV cache, or about 560 pages of text, at about 47 tok/s. Almost the full context of the model fits, and the speed stays well above 30 tok/s. Add this flag:

--enforce-eager

Experiment: a 24 GB GPU

We did not test a real 24 GB card: as in all tests, we limited the memory of our GPU to the budget of such a card. With 1.5 GB left free, a 24 GB card gives Qwen 3.8 27B only about 47,000 tokens of context, or about 100 pages of text, at about 66 tok/s. Start from the command above, remove --speculative-config and set --gpu-memory-utilization so that 1.5 GB stay free. The model starts only without MTP and with 16 parallel requests and 4,096 tokens per step:

• 64 requests, 16,384 tokens per step: 18.89 GB of weights and about 3.8 GB of activations and CUDA graphs leave nothing for the KV cache. The server does not start.
• 16 requests, 4,096 tokens per step: activations and CUDA graphs need only 1.82 GB, the KV cache gets 1.79 GB, and the context is 47,040 tokens.

--max-num-seqs sets how many requests run at the same time, and --max-num-batched-tokens sets how many tokens vLLM processes in one step. Smaller values save about 2 GB, but fewer requests run in parallel and long prompts are processed in smaller parts. On a GPU without a desktop, where 0.5 GB free is enough, the context grows to about 80,000 tokens. MTP does not fit on 24 GB: even without CUDA graphs and with a small batch, it leaves only about 20,000 tokens for the KV cache. Without MTP and without CUDA graphs, the KV cache grows to about 66,000 tokens, but the model then writes only about 17 tok/s.

47,000 tokens are still useful for chat, for experiments and for learning what local AI can do. For agentic coding, the context will be too small even for small projects: the agent's instructions, the tool descriptions and the files it reads fill it quickly. A client must also ask for fewer output tokens than usual, because vLLM rejects a request when the prompt plus the requested output does not fit into the context. On a 24 GB card without a desktop, for example a second GPU, about 80,000 tokens are good enough for experiments and learning. If you already have a 24 GB card, it is worth trying.

Recommendation

Every hardware option for local LLMs has a trade-off, so we do not recommend any specific hardware:

• One 32 GB GPU is fast, but expensive.
• Two cheaper GPUs with 32 GB together cost much less and still run at an acceptable speed, but need a setup for two GPUs.
• A compact NVIDIA system with 128 GB of unified memory, such as the DGX Spark, holds large models and many parallel sessions, but it is much slower. If you consider it, watch reviews that test real everyday use, not only short benchmarks.

The amount of memory is not the only thing to look at when you choose hardware: it decides which models and how much context fit, but the speed of the memory decides how fast the model writes.

There are many unknowns at every step, which makes the start with local LLMs hard. Before you buy hardware, check the Hardware page for the estimated speed of models on different hardware and which models fit, and then do your own research.

References

• vLLM and its documentation
• nvidia/Qwen3.8-27B-NVFP4 on Hugging Face