Introduction

For local use on a smaller GPU, three vLLM features matter most, and together they decide how large a model fits and how fast it writes:

• NVFP4 weights need about half the memory of FP8.
• An FP8 KV cache (--kv-cache-dtype fp8) stores the KV cache, the memory that holds the context, in 8 bits instead of 16 bits. This practically doubles the number of tokens that fit.
• MTP (multi-token prediction) lets the model guess several tokens ahead and check them in one step, so it writes faster.

For a long time, this combination did not run reliably in vLLM. We ran Qwen 3.6 and Qwen 3.8 models in NVFP4 on vLLM 0.25.0, 0.27.1, 0.29.0 and 0.30.0, and some of them crashed, stopped responding or needed one of the features turned off. This article collects these problems as a short history. If you use an older vLLM version, you can expect them. On vLLM 0.30.0, the Qwen 3.6 and Qwen 3.8 models built on the Qwen 3.5 architecture run with all three features without crashes.

Symptoms that suggest an upgrade

vLLM 0.30.0 is quite new, so many setups still run an older version. A working vLLM setup is often not updated for months, so these problems can stay with you long after they are fixed. These are the problems we saw on older versions with NVFP4 models. If you see them, update vLLM before you change your model settings:

• The server crashes or restarts under parallel load, most of all with a mixture-of-experts model such as Qwen 3.6 35B.
• The server stops responding: requests hang until the client gives up, while the process still runs.
• The model runs only with MTP turned off, or crashes with MTP when more requests run at the same time. On older versions, this happened even with FP8 models, not only with NVFP4.
• The model starts only with special flags: another checkpoint format, explicit backends or a 16-bit KV cache.

Check your version with vllm --version, or with curl http://127.0.0.1:8000/version while the server runs.

Problems on older vLLM versions

vLLM 0.25.0 and 0.27.1: MTP and NVFP4 did not run reliably

On vLLM 0.25.0 and 0.27.1, Qwen 3.6 35B in NVFP4 crashed again and again. Our first setup on 0.25.0 used MTP with five proposed tokens. Soon after, we reduced MTP to one token and removed --calculate-kv-scales. On 0.27.1, the model still crashed, and in one of our benchmark runs the server stopped responding in six tests. We tried many combinations of flags, often several in one day.

Qwen 3.6 27B in FP8: MTP crashed under load

Even the dense Qwen 3.6 27B in FP8, the most reliable setup we had, was not stable with MTP on older versions. It ran with FP8 weights and a 16-bit KV cache. With MTP turned on, the result was a matter of luck: under light load it worked, but with more parallel requests the server crashed. In the end, all our older configurations ran with MTP turned off.

vLLM 0.29.0: a different checkpoint, explicit backends and MTP off

Qwen 3.6 35B became stable only after we changed its setup almost completely:

• the NVIDIA NVFP4 checkpoint with --quantization modelopt_fp4 instead of compressed-tensors;
• MTP turned off;
• explicit backends: --moe-backend marlin for the expert layers and --attention-backend flashinfer;
• --async-scheduling, --block-size 128 and --load-format fastsafetensors.

We made this change together with the update to vLLM 0.29.0. Since then, the model has run without crashes. The start command on 0.29.0:

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 --port 8000 \
  --dtype auto \
  --language-model-only \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --block-size 128 \
  --moe-backend marlin \
  --attention-backend flashinfer \
  --attention-config '{"use_trtllm_attention":true}' \
  --async-scheduling \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --load-format fastsafetensors \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --skip-mm-profiling

What to do on an older vLLM version

Where we are on vLLM 0.30.0

On vLLM 0.30.0, the models built on the Qwen 3.5 architecture run without crashes so far:

• Qwen 3.8 27B runs with the full combination: NVFP4, an FP8 KV cache and MTP with three proposed tokens. It also ran our tests with 16 parallel sessions without a single server error.
• Qwen 3.6 27B in FP8 runs with an FP8 KV cache and MTP of three tokens.
• Qwen 3.6 35B runs with the full combination as well, with image support and the full context of 262,144 tokens. It ran 331 coding and agentic tasks in thinking mode with 16 parallel sessions without a single server error or restart.

Qwen 3.6 35B with all features on vLLM 0.30.0

On vLLM 0.30.0, Qwen 3.6 35B runs with all features on, but only with --moe-backend auto. With --moe-backend marlin, the fix from vLLM 0.29.0, the server does not start with MTP: moe_backend='marlin' is not supported for unquantized MoE. The MTP layer of this model is also a mixture-of-experts block, but it is not quantized, and Marlin supports only quantized ones. With auto, vLLM picks a suitable backend for both, and MTP accepts about 3.1 tokens per step on average. The start command:

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 --port 8000 \
  --dtype auto \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --block-size 128 \
  --moe-backend auto \
  --attention-backend flashinfer \
  --attention-config '{"use_trtllm_attention":true}' \
  --async-scheduling \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --load-format fastsafetensors \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --limit-mm-per-prompt '{"image":4,"video":1}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Qwen 3.8 Flash Next: still not stable on vLLM 0.30.0

It needed four changes in two days:

• The FP8 KV cache did not work, so the cache runs in bfloat16. The same cache size then needs twice the VRAM.
• MTP turned off.
• With CUDA 13.1, the kernel compilation failed on a conflict with the system C library. We fixed it with GCC 14 and a small header file passed to nvcc.
• --moe-backend auto instead of a fixed backend.

Recommendation

Bonus: Qwen 3.6 35B fits its full context on a 32 GB GPU

On vLLM 0.30.0, Qwen 3.6 35B with NVFP4, an FP8 KV cache and MTP fits its full context of 262,144 tokens into the memory of a 32 GB GPU, and writes about three times faster than the dense Qwen 3.8 27B. We limited vLLM to 30.34 GB, the memory of an RTX 5090 minus 1.5 GB for the system, and used the same settings for both models: MTP with three proposed tokens, an FP8 KV cache, up to 16 requests at the same time, 4,096 tokens per step and no image support. The charts below compare both models; in the token generation test, each request generated 512 output tokens.

The reason is the design of the model. Qwen 3.6 35B is a mixture-of-experts model: it uses only about 3B of its parameters for each token, so every step is short. It also needs much less KV cache memory per token than Qwen 3.8 27B, so the same memory holds almost three times as many tokens. In our benchmarks, however, Qwen 3.6 35B gives weaker answers than the 27B models, so it is a fast model, not a better one.

The start command for an RTX 5090:

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host 127.0.0.1 --port 8000 \
  --dtype auto \
  --language-model-only \
  --skip-mm-profiling \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --block-size 128 \
  --moe-backend auto \
  --attention-backend flashinfer \
  --attention-config '{"use_trtllm_attention":true}' \
  --async-scheduling \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --load-format fastsafetensors \
  --max-model-len auto \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.953 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

--gpu-memory-utilization 0.953 gives vLLM 30.34 GB of the 31.84 GB that an RTX 5090 reports and leaves 1.5 GB for the system.

This setup is stable on the 32 GB budget too: it ran the same 331 coding and agentic tasks in thinking mode with 16 parallel sessions without a single server error or restart. MTP accepted about 3.2 tokens per step on average.

References

• vLLM and its documentation
• NVIDIA NVFP4 checkpoints: Qwen3.6-35B-A3B-NVFP4, Qwen3.8-27B-NVFP4, Qwen3.8-Flash-Next-NVFP4
• Our benchmark suite, which runs on the setups described here