VLLM users encounter CUDA errors during inference

“Your current environment GPU: NVIDIA B300, 8 GPUs CUDA: 13.0.x NVIDIA driver: 595.99.02 vLLM: 0.31.0 PyTorch: 2.13.0 Model: Kimi K3, hybrid Attention/Mamba architecture Tensor parallel size: 8 KV cache dtype: FP8 KV offloading backend: native KV offloading size: 512 Prefix caching: enabled 🐛 Describe the bug vLLM EngineCore crash in native KV offloading for hybrid Attention/Mamba model Summary When using vLLM native KV offloading with a hybrid Attention/Mamba model, the EngineCore can terminate”

“Your current environment vLLM: ROCm build based on 0.29; vllm/models/qwen4 exp/amd/model state.py sha1 81e31496 (identical to main e126687a9a); ple layer.py path checked on main (e2414ec1db). GPUs: 2x AMD Radeon AI PRO R9700 (gfx1201), ROCm 10. Model: Qwen3.8 Flash Next (quantized), MTP speculative decoding, CUDA graphs FULL AND PIECEWISE. 🐛 Describe the bug Summary On the AMD backend, the PLE n gram ID generation for Qwen4Exp (Qwen3.8 Flash Next) runs inside the PIECEWISE CUDA graph pieces. It”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details Summary: GPU: NVIDIA H200 NVL exposed to the guest as an SR IOV vGPU (70 GiB profile), Proxmox VE host vGPU manager (host): 595.91.04, guest driver: 595.91.07 Reproduced on vLLM 0.31.0 (wheel) and on main @ f8471164b (precompiled), Python 3.14, torch 2.13.0, TP=1 🐛 Describe the bug With CPU KV offloading enabled ( OffloadingConnector ), the engine dies on the first batch of GPU→CPU sto”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug On a 24 GiB GPU, LLM(..., kv cache dtype="fp8") is accepted but fails during startup with an out of memory error, while the identical engine using the default KV cache dtype answers successfully. Observed failure: The startup accounting allocates about 19.93 GiB to the KV cache after subtracting a near zero CUDA graph estimate. CUDA graph capture happens afterwards a”

“Your current environment vLLM version: 0.30.0 Model: Gemma 4 26B A4B (NVFP4, ModelOpt quantization) GPU: NVIDIA RTX PRO 4500 Blackwell (32 GiB), driver 610.57.04 Deployment: official image docker.io/vllm/vllm openai:v0.30.0, podman 🐛 Describe the bug Describe the bug On /v1/chat/completions, the presence of any request level chat template kwargs causes all images in the request to be silently dropped. The rendered prompt contains only the text portion (usage.prompt tokens ≈ 20 instead of ≈ 276”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Env vLLM 0.27.1 and 0.29.0 (pip wheels, same machine and same torch), torch 2.13.0+cu130, triton 3.7.1, CUDA 13.0; 0.29.0 reproduces this too NVIDIA GeForce RTX 4090 (sm89), driver 580.126.09, single GPU, Linux 5.15 model: Qwen/Qwen3 8B AWQ (listed under "Tested Models" in docs/features/batch invariance.md); quantization=auto awq controls: Qwen/Qwen3 8B (bf16), Qwen/”

“Your current environment vLLM 0.29.0 (image vllm/vllm openai:v0.29.0 + ray 2.56.1, pip package b12x 1.3.0) torch 2.13.0+cu130, CUDA 13.0, transformers 5.16.1 GPU: 6x NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120), driver 595.91.07 Host: Ubuntu, kernel 7.0.0 31 generic, Docker Cluster: two nodes (4 + 2 GPUs), Ray 2.56.1, 100 GbE direct link Model: nvidia/GLM 5.2 NVFP4, PP=3 / TP=2, VLLM PP LAYER PARTITION=31,29,18 Launch command: vllm serve nvidia/GLM 5.2 NVFP4 distributed executor backend”

“Your current environment <details <summary The output of <code python collect env.py</code (run inside the vLLM nightly container; the host driver is 535.183.01)</summary vLLM version: 0.2.1.dev58+g9854b580dfe518171eed3a6ad76cbd402ca2bdfa (nightly wheel, 2026 09 17) </details 🐛 Describe the bug With engram config '{"cpu offload": true}' (pinned host PLE / UVA lookup, 54371) and compilation config '{"cudagraph mode": "FULL AND PIECEWISE"}' , engine startup fails intermittently during profile cud”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details Notes: collect env.py ran inside a one GPU scheduler reservation (hence CUDA VISIBLE DEVICES=5 ); the reproduction below uses two of the eight B300s. vLLM is the upstream main commit 5be911ee871604895c5a950ac1a5675c71896fd2 (2026 09 15), installed with the matching precompiled wheel, unmodified tree. Model RedHatAI/GLM 5.2 NVFP4 FP8 . 🐛 Describe the bug Summary Adding the two HiSparse”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug My environment (I have been copy and pasting the same env config above for a while so some info there maybe in accurate, but I have to be in queue for a compute node to run those command, which takes forever, so below is a short des of my enviroment nowadays) vLLM 0.27.1 (pip wheel), torch 2.13.0+cu130. RTX 4090 (sm 89) and H100 80GB (sm 90), eager mode ( enforce eag”

“Your current environment <details <summary The output of <code python collect env.py</code (run inside the vLLM nightly container; the host driver is 610.57.04 — the container itself cannot collect it)</summary </details Driver version note: collect env.py reports "Could not collect" on this box because driver 610.57.04 removed the Driver Version: <x string it greps for — plain nvidia smi now prints only NVIDIA SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 , and nvidia smi q pads t”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug For encoder decoder models (e.g. Whisper class ASR serving), the V1 scheduler releases encoder inputs through Scheduler. free encoder inputs on every step for requests with encoder inputs ( vllm/v1/core/sched/scheduler.py:1951 , called from update from output ). Once the request has generated at least one token ( num computed tokens 0 ), that path frees all cached in”

“Your current environment <details <summary Output of <code vllm collect env</code (inside the container) plus host details</summary </details Command line (docker image entrypoint = vllm serve ): Effective config as logged by core.py:122 (excerpt): dtype=torch.bfloat16, quantization=fp8, max seq len=65536, enforce eager=False, kv cache dtype=auto, enable prefix caching=True, enable chunked prefill=True (max num batched tokens=8192), compilation config={'mode': VLLM COMPILE, 'cudagraph mode': FUL”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details The crashes below were on a 2x H100 container built from the same image ( nvidia/cuda:12.8.1 devel ubuntu22.04 + pip); collect env.py was run on a 1 GPU container of that image, so the GPU topology section shows one card. Your code version <details <summary The commit id or version of vllm</summary </details <details <summary The commit id or version of vllm omni</summary </details 🐛 D”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Describe the bug BGE embedding and reranker startup aborts during engine construction. To Reproduce Run this standalone Python program on the target machine after ensuring the selected model path exists and the GPU has sufficient free memory: python import subprocess VLLM = "/root/anaconda3/envs/vllm/bin/vllm" MODEL = "/root/.cache/huggingface/hub/models BAAI bge lar”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Describe the bug Qwen3 Embedding reaches backend selection and exits instead of receiving a preflight compatibility error. To Reproduce Run this standalone Python program on the target machine after ensuring the selected model path exists and the GPU has sufficient free memory: python import subprocess VLLM = "/root/anaconda3/envs/vllm/bin/vllm" MODEL = "/root/.cache”

“Your current environment . 🐛 Describe the bug Description After running successfully for several days, our vLLM cluster crashed with EngineDeadError due to a KV cache block not found during offloading load operation. Environment Model : DeepSeek V4 Pro (custom, 671B MoE) Hardware : 3 nodes × 8 H100 (24 GPUs) Deployment : TP=8, DP=3 vLLM Version : v0.28.0 Uptime before crash : Several days Deployment Command Error Stack trace : Node1 : Node2 : Before submitting a new issue... [x] Make sure you a”

“Your current environment Environment vLLM image version: vllm project/vllm:v0.27.1 (built with INSTALL KV CONNECTORS=true) Host GPU driver / CUDA runtime: CUDA 13.0 PyTorch in image: torch==2.13.0+cu130 LMCache resolved version: lmcache==0.5.3 Working comparison: vllm project/vllm:v0.22.0 (torch==2.11.0) loads lmcache.c ops correctly. 🐛 Describe the bug Title: INSTALL KV CONNECTORS=true in vLLM 0.27.1 Docker image installs LMCache wheel incompatible with shipped PyTorch, causing lmcache.c ops t”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug When launch vllm serve in EP and use deepep v2 + deep gemm + cudagraph: all2all backend: deepep v2、moe backend: deep gemm、cudagraph mode: FULL AND PIECEWISE It would crash in deep gemm as showing in crash log Before submitting a new issue... [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the document”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Setup: vLLM 0.25.2.dev0+g752a3a504 , built from source (editable install) PyTorch 2.11.0, CUDA toolkit 13.2 (nvcc 13.2.51), Python 3.12 3 nodes x 4x NVIDIA H100 80GB SXM (HBM3), Ray distributed executor tensor parallel size 4 pipeline parallel size 3 Model: nvidia/NVIDIA Nemotron 3 Ultra 550B A55B NVFP4 🐛 Describe the bug MTP speculative decoding ( speculative confi”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details Deployment Environment vLLM version: v0.25.1 (image vllm/vllm openai:v0.25.1 ) — also reproduced on v0.25.0 NIXL version: 1.3.1 (git 7cf3019f ), unchanged across both reproductions Connector: NixlConnector ( NixlPullConnector ), kv role=kv both Deployment: P/D disaggregation on Kubernetes (llm d pd disaggregation guide), with the llm d routing sidecar ( ghcr.io/llm d/llm d routing sidec”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Summary Starting with vLLM 0.25.1, a large dense model (e.g. Llama 3.1 70B Instruct FP8, TP=8) that previously started fine now OOMs during CUDA graph capture ( capture model() ), even with an otherwise unchanged config. The regression is caused by the interaction of two things: 1. Model Runner V2 is now the default for dense models. 2. The V2 runner's profile cudagr”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Nvidia H100 4 test cmd Exception log others Before submitting a new issue... [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.”

“=== My current environment === <details <summary Output of <code python m vllm.collect env</code (run inside the affected H20 container)</summary Also reproduced on vLLM 0.24.0 and 0.25.0 — vllm/lora/ops/triton ops/lora expand op.py and kernel utils.py are byte identical across 0.23/0.24/0.25, so upgrading does not fix it. </details === 🐛 Describe the bug === Summary On NVIDIA Hopper (sm 90, e.g. H20/H100) , serving a Qwen3 VL 8B Instruct LoRA adapter with vLLM produces completely garbled outpu”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Describe the bug quantization humming crashes reproducibly during startup on vLLM 0.24.0 with CUDA illegal memory access for a local Qwen3.5 MoE model. The same model / hardware / runtime setup works with the default marlin backend, so this appears to be humming specific rather than a generic startup, NCCL, or environment issue. Repro command text RuntimeError: Trito”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Before submitting a new issue... [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.”

“(vllm env) [root@ip 172 31 33 115 ] python collect env.py Collecting environment information... ============================== System Info ============================== OS : Amazon Linux 2023.11.20260526 (x86 64) GCC version : (GCC) 11.5.0 20240719 (Red Hat 11.5.0 5) Clang version : Could not collect CMake version : version 3.22.2 Libc version : glibc 2.34 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used t”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Trying to load the GPT2 model inferno project/vllm gpt2 1 leads to the following assertion error when kv cache dtype is one of float16 , bfloat16 : Triton attention should support these dtypes and indeed this doesn't happen with the model openai community/gpt2 . Reproducer Full traceback traceback.log Here is the error traceback: Before submitting a new issue... [x]”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Describe the bug Version: vLLM 0.17.1 Model: Qwen/Qwen2 0.5B Hardware reproduced on: NVIDIA GeForce RTX 4090 , single GPU Summary I found a double streaming update bug in the AsyncLLM path that can break the worker's request ID previous batch row mapping for a resumed request and then kill the engine. The reproducer submits one resumable request and then applies two”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Issue: Unsloth trained Gemma 4 Adapters Ineffective in vLLM While the bitsandbytes compatibility for Gemma 4 has been addressed (see PR 40321), LoRA adapters trained via Unsloth (e.g., for the 31B model) currently fail to apply correctly in vLLM. Observations: Tested Hardware: RTX 4090 and A6000 Pro. Behavior: The engine loads without crashing, but the adapter is eff”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug When hosting Gemma4 31B it on vLLM 0.20.0 with streaming interval larger than 1 and invoking the model in streaming mode, the last few reasoning output tokens are missing. Script to launch the vLLM server: Script to demonstrate the bug: Outputs: You can clearly see that in streaming mode, the last few tokens are missing, as the reasoning content ends with: Explain th”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Describe the bug When running vLLM V1 with both decode context parallel size 1 and speculative config '{"method": "eagle3", ...}' , the engine crashes during the first request at the speculative decoding phase. The error indicates that the temporary workspace required by DCP attention ( forward with dcp ) has a size of 0.00 MB and is locked, preventing allocation. To”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Gemma 4 fails to initialize in vLLM V1 when kv cache dtype int8 per token head is enabled. The same issue also affects fp8 per token head . The failure is caused by Gemma 4's hybrid attention layout using two different KV head dimensions: sliding/local layers use head dim = 256 global/full attention layers use head dim = 512 With per token head KV quantization, each”

“Your current environment Description I am developing an Always On voice chatbot using Voxtral via the vLLM Realtime WebSocket API. In this architecture, the WebSocket stays open indefinitely, continuously streaming audio chunks (including silence/ambient noise) to the backend. I have discovered that the Encoder KV Cache accumulates acoustic tokens indefinitely. When the total token count reaches max model len , instead of evicting old tokens (as a sliding window should), vLLM attempts to append”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details vLLM version : v0.17.1rc1.dev126+gbc2c0c86e (nightly, vllm/vllm openai:nightly ) Model : cyankiwi/Qwen3.5 27B AWQ BF16 INT4 Quantization : compressed tensors (AWQ BF16 INT4) Hardware : 2× NVIDIA GPU (tensor parallel size=2), WSL2 (Linux 6.6.87.2 microsoft standard WSL2) Attention backend : FlashInfer ( attention backend FLASHINFER ) Python : 3.12 PyTorch : inferred from nightly image 🐛”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Environment Host OS: Ubuntu 22.04 GPUs: 4x NVIDIA GeForce RTX 3060 12GB docker model version : Client: v1.1.8 Server: v1.1.11 docker model status : llama.cpp Running vllm Running (0.17.0) Model huggingface.co/qwen/qwen3.5 9b Format shown by docker model show : safetensors Size: 19.31GB What I expected Since this is a safetensors model and Docker Model Runner routes s”

“🚀 The feature, motivation and pitch Currently when vllm is used in conjunction with other gpu programs, e.g. RL (verl+vllm) and the sleep mode is used, we end up with very bogus torch.cuda memory counters. The torch.cuda memory reporting is all broken in that situation, since vllm somehow frees up all the kv cache and weights when put to "sleep" (because we need the same gpu's mem to be free to do a training step) but torch isn't the wiser the freeing happened the same gpus are shared between i”

“Your current environment <details <summary The output of <code python collect env.py</code </summary (executed within the vLLM v0.14.0 docker container) </details 🐛 Describe the bug Summary nvidia/DeepSeek R1 0528 NVFP4 works with vLLM v0.13.0 on 8 x RTX Pro 6000 (Blackwell), but fails with vLLM v0.14.0 with the following error: [v1/executor/multiproc executor.py:749] ValueError: No valid attention backend found for cuda with AttentionSelectorConfig(head size=576, dtype=torch.bfloat16, kv cache”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug After upgrading from vLLM v0.12.0 to v0.13.0, I encountered a torch.OutOfMemoryError during the model weight loading phase. The same configuration and environment work perfectly fine in v0.12.0. The error occurs when loading Qwen2.5 72B Instruct with FP8 quantization on an H20 (141GB) GPU. It seems the memory management or the weight loading logic in the new version”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details GPU Hardware Information Product Name : NVIDIA RTX PRO 6000 Blackwell Server Edition Product Brand : NVIDIA Product Architecture : Blackwell Video Memory : 96GB GDDR7 Software Versions: Driver Version: 580.82.07 CUDA Version: 12.9 vllm version: v0.10.2rc1 Installed via command: uv pip install vllm torch backend=auto extra index url https://wheels.vllm.ai/5438967fbc7a10ae6eee7a98182f4de9”