cognitivecomputations

/

DeepSeek-R1-AWQ

like 75

Follow

Cognitive Computations 1.77k

What's your environment? OS version? Also please use the newer version now, I have updated the wheel to v0.8.3.

I have met the same error , would you please tell me what's the version of your system ? would you share the dockerfile or docker images to close this question?

I am using the official CUDA Docker image for Ubuntu 24.04, try using this.

I see that the vllm.whl file you shared was compiled under CUDA 12.8, but I couldn’t find a PyTorch 2.6 version for CUDA 12.8. May I ask which PyTorch version you installed? During execution, I encountered the following error:Have you encountered this issue before? Could you provide guidance or share your pip requirements.txt file?
ImportError: /workspace/vllm/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

I used the default CUDA version for PyTorch 2.6. You can get the same version by just running pip install torch.
I admit that my system config is kind of strange, but I used the Ubuntu 24.04 CUDA 12.8.1 CUDNN DEV version of the Docker CUDA image, updated all packages by running apt update && apt upgrade -y, then installed PyTorch using pip, compiled the library using this config.
You can try the same setup and try the wheel. It runs normally for me.
The GLIBC issue OP posted looked like it's caused by him using an outdated version of the OS, potentially Ubuntu 20.04 or Ubuntu 22.04, which use an older GLIBC. But your issue looks more like a PyTorch version issue. I was thinking that maybe you used the CXX11 ABI version of PyTorch, the default PyTorch until 2.6 was built with CXX11 ABI disabled, did you have anything weird going on when installing PyTorch? Are you using an older version or a newer, nightly version of PyTorch? PyTorch versions are generally not ABI compatible, you should match the version exactly.
If the above doesn't work for you, merge the 2 PRs I linked, and merge my forked branch, then build it yourself, and it should work fine.

My environment is: CentOS Linux release 7.9.2009 (Core), CUDA 12.4, 8xA100 (80GB).
I deployed the Deepseek-AWQ model using the following commands:

1.conda create -n vllm python=3.12 -y
2.conda activate vllm
3.pip install vllm-0.8.3.dev250%2Bg10afedcfd.cu128-cp312-cp312-linux_x86_64.whl
4.VLLM_USE_V1=0 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_MARLIN_USE_ATOMIC_ADD=1 python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 12345 --max-model-len 65536 --max-seq-len-to-capture 65536 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --tensor-parallel-size 8 --gpu-memory-utilization 0.95 --served-model-name deepseek-reasoner --model /local/DeepSeek-R1-AWQ
It still reported an error:
ImportError: /lib64/libc.so.6: version 'GLIBC_2.32' not found

I tried building VLLM using alternative methods with the following commands:

1.export VLLM_COMMIT=xxxx # use full commit hash from the main branch
2.pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
3.VLLM_USE_PRECOMPILED=1 pip install --editable .
4.python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 12345 --max-model-len 24000 --trust-remote-code --tensor-parallel-size 8 --gpu-memory-utilization 0.85 --max-num-seqs 8 --dtype float16 --served-model-name deepseek-reasoner --model /local/DeepSeek-R1-AWQ

This time, the previous error disappeared, but a new error occurred:
TypeError: BackendCompilerFailed.init() missing 1 required positional argument: 'inner_exception'

I believe this is due to an issue with the VLLM framework, so I searched for answers on VLLM's GitHub but couldn't find similar errors. I have submitted these issues to VLLM's GitHub, hoping for a resolution.

@lastsummerLi Your OS version is too low, try updating. The easiest way without updating is to use Docker, it is designed for environment issues. Use the Docker image I specified, and it should work.

@lastsummerLi Your OS version is too low, try updating. The easiest way without updating is to use Docker, it is designed for environment issues. Use the Docker image I specified, and it should work.

Thank you, I will try to use the Docker approach for deployment.