catid commited on
Commit
fbc0638
·
verified ·
1 Parent(s): f457f78

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -0
README.md ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ AI Model Name: Llama 3 8B "Built with Meta Llama 3" https://llama.meta.com/llama3/license/
2
+
3
+ This is the result of running AutoAWQ to quantize the LLaMA-3 8B model to ~4 bits/parameter.
4
+
5
+ To launch an OpenAI-compatible API endpoint on your Linux server with 2x 3090 or 4090 GPUs:
6
+
7
+ ```
8
+ git lfs install
9
+ git clone https://huggingface.co/catid/cat-llama-3-70b-awq-q128-w4-gemm
10
+
11
+ conda create -n vllm70 python=3.10 -y && conda activate vllm70
12
+
13
+ pip install git+https://github.com/vllm-project/vllm.git
14
+
15
+ python -m vllm.entrypoints.openai.api_server --model cat-llama-3-70b-awq-q128-w4-gemm --tensor-parallel-size 2 --gpu-memory-utilization 0.95
16
+ ```