Can you provide a FP8 version?

#11

by xjpang85 - opened Jan 17

Discussion

xjpang85

Jan 17

Can you provide a FP8 version for less GPUs.

MiniMax-AI

MiniMax org Jan 18

Can the int8 or int4 version meet the requirements?

21world

Jan 18

int4 ok

zzzsssqqq

MiniMax org Feb 20

We have submitted a 13454 to vllm, and the inference performance improvement compared to the Hugging Face's implementation is very significant. You can give it a try.

sriting

MiniMax org 23 days ago

Hi @xjpang85 , our vllm support has been merged today. Feel free to use it with our Text-01 model
https://github.com/vllm-project/vllm/pull/13454

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment