Nicolas Patry's picture

Nicolas Patry

Narsil

·

https://github.com/Narsil/

AI & ML interests

None yet

Recent Activity

posted an update 12 days ago

Performance leap: TGI v3 is out. Processes 3x more tokens, 13x faster than vLLM on long prompts. Zero config ! 3x more tokens. By reducing our memory footprint, we’re able to ingest many more tokens and more dynamically than before. A single L4 (24GB) can handle 30k tokens on llama 3.1-8B, while vLLM gets barely 10k. A lot of work went into reducing the footprint of the runtime and its effect are best seen on smaller constrained environments. 13x faster On long prompts (200k+ tokens) conversation replies take 27.5s in vLLM, while it takes only 2s in TGI. How so ? We keep the initial conversation around, so when a new reply comes in, we can answer almost instantly. The overhead of the lookup is ~5us. Thanks @Daniël de Kok for the beast data structure. Zero config That’s it. Remove all the flags your are using and you’re likely to get the best performance. By evaluating the hardware and model, TGI carefully selects automatic values to give best performance. In production, we don’t have any flags anymore in our deployments. We kept all existing flags around, they may come in handy in niche scenarios. Read more: https://huggingface.co/docs/text-generation-inference/conceptual/chunking

new activity 14 days ago

huggingchat/chat-ui:Your feedback on HuggingChat

upvoted a paper about 2 months ago

GPT-4o System Card

View all activity

Articles

Hugging Face partners with Wiz Research to Improve AI Security

Safetensors audited as really safe and becoming the default

Optimization story: Bloom inference

Making automatic speech recognition work on large files with Wav2Vec2 in 🤗 Transformers

Organizations

Narsil's activity

New activity in huggingchat/chat-ui 14 days ago

Your feedback on HuggingChat

#1 opened over 1 year ago by

New activity in CohereForAI/c4ai-command-r-plus 4 months ago

Fix the post processor to reflect what happens in transformers.

#61 opened 4 months ago by

New activity in mistralai/Mistral-Nemo-Instruct-2407 5 months ago

"Model mistralai/Mistral-Nemo-Instruct-2407 time out" in Inference APIs

#42 opened 5 months ago by

New activity in microsoft/Phi-3-mini-4k-instruct 6 months ago

Create the tokenizer.json properly (with TemplateProcessing included).

#75 opened 6 months ago by

New activity in 01-ai/Yi-1.5-34B-Chat 7 months ago

Adding a fast tokenizer

#10 opened 7 months ago by

New activity in 01-ai/Yi-1.5-9B-Chat 7 months ago

Adding a fast tokenizer.

#10 opened 7 months ago by

Add fast tokenizer

#9 opened 7 months ago by

New activity in ibm-fms/llama3-8b-accelerator 7 months ago

ValueError: Unsupported model type mlp_speculator using TGI server

#2 opened 7 months ago by

rishabh-simpplr

New activity in text-generation-inference/commandrplus-medusa 8 months ago

Is `--speculate` needed for Medusa in TGI?

#1 opened 9 months ago by

New activity in CohereForAI/c4ai-command-r-plus 9 months ago

Is <BOS_TOKEN> really needed?

#22 opened 9 months ago by

New activity in diffusers/convert 9 months ago

Error: 'NoneType' object has no attribute 'pr_url'

#4 opened 9 months ago by

New activity in cognitivecomputations/dolphin-2.1-mistral-7b 10 months ago

Adding `safetensors` variant of this model

#21 opened 10 months ago by

New activity in FasterDecoding/medusa-vicuna-7b-v1.3 10 months ago

Adding safetensors variant

#1 opened 10 months ago by

New activity in coqui/XTTS-v2 11 months ago

Add safetensors variants

#41 opened 11 months ago by

New activity in joaogante/medusa-maker 12 months ago

Update app.py

#1 opened 12 months ago by

New activity in mistralai/Mixtral-8x7B-Instruct-v0.1 about 1 year ago

CUDA error when initialising model with text-generation-inference

#22 opened about 1 year ago by

sliding_window

#37 opened about 1 year ago by

New activity in cognitivecomputations/dolphin-2.2-70b about 1 year ago

Safetensors version?

#2 opened about 1 year ago by

New activity in Narsil/small2 about 1 year ago

Adding `safetensors` variant of this model

#5 opened about 1 year ago by

Adding `safetensors` variant of this model

#4 opened about 1 year ago by