nicolo's picture

nicolo

nicolollo

·

AI & ML interests

None yet

Recent Activity

liked a Space 17 days ago

fancyfeast/joy-caption-alpha-two

liked a model about 1 month ago

nomic-ai/nomic-embed-text-v2-moe

reacted to grimjim's post with ❤️ about 1 month ago

This recent paper points to an explanation for the unreasonable effectiveness of Frankenmerges: https://huggingface.co/papers/2502.05171 Specifically, the duplication of layers in Frankenmerges serves a purpose similar to what occurs in their recurrent-depth architecture. Successful frankenmerges that operate without additional fine-tuning are able to recover or "heal" from any damage due to abrupt transitions between layer blocks. Operational replicated layer blocks can provide functional benefits grounded in latent reasoning. Frankenmerges can also result in hybrid reasoning, by splicing together the latent reasoning of different models. Back in April 2024, I was able to duplicate a few layers in the Llama 3 8B model, turning it into a 9B model, without harming benchmarks significantly, despite any transition damage. https://huggingface.co/grimjim/llama-3-experiment-v1-9B My informal experimentation suggested that latent reasoning circuits could occupy continguous stacks of 2-4 layers, though the result was highly sensitive to the choice of transition location between layers.

View all activity

Organizations

nicolollo's activity

upvoted 2 collections about 2 months ago

Qwen2.5-1M

The long-context version of Qwen2.5, supporting 1M-token context lengths • 3 items • Updated 28 days ago • 108

Qwen2.5-VL

Vision-language model series based on Qwen2.5 • 10 items • Updated 2 days ago • 416

upvoted a collection 2 months ago

VideoChat-Flash

Faster and more powerful VideoChat. • 15 items • Updated 18 days ago • 11

upvoted a collection 5 months ago

SmolLM2

State-of-the-art compact LLMs for on-device applications: 1.7B, 360M, 135M • 16 items • Updated Feb 20 • 251

upvoted a collection 7 months ago

SFT

9 items • Updated Aug 18, 2024 • 2

upvoted a collection 8 months ago

main releases

powerful small models aimed to become good at chat/text while avoiding the usage of system prompts • 4 items • Updated Nov 11, 2024 • 2