rwkv-x-dev

community

AI & ML interests

hugging face space, for rwkv-x related developments, including build assets, etc Nothing in here are considered "official releases" AKA - we treat this as a giant file dump

SmerkyG

authored a paper 3 months ago

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale

Paper • 2505.03005 • Published May 5 • 35

m8than

authored a paper 5 months ago

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18 • 153

SmerkyG

authored a paper 5 months ago

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Paper • 2404.05892 • Published Apr 8, 2024 • 39

BlinkDL

authored a paper 5 months ago

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Paper • 2404.05892 • Published Apr 8, 2024 • 39

SmerkyG

authored 2 papers 5 months ago

GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression

Paper • 2407.12077 • Published Jul 16, 2024 • 57

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18 • 153

ZhangRC

authored a paper 5 months ago

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18 • 153

BlinkDL

authored a paper 5 months ago

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Paper • 2503.14456 • Published Mar 18 • 153

ZhangRC

in rwkv-x-dev/rwkv-world-3-subsample-preview 5 months ago

[bot] Conversion to Parquet

#1 opened 5 months ago by

parquet-converter

ZhangRC

updated a dataset 5 months ago

rwkv-x-dev/rwkv-world-3-subsample-preview

Viewer • Updated Mar 16 • 1M • 2 • 2

ZhangRC

published a dataset 5 months ago

rwkv-x-dev/rwkv-world-3-subsample-preview

Viewer • Updated Mar 16 • 1M • 2 • 2

BlinkDL

posted an update 8 months ago

Post

9232

RWKV-7 "Goose" 0.4B trained w/ ctx4k automatically extrapolates to ctx32k+, and perfectly solves NIAH ctx16k 🤯 100% RNN and attention-free. Only trained on the Pile. No finetuning. Replicable training runs. tested by our community: https://github.com/Jellyfish042/LongMamba

Hazzzardous

authored a paper 9 months ago

Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis

Paper • 2410.23320 • Published Oct 30, 2024 • 8

BlinkDL

posted an update 9 months ago

Post

11483

RWKV-6-world-v3 (+3.1T tokens) is our best multilingual 7B model as of now: BlinkDL/rwkv-6-world

It's 100% RNN and attention-free. MMLU 54.2% (previous world-v2.1 = 47.9%. note: without eval-boosting tricks such as annealing).

RWKV-7-world-v4 soon :)

1 reply

BlinkDL

posted an update 11 months ago

Post

5708

RWKV-7 "Goose" preview rc2 => Peak RNN architecture?😃Will try to squeeze more performance for the final release. Preview code & model: https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7