Spaces:
Running
Running
Update README.md
Browse files
README.md
CHANGED
@@ -51,6 +51,11 @@ Processing single interactions in real-time by **Reactive Language Models** lead
|
|
51 |
> In example, for a dialog with **DeepSeek R1**, that have overally ~90k tokens, I paid for about 1.5M tokens. With **RxLM** it will cost only that ~90k tokens, so it
|
52 |
> will be about **15x cheaper**
|
53 |
|
|
|
|
|
|
|
|
|
|
|
54 |
### RxT-Alpha Open Research
|
55 |
We are currently working on **Reactive Transformer Proof-of-Concept - RxT-Alpha**, especially on the new reinforcement learning stage - **Memory Reinforcement Learning**,
|
56 |
that's required for our reactive models, between the _Supervised Fine-Tuning_ and _Reinforcement Learning from Human Feedback for reactive models (RxRLHF)_. The research
|
@@ -75,4 +80,7 @@ capacity (it will be improved in Preactor). Limits are:
|
|
75 |
We are working on complete Reactive Neural Networks development framework - [RxNN github](https://github.com/RxAI-dev/RxNN)
|
76 |
|
77 |
## Additional Research
|
78 |
-
- **Sparse Query Attention** - the most cost-effective GQA variant, even 2-3x faster for long sequences! Research in progress - [draft](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/sparse_query_attention.md)
|
|
|
|
|
|
|
|
51 |
> In example, for a dialog with **DeepSeek R1**, that have overally ~90k tokens, I paid for about 1.5M tokens. With **RxLM** it will cost only that ~90k tokens, so it
|
52 |
> will be about **15x cheaper**
|
53 |
|
54 |
+
## Reactive Transformer - drafts
|
55 |
+
- [Architecture introduction](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/ReactiveTransformer/reactive-transformer.md)
|
56 |
+
- [Supervised Training stages](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/ReactiveTransformer/supervised-training.md)
|
57 |
+
- [Reinforcement Learning stages](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/ReactiveTransformer/mrl.md)
|
58 |
+
|
59 |
### RxT-Alpha Open Research
|
60 |
We are currently working on **Reactive Transformer Proof-of-Concept - RxT-Alpha**, especially on the new reinforcement learning stage - **Memory Reinforcement Learning**,
|
61 |
that's required for our reactive models, between the _Supervised Fine-Tuning_ and _Reinforcement Learning from Human Feedback for reactive models (RxRLHF)_. The research
|
|
|
80 |
We are working on complete Reactive Neural Networks development framework - [RxNN github](https://github.com/RxAI-dev/RxNN)
|
81 |
|
82 |
## Additional Research
|
83 |
+
- **Sparse Query Attention (SQA)** - the most cost-effective GQA variant, even 2-3x faster for long sequences! Research in progress - [draft](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/sparse_query_attention.md)
|
84 |
+
- **Flex-SQA** - combination of Flex Attention and (symmetric) Sparse Query Attention, enabling 4-8x longer sliding windows
|
85 |
+
- **Flex Memory Attention/Memory Cross-Attention** - connecting spatially sparse attention with memory layers to enable very long single interactions - smaller sliding window for input sequences attends to full memory, or the opposite
|
86 |
+
- **Mixture-of-Experts for Grouped Attention** - MoE Router dynamically selects GQA/SQA groups, instead of static selection. Abandoned, because results were worse than for GQA/SQA - [more](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/moe_attention.md)
|