AdamF92 commited on
Commit
a8eadf7
·
verified ·
1 Parent(s): 87bc015

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -1
README.md CHANGED
@@ -51,6 +51,11 @@ Processing single interactions in real-time by **Reactive Language Models** lead
51
  > In example, for a dialog with **DeepSeek R1**, that have overally ~90k tokens, I paid for about 1.5M tokens. With **RxLM** it will cost only that ~90k tokens, so it
52
  > will be about **15x cheaper**
53
 
 
 
 
 
 
54
  ### RxT-Alpha Open Research
55
  We are currently working on **Reactive Transformer Proof-of-Concept - RxT-Alpha**, especially on the new reinforcement learning stage - **Memory Reinforcement Learning**,
56
  that's required for our reactive models, between the _Supervised Fine-Tuning_ and _Reinforcement Learning from Human Feedback for reactive models (RxRLHF)_. The research
@@ -75,4 +80,7 @@ capacity (it will be improved in Preactor). Limits are:
75
  We are working on complete Reactive Neural Networks development framework - [RxNN github](https://github.com/RxAI-dev/RxNN)
76
 
77
  ## Additional Research
78
- - **Sparse Query Attention** - the most cost-effective GQA variant, even 2-3x faster for long sequences! Research in progress - [draft](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/sparse_query_attention.md)
 
 
 
 
51
  > In example, for a dialog with **DeepSeek R1**, that have overally ~90k tokens, I paid for about 1.5M tokens. With **RxLM** it will cost only that ~90k tokens, so it
52
  > will be about **15x cheaper**
53
 
54
+ ## Reactive Transformer - drafts
55
+ - [Architecture introduction](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/ReactiveTransformer/reactive-transformer.md)
56
+ - [Supervised Training stages](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/ReactiveTransformer/supervised-training.md)
57
+ - [Reinforcement Learning stages](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/ReactiveTransformer/mrl.md)
58
+
59
  ### RxT-Alpha Open Research
60
  We are currently working on **Reactive Transformer Proof-of-Concept - RxT-Alpha**, especially on the new reinforcement learning stage - **Memory Reinforcement Learning**,
61
  that's required for our reactive models, between the _Supervised Fine-Tuning_ and _Reinforcement Learning from Human Feedback for reactive models (RxRLHF)_. The research
 
80
  We are working on complete Reactive Neural Networks development framework - [RxNN github](https://github.com/RxAI-dev/RxNN)
81
 
82
  ## Additional Research
83
+ - **Sparse Query Attention (SQA)** - the most cost-effective GQA variant, even 2-3x faster for long sequences! Research in progress - [draft](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/sparse_query_attention.md)
84
+ - **Flex-SQA** - combination of Flex Attention and (symmetric) Sparse Query Attention, enabling 4-8x longer sliding windows
85
+ - **Flex Memory Attention/Memory Cross-Attention** - connecting spatially sparse attention with memory layers to enable very long single interactions - smaller sliding window for input sequences attends to full memory, or the opposite
86
+ - **Mixture-of-Experts for Grouped Attention** - MoE Router dynamically selects GQA/SQA groups, instead of static selection. Abandoned, because results were worse than for GQA/SQA - [more](https://github.com/RxAI-dev/RxNN/blob/main/docs/research/moe_attention.md)