DeMo: Decoupled Momentum Optimization
Paper
•
2411.19870
•
Published
•
5
NousResearch
Enterprise
company
AI & ML interests
We are dedicated to advancing the field of natural language processing, in collaboration with the open-source community, through bleeding-edge research and a commitment to symbiotic development.
Recent Activity
View all activity
NousResearch's activity
bloc97
authored
a
paper
6 days ago
Update README.md
1
#2 opened about 1 month ago
by
not-lain
teknium
updated
a
model
25 days ago
hjc-puro
updated
2
models
25 days ago
emozilla
authored
a
paper
about 1 month ago
emozilla
authored
a
paper
5 months ago
Post
Memphis: Advancing language model reasoning without relying on proprietary model outputs
Memphis is a series of models which advance human-data models, offering good performance without relying on proprietary model outputs (e.g. GPT-generated datasets). I've developed a new iterative finetuning procedure to improve the reasoning ability of these models beyond what is possible using only SFT on the same data.
Currently, I've released two models: Memphis-CoT-3B, and Memphis-scribe-3B.
To create these models, I've created new datasets:
- euclaise/reddit-instruct : A dataset of instruction/QA-like data scraped from Reddit. A curated version, filtered using Lilac and neural embedding models, is available at euclaise/reddit-instruct-curated
- euclaise/TinyCoT : TinyCoT is a mtea-dataset that aggregates a variety of different human-sourced reasoning data. It is a curated version of my previous MegaCoT dataset euclaise/MegaCoT, which contains 629k responses which get cut down to 28k for TinyCoT. There's also an intermediate version euclaise/MiniCoT, which has 129k responses.
Memphis-CoT is trained on reddit-instruct, a filtered version of oasst2 sablo/oasst2_curated, and TinyCoT. Multiple iterations were performed on TinyCoT, while reddit-instruct and oasst2 were only used for the initial model.
Memphis-scribe further finetunes Memphis-CoT on more creative tasks. It was finetuned from Memphis-CoT on 18 different datasets, including datasets like euclaise/WritingPrompts_curated, lemonilia/LimaRP, and more.
To prevent catastrophic forgetting, I used weight averaging between iterations.
- euclaise/Memphis-CoT-3B
- euclaise/Memphis-scribe-3B
Memphis is a series of models which advance human-data models, offering good performance without relying on proprietary model outputs (e.g. GPT-generated datasets). I've developed a new iterative finetuning procedure to improve the reasoning ability of these models beyond what is possible using only SFT on the same data.
Currently, I've released two models: Memphis-CoT-3B, and Memphis-scribe-3B.
To create these models, I've created new datasets:
- euclaise/reddit-instruct : A dataset of instruction/QA-like data scraped from Reddit. A curated version, filtered using Lilac and neural embedding models, is available at euclaise/reddit-instruct-curated
- euclaise/TinyCoT : TinyCoT is a mtea-dataset that aggregates a variety of different human-sourced reasoning data. It is a curated version of my previous MegaCoT dataset euclaise/MegaCoT, which contains 629k responses which get cut down to 28k for TinyCoT. There's also an intermediate version euclaise/MiniCoT, which has 129k responses.
Memphis-CoT is trained on reddit-instruct, a filtered version of oasst2 sablo/oasst2_curated, and TinyCoT. Multiple iterations were performed on TinyCoT, while reddit-instruct and oasst2 were only used for the initial model.
Memphis-scribe further finetunes Memphis-CoT on more creative tasks. It was finetuned from Memphis-CoT on 18 different datasets, including datasets like euclaise/WritingPrompts_curated, lemonilia/LimaRP, and more.
To prevent catastrophic forgetting, I used weight averaging between iterations.
- euclaise/Memphis-CoT-3B
- euclaise/Memphis-scribe-3B
- 2 replies
bloc97
authored
a
paper
over 1 year ago
emozilla
authored
a
paper
over 1 year ago
bloc97
updated
3
models
over 1 year ago
teknium
updated
2
models
over 1 year ago
emozilla
updated
5
models
over 1 year ago
NousResearch/CodeLlama-7b-Instruct-hf-flash
Text Generation
•
Updated
•
18
NousResearch/CodeLlama-13b-Instruct-hf-flash
Text Generation
•
Updated
•
14
NousResearch/CodeLlama-34b-hf-flash
Text Generation
•
Updated
•
11
•
1
NousResearch/CodeLlama-13b-hf-flash
Text Generation
•
Updated
•
2.36k
•
1
NousResearch/CodeLlama-7b-hf-flash
Text Generation
•
Updated
•
3.99k
•
1