sesame/csm-1b
Text-to-Speech
ā¢
Updated
ā¢
27.9k
ā¢
2.17k
Keane Moraes
lordvader31
AI & ML interests
Knowledge graphs, NLP and PKM
Recent Activity
liked
a model
5 months ago
sesame/csm-1b
liked
a model
5 months ago
canopylabs/orpheus-3b-0.1-ft
liked
a dataset
6 months ago
open-r1/OpenR1-Math-220k
Organizations
None yet
liked
2
models
5 months ago
liked
a
dataset
6 months ago
liked
a
model
6 months ago
liked
a
dataset
9 months ago
liked
a
model
11 months ago
reacted to
mlabonne's
post with š
about 1 year ago
Post
19657
Large models are surprisingly bad storytellers.
I asked 8 LLMs to "Tell me a bedtime story about bears and waffles."
Claude 3.5 Sonnet and GPT-4o gave me the worst stories: no conflict, no moral, zero creativity.
In contrast, smaller models were quite creative and wrote stories involving talking waffle trees and bears ostracized for their love of waffles.
Here you can see a comparison between Claude 3.5 Sonnet and NeuralDaredevil-8B-abliterated. They both start with a family of bears but quickly diverge in terms of personality, conflict, etc.
I mapped it to the hero's journey to have some kind of framework. Prompt engineering can definitely help here, but it's still disappointing that the larger models don't create better stories right off the bat.
Do you know why smaller models outperform the frontier models here?
I asked 8 LLMs to "Tell me a bedtime story about bears and waffles."
Claude 3.5 Sonnet and GPT-4o gave me the worst stories: no conflict, no moral, zero creativity.
In contrast, smaller models were quite creative and wrote stories involving talking waffle trees and bears ostracized for their love of waffles.
Here you can see a comparison between Claude 3.5 Sonnet and NeuralDaredevil-8B-abliterated. They both start with a family of bears but quickly diverge in terms of personality, conflict, etc.
I mapped it to the hero's journey to have some kind of framework. Prompt engineering can definitely help here, but it's still disappointing that the larger models don't create better stories right off the bat.
Do you know why smaller models outperform the frontier models here?
- 44 replies
upvoted
a
paper
about 1 year ago
liked
a
model
about 1 year ago
upvoted
a
collection
about 1 year ago
upvoted
an
article
over 1 year ago
Article
Finetune Mixtral 8x7B with AutoTrain
By
ā¢
ā¢
9
upvoted
a
collection
over 1 year ago
reacted to
gsarti's
post with ā¤ļø
over 1 year ago
Post
2158
š Today's pick in Interpretability & Analysis of LMs: Do language models plan ahead for future tokens? by W. Wu
@jxm
@lionellevine
This work aims to evaluate whether language models exhibit implicit planning during generation.
Authors propose two hypotheses that could result in planning-like behavior:āØ
- Pre-caching: the model engages in computation that is functional to future, but not current, predictions.āØ
- Breadcrumbs: Features contributing to the current prediction happen to also be the ones improving future ones.
To validate which behavior is observed in practice, authors note that off-diagonal gradients for weight matrices across the model are the ones responsible for pre-caching, and craft a variant of gradient descent (myopic descent) to remove such terms from the optimization procedure.
Using a synthetic dataset, authors demonstrate that pre-caching occurs in Transformers language models. However, for natural language settings the LM is observed to leverage breadcrumbs from previous passes even in the case of myopic training, rendering the latter hypothesis more plausible to account for model behavior.
š Paper: Do language models plan ahead for future tokens? (2404.00859)
š All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-ofc-lms-65ae3339949c5675d25de2f9
This work aims to evaluate whether language models exhibit implicit planning during generation.
Authors propose two hypotheses that could result in planning-like behavior:āØ
- Pre-caching: the model engages in computation that is functional to future, but not current, predictions.āØ
- Breadcrumbs: Features contributing to the current prediction happen to also be the ones improving future ones.
To validate which behavior is observed in practice, authors note that off-diagonal gradients for weight matrices across the model are the ones responsible for pre-caching, and craft a variant of gradient descent (myopic descent) to remove such terms from the optimization procedure.
Using a synthetic dataset, authors demonstrate that pre-caching occurs in Transformers language models. However, for natural language settings the LM is observed to leverage breadcrumbs from previous passes even in the case of myopic training, rendering the latter hypothesis more plausible to account for model behavior.
š Paper: Do language models plan ahead for future tokens? (2404.00859)
š All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-ofc-lms-65ae3339949c5675d25de2f9
updated
3
models
over 1 year ago
upvoted
a
collection
over 1 year ago
upvoted
2
papers
over 1 year ago
liked
a
model
over 1 year ago