udkai_Turdus

A less contaminated version of udkai/Garrulus and the second model to be discussed in the paper Subtle DPO-Contamination with modified Winogrande increases TruthfulQA, Hellaswag & ARC.

Contrary to Garrulus which was obtained after 2 epochs, this model was obtained after one single epoch of "direct preference optimization" of NeuralMarcoro14-7B with [https://huggingface.co/datasets/hromi/winograd_dpo ] .

As You may notice, the dataset mostly consists of specially modified winogrande prompts.

But before flagging this (or recommending this to be flagged), consider this:

Subtle DPO-Contamination with modified Winogrande causes the average accuracy of all 5-non Winogrande metrics (e.g. including also MMLU and GSM8K) to be 0.2% higher than the underlying model.

Model ARC HellaSwag MMLU Truthful QA GSM8K Average
mlabonne/NeuralMarcoro14-7B 71.42 87.59 64.84 65.64 70.74 72.046
udkai/Turdus 73.38 88.56 64.52 67.11 67.7 72,254

Yes, as strange as it may sound, one can indeed increase ARC from 71.42% to 73.38 % with one single epoch of cca 1200 repetitive winograd schematas...

BibTex

Should this model - or quasi-methodology which lead to it - be of certain pratical or theoretical interest for You, would be honored if You would refer to it in Your work:

@misc {udk_dot_ai_turdus,
    author       = { {UDK dot AI, Daniel Devatman Hromada} },
    title        = { Turdus (Revision 923c305) },
    year         = 2024,
    url          = { https://huggingface.co/udkai/Turdus },
    doi          = { 10.57967/hf/1611 },
    publisher    = { Hugging Face }
}
Downloads last month
60
Safetensors
Model size
7.24B params
Tensor type
BF16
Β·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for udkai/Turdus

Finetuned
(2)
this model
Adapters
1 model
Finetunes
1 model
Merges
49 models

Spaces using udkai/Turdus 15