grimjim/empatheticdialogues
Updated
Jim Lai
grimjim
AI & ML interests
Experimenting primarily with 7B-12B parameter text completion models. Not all models are intended for direct use, but aim for educational and/or merge purposes.
Recent Activity
updated
a dataset
about 17 hours ago
grimjim/empatheticdialogues
updated
a model
5 days ago
grimjim/Magnolia-v4-Gemma2-8k-9B
updated
a collection
5 days ago
Highlighted work
Organizations
grimjim's activity
updated
a
dataset
about 17 hours ago
updated
a
model
5 days ago
updated
a
collection
5 days ago
updated
a
model
5 days ago
Adding Evaluation Results
#1 opened 5 days ago
by
T145
Please submit this model to the Open LLM Leaderboard
2
#1 opened 6 days ago
by
grimjim
posted
an
update
6 days ago
Post
1393
I've arrived at an interesting result on the current Open LLM leaderboard.
open-llm-leaderboard/open_llm_leaderboard
After I narrowed down the filter of models to be between 8-9B parameters, my recent merge of o1 reasoning models achieved the highest MATH eval result of any Llama 3.x 8B model currently on the board, hitting 33.99%, placing 973/2795.
grimjim/HuatuoSkywork-o1-Llama-3.1-8B
Unfortunately, I need more information to evaluate the parent models used in the merge.
The Skywork/Skywork-o1-Open-Llama-3.1-8B model scored 0% on the MATH eval, which I suspect was due to output formatting that was baked too hard into the model, and placed 2168/2795; the merge achieved a significant uplift in every benchmark across the board.
Unfortunately, FreedomIntelligence/HuatuoGPT-o1-8B was not currently benched as of this post, so I am unable to assess relative benchmarks. Nevertheless, it is intriguing that an ostensibly medical o1 model appears to have resulted in a sizable MATH boost.
open-llm-leaderboard/open_llm_leaderboard
After I narrowed down the filter of models to be between 8-9B parameters, my recent merge of o1 reasoning models achieved the highest MATH eval result of any Llama 3.x 8B model currently on the board, hitting 33.99%, placing 973/2795.
grimjim/HuatuoSkywork-o1-Llama-3.1-8B
Unfortunately, I need more information to evaluate the parent models used in the merge.
The Skywork/Skywork-o1-Open-Llama-3.1-8B model scored 0% on the MATH eval, which I suspect was due to output formatting that was baked too hard into the model, and placed 2168/2795; the merge achieved a significant uplift in every benchmark across the board.
Unfortunately, FreedomIntelligence/HuatuoGPT-o1-8B was not currently benched as of this post, so I am unable to assess relative benchmarks. Nevertheless, it is intriguing that an ostensibly medical o1 model appears to have resulted in a sizable MATH boost.
updated
a
model
6 days ago
replied to
their
post
6 days ago
Example of a fixed model.
https://huggingface.co/grimjim/lemon07r_Gemma-2-Ataraxy-v4c-9B_fixed
updated
a
collection
7 days ago
updated
a
model
9 days ago
posted
an
update
10 days ago
Post
2616
I'm (finally) releasing a Python script that trims excess weights in Gemma2 full-weight models that bloated by ~1B parameters due to an early mergekit bug.
https://github.com/jim-plus/Gemma2-mergekit-remediation
I'd noticed something was off when merges of Gemma2 9B models ended up having ~10B parameters. The current mergekit package is fine, but there are still bloated models on HF that could stand to be fixed.
The script assumes that it will be run from the same directory as the model weights, and will trim the unnecessary lm_head.weight tensor and corresponding index entry.
https://github.com/jim-plus/Gemma2-mergekit-remediation
I'd noticed something was off when merges of Gemma2 9B models ended up having ~10B parameters. The current mergekit package is fine, but there are still bloated models on HF that could stand to be fixed.
The script assumes that it will be run from the same directory as the model weights, and will trim the unnecessary lm_head.weight tensor and corresponding index entry.
-
- 2 replies
updated
a
model
13 days ago
updated
a
collection
13 days ago
updated
a
model
14 days ago
updated
2
models
15 days ago
updated
a
model
21 days ago
updated
a
collection
25 days ago