Coloss/hacking-v0.4-sft
Viewer
โข
Updated
โข
65.4k
โข
12
โข
3
Coloss
community
AI & ML interests
Ingegno c'era nell'allenare congegni, tailored SLMs
Coloss's activity
efedericiย
updated
a
dataset
5 months ago
mferrarettoย
updated
a
dataset
5 months ago
Post
1695
We https://mii-llm.ai just released a new LLM Italian benchmark and a set of evaluation: MMLU-PRO-ITA
Thanks to @efederici who released efederici/MMLU-Pro-ita a machine translated version of MMLU-PRO and thanks to a community shared computational effort we published in the "Eval Aggiuntive" tab of https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard the results on Italian open source LLMs.
If you want to deepen read the blog article on hf https://huggingface.co/blog/giux78/mmlu-pro-ita
Thanks to @efederici who released efederici/MMLU-Pro-ita a machine translated version of MMLU-PRO and thanks to a community shared computational effort we published in the "Eval Aggiuntive" tab of https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard the results on Italian open source LLMs.
If you want to deepen read the blog article on hf https://huggingface.co/blog/giux78/mmlu-pro-ita
Post
1496
@FinancialSupport
and I just released a new version of the Italian LLMs leaderboard https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard
using the super useful https://huggingface.co/demo-leaderboard template from @clefourrier .
Weโve evaluated over 50 models (base, merged, fine-tuned, etc.) from:
- Major companies like Meta, Mistral, Google ...
- University groups such as https://huggingface.co/sapienzanlp or https://huggingface.co/swap-uniba
- Italian Companies like https://huggingface.co/MoxoffSpA , https://huggingface.co/FairMind or https://huggingface.co/raicrits
- Various communities and individuals
All models were tested on #Italian benchmarks #mmlu #arc-c #hellaswag, which we contributed to the opensource lm-evaluation-harness library from https://huggingface.co/EleutherAI.
Plus, you can now submit your model for automatic evaluation, thanks to to https://huggingface.co/seeweb sponsored computation.
Curious about the top Italian models? Check out the leaderboard and submit your model!
https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard
using the super useful https://huggingface.co/demo-leaderboard template from @clefourrier .
Weโve evaluated over 50 models (base, merged, fine-tuned, etc.) from:
- Major companies like Meta, Mistral, Google ...
- University groups such as https://huggingface.co/sapienzanlp or https://huggingface.co/swap-uniba
- Italian Companies like https://huggingface.co/MoxoffSpA , https://huggingface.co/FairMind or https://huggingface.co/raicrits
- Various communities and individuals
All models were tested on #Italian benchmarks #mmlu #arc-c #hellaswag, which we contributed to the opensource lm-evaluation-harness library from https://huggingface.co/EleutherAI.
Plus, you can now submit your model for automatic evaluation, thanks to to https://huggingface.co/seeweb sponsored computation.
Curious about the top Italian models? Check out the leaderboard and submit your model!
https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard
Post
1769
Finally, I can post! ๐
I created a Capybara-inspired Italian dataset by translating the initial instruction and running it through a pipeline to generate conversations. I used Claude Sonnet for translation and instruction generation, and Opus for generating the answers.
I hope this dataset proves useful for people working on ๐ฎ๐น language models.
โ Open sourcing the dataset here: efederici/capybara-claude-15k-ita
I created a Capybara-inspired Italian dataset by translating the initial instruction and running it through a pipeline to generate conversations. I used Claude Sonnet for translation and instruction generation, and Opus for generating the answers.
I hope this dataset proves useful for people working on ๐ฎ๐น language models.
โ Open sourcing the dataset here: efederici/capybara-claude-15k-ita
- 1 reply
Post
1587
@mik3ml
just released
ReDiX/wikipediaQA-ita an interesting synthetic dataset originated from wikipedia using a fine tuned version of mistral-7B specific for the Italian language ๐ฎ๐น .
- 1 reply
Post
1814
๐ Super
@DeepMount00
just released ๐๐ฒ๐บ๐บ๐ฎ_๐ค๐_๐๐ง๐_๐๐ฏ ๐น๐ฒ๐ฎ๐ฑ๐ถ๐ป๐ด the ๐ฅ๐๐ ๐๐ฎ๐๐ธ on the Italian ๐๐๐ _๐๐ง๐_๐๐๐๐๐๐ฅ๐๐ข๐๐ฅ๐. The model is a fine tuned version of Gemma 2B.
Model details: https://huggingface.co/DeepMount00/Gemma_QA_ITA_v3
Explore the full RAG section rankings here: https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard on section Classifica RAG
Model details: https://huggingface.co/DeepMount00/Gemma_QA_ITA_v3
Explore the full RAG section rankings here: https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard on section Classifica RAG
Post
1781
On evaluating fine tuned 7B Italian open source LLMs I have collected many data points and I created a super simple explorative analyses. My hypothesis based on data are:
- mmlu is hard to improve when fine tuning a base model on a different language
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning can specialize well but at cost of loosing some foundational knowledge.
Here the data https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
Here the colab https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
Here an article with some considerations https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1
- mmlu is hard to improve when fine tuning a base model on a different language
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning can specialize well but at cost of loosing some foundational knowledge.
Here the data https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
Here the colab https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
Here an article with some considerations https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1
Post
1287
Based on the work of
@mrinaldi
and
@ruggsea
we just released the biggest - ready for training - conversational dataset based on Usenet data in the Italian language ๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น. It contains about 9 millions of conversations made by real humans.
mii-community/UsenetArchiveIT-conversations
mii-community/UsenetArchiveIT-conversations
Post
2664
I and
@FinancialSupport
just release the new Italian Leaderboard:
https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard
It is based on lm-evaluation-harness and at the moment , mainly, on 7 billion models. In the next weeks we will add more models. If you have suggestion or need explanations join our community discord https://discord.gg/a26cRkBCNH
https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard
It is based on lm-evaluation-harness and at the moment , mainly, on 7 billion models. In the next weeks we will add more models. If you have suggestion or need explanations join our community discord https://discord.gg/a26cRkBCNH
- 9 replies
Post
Wonderful open source Italian dataset from
@manalog
and
@ruggsea
:
https://huggingface.co/datasets/manalog/UsenetArchiveIT
The dataset contributes to the https://huggingface.co/mii-community project, aimed at advancing the creation of Italian open-source Language Models (LLMs).๐ฎ๐น ๐ค About 10-20 billion token, probably the best conversational open source dataset in the Italian language. ๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น
https://huggingface.co/datasets/manalog/UsenetArchiveIT
The dataset contributes to the https://huggingface.co/mii-community project, aimed at advancing the creation of Italian open-source Language Models (LLMs).๐ฎ๐น ๐ค About 10-20 billion token, probably the best conversational open source dataset in the Italian language. ๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น๐ฎ๐น
- 2 replies
Post
Super work from
@DeepMount00
:
๐ ๐๐ข๐ฌ๐๐จ๐ฏ๐๐ซ ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ: ๐ ๐๐ฅ๐ข๐๐๐ซ-๐๐๐ฌ๐๐ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐
Introducing ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐ง๐ ๐ฎ๐๐ ๐, a revolutionary Named Entity Recognition (NER) model evolved from the GliNer architecture and meticulously tailored for the Italian language. This advanced model is a beacon of efficiency and versatility, engineered to ๐ซ๐๐๐จ๐ ๐ง๐ข๐ณ๐ ๐๐ง๐ฒ ๐๐ง๐ญ๐ข๐ญ๐ฒ ๐ญ๐ฒ๐ฉ๐ within the rich nuances of Italian, using a bidirectional transformer encoder. It stands out as an ideal solution for those navigating the challenges of resource-limited environments or seeking an efficient alternative to the cumbersome Large Language Models (LLMs).
๐๐ฎ๐ง๐ฌ ๐๐๐ฌ๐ญ ๐๐ฅ๐ฌ๐จ ๐จ๐ง ๐๐๐!
Experience this Italian-focused innovation live on Hugging Face Spaces:
DeepMount00/universal_ner_ita
Paper: https://arxiv.org/abs/2311.08526 Urchade Zaratiana et all. great work!
๐ ๐๐ข๐ฌ๐๐จ๐ฏ๐๐ซ ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ: ๐ ๐๐ฅ๐ข๐๐๐ซ-๐๐๐ฌ๐๐ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐
Introducing ๐๐ง๐ข๐ฏ๐๐ซ๐ฌ๐๐ฅ ๐๐๐ซ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐๐ง๐ ๐ฎ๐๐ ๐, a revolutionary Named Entity Recognition (NER) model evolved from the GliNer architecture and meticulously tailored for the Italian language. This advanced model is a beacon of efficiency and versatility, engineered to ๐ซ๐๐๐จ๐ ๐ง๐ข๐ณ๐ ๐๐ง๐ฒ ๐๐ง๐ญ๐ข๐ญ๐ฒ ๐ญ๐ฒ๐ฉ๐ within the rich nuances of Italian, using a bidirectional transformer encoder. It stands out as an ideal solution for those navigating the challenges of resource-limited environments or seeking an efficient alternative to the cumbersome Large Language Models (LLMs).
๐๐ฎ๐ง๐ฌ ๐๐๐ฌ๐ญ ๐๐ฅ๐ฌ๐จ ๐จ๐ง ๐๐๐!
Experience this Italian-focused innovation live on Hugging Face Spaces:
DeepMount00/universal_ner_ita
Paper: https://arxiv.org/abs/2311.08526 Urchade Zaratiana et all. great work!
-
- 3 replies
Post
The dataset of all Italian laws just released
mii-llm/gazzetta-ufficiale
Kudos to @efederici @mferraretto @z-uo
Kudos to @efederici @mferraretto @z-uo