|
--- |
|
language: |
|
- pt |
|
- en |
|
license: cc |
|
tags: |
|
- text-generation-inference |
|
- transformers |
|
- mistral |
|
- mixtral |
|
- gguf |
|
- brazil |
|
- brasil |
|
- portuguese |
|
|
|
--- |
|
# BotBot Cabra Mixtral 8x7b |
|
|
|
Esse modelo é um finetune do [Mixtral 8x7b](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) com o dataset Cabra 30k. Esse modelo é optimizado para português. Ele apresenta melhoria em varios benchmarks brasileiros em comparação com o modelo base. |
|
|
|
|
|
**Conheça os nossos outros modelos: [Cabra](https://huggingface.co/collections/botbot-ai/models-6604c2069ceef04f834ba99b).** |
|
|
|
### dataset: Cabra 30k |
|
|
|
Dataset interno para finetuning. Vamos lançar em breve. |
|
|
|
### Quantização / GGUF |
|
|
|
Colocamos diversas versões (GGUF) quantanizadas no branch "quantanization". |
|
|
|
### Exemplo |
|
|
|
``` |
|
<s> [INST] who is Elon Musk? [/INST]Elon Musk é um empreendedor, inventor e capitalista americano. Ele é o fundador, CEO e CTO da SpaceX, CEO da Neuralink e fundador do The Boring Company. Musk também é o proprietário do Twitter.</s> |
|
|
|
``` |
|
|
|
## Uso |
|
O modelo é destinado, por agora, a fins de pesquisa. As áreas e tarefas de pesquisa possíveis incluem: |
|
|
|
- Pesquisa sobre modelos gerativos. |
|
- Investigação e compreensão das limitações e viéses de modelos gerativos. |
|
|
|
**Proibido para uso comercial. Somente Pesquisa.** |
|
|
|
### Evals |
|
|
|
|
|
Tasks Version Filter n-shot Metric Value Stderr |
|
assin2_rte 1.1 all 15 f1_macro 0.9095 ± 0.0041 |
|
all 15 acc 0.9097 ± 0.0041 |
|
assin2_sts 1.1 all 15 pearson 0.7763 ± 0.0068 |
|
all 15 mse 0.4610 ± N/A |
|
bluex 1.1 all 3 acc 0.6412 ± 0.0103 |
|
exam_id__UNICAMP_2021_2 3 acc 0.5882 ± 0.0397 |
|
exam_id__USP_2023 3 acc 0.7045 ± 0.0397 |
|
exam_id__UNICAMP_2020 3 acc 0.6545 ± 0.0371 |
|
exam_id__UNICAMP_2023 3 acc 0.7442 ± 0.0384 |
|
exam_id__UNICAMP_2018 3 acc 0.5926 ± 0.0386 |
|
exam_id__UNICAMP_2022 3 acc 0.6154 ± 0.0451 |
|
exam_id__USP_2019 3 acc 0.6250 ± 0.0442 |
|
exam_id__USP_2021 3 acc 0.6346 ± 0.0387 |
|
exam_id__UNICAMP_2021_1 3 acc 0.5000 ± 0.0427 |
|
exam_id__USP_2022 3 acc 0.6531 ± 0.0393 |
|
exam_id__UNICAMP_2024 3 acc 0.5556 ± 0.0428 |
|
exam_id__USP_2024 3 acc 0.8537 ± 0.0319 |
|
exam_id__USP_2018 3 acc 0.6296 ± 0.0379 |
|
exam_id__USP_2020 3 acc 0.6071 ± 0.0375 |
|
exam_id__UNICAMP_2019 3 acc 0.7000 ± 0.0374 |
|
enem 1.1 all 3 acc 0.7810 ± 0.0063 |
|
exam_id__2013 3 acc 0.7685 ± 0.0236 |
|
exam_id__2010 3 acc 0.8205 ± 0.0205 |
|
exam_id__2012 3 acc 0.8276 ± 0.0202 |
|
exam_id__2016_2 3 acc 0.7886 ± 0.0213 |
|
exam_id__2016 3 acc 0.8017 ± 0.0209 |
|
exam_id__2022 3 acc 0.6541 ± 0.0238 |
|
exam_id__2009 3 acc 0.8087 ± 0.0212 |
|
exam_id__2015 3 acc 0.8067 ± 0.0208 |
|
exam_id__2017 3 acc 0.7759 ± 0.0223 |
|
exam_id__2014 3 acc 0.7798 ± 0.0230 |
|
exam_id__2023 3 acc 0.7037 ± 0.0228 |
|
exam_id__2011 3 acc 0.8632 ± 0.0183 |
|
faquad_nli 1.1 all 15 f1_macro 0.7893 ± 0.0137 |
|
all 15 acc 0.8554 ± 0.0097 |
|
hatebr_offensive_binary 1.0 all 25 f1_macro 0.7800 ± 0.0080 |
|
all 25 acc 0.7879 ± 0.0077 |
|
oab_exams 1.5 all 3 acc 0.5549 ± 0.0061 |
|
exam_id__2014-14 3 acc 0.6375 ± 0.0311 |
|
exam_id__2010-01 3 acc 0.4471 ± 0.0312 |
|
exam_id__2016-20a 3 acc 0.5000 ± 0.0323 |
|
exam_id__2013-10 3 acc 0.5750 ± 0.0318 |
|
exam_id__2010-02 3 acc 0.5500 ± 0.0288 |
|
exam_id__2011-04 3 acc 0.5750 ± 0.0319 |
|
exam_id__2013-12 3 acc 0.6000 ± 0.0316 |
|
exam_id__2017-22 3 acc 0.6000 ± 0.0316 |
|
exam_id__2015-16 3 acc 0.5625 ± 0.0320 |
|
exam_id__2012-09 3 acc 0.4416 ± 0.0326 |
|
exam_id__2016-19 3 acc 0.5641 ± 0.0326 |
|
exam_id__2017-23 3 acc 0.5500 ± 0.0321 |
|
exam_id__2013-11 3 acc 0.5500 ± 0.0321 |
|
exam_id__2012-08 3 acc 0.5500 ± 0.0323 |
|
exam_id__2011-03 3 acc 0.5152 ± 0.0290 |
|
exam_id__2015-17 3 acc 0.6538 ± 0.0310 |
|
exam_id__2015-18 3 acc 0.6125 ± 0.0314 |
|
exam_id__2014-15 3 acc 0.6538 ± 0.0310 |
|
exam_id__2018-25 3 acc 0.5750 ± 0.0320 |
|
exam_id__2012-06a 3 acc 0.5750 ± 0.0318 |
|
exam_id__2017-24 3 acc 0.4875 ± 0.0323 |
|
exam_id__2011-05 3 acc 0.5250 ± 0.0323 |
|
exam_id__2012-06 3 acc 0.5750 ± 0.0319 |
|
exam_id__2016-21 3 acc 0.4875 ± 0.0322 |
|
exam_id__2012-07 3 acc 0.5000 ± 0.0323 |
|
exam_id__2016-20 3 acc 0.5500 ± 0.0322 |
|
exam_id__2014-13 3 acc 0.5875 ± 0.0316 |
|
portuguese_hate_speech_binary 1.0 all 25 f1_macro 0.6954 ± 0.0114 |
|
all 25 acc 0.7086 ± 0.0110 |
|
|
