Hugging Face's logo

HuggingFaceM4
/
idefics-80b

Text Generation
Transformers
PyTorch
Safetensors
English
idefics
image-text-to-text
multimodal
text
image
image-to-text
text-generation-inference
Model card Files Community
23
Leyo commited on
Commit
87c2f77
•
1 Parent(s): 7fd6492

fix detailed table column names

Browse files
Files changed (1) hide show
  1. README.md +38 -38
README.md CHANGED
@@ -366,44 +366,44 @@ Besides, we also computed the classification accuracy on FairFace for both the b
366
  | IDEFICS 9B Instruct | 0 | 92.7 | 6.3 | 59.6 | 22.2 | 43.9 | 3.9 |
367
 
368
 
369
- | Model | Shots | Task | Gender | White | Black | Latino_Hispanic | East | Southeast Asian | Indian | Middle Eastern | Average |
370
- |:---------------------|--------:|:----------------|:---------|--------:|--------:|------------------:|-------:|------------------:|---------:|-----------------:|----------:|
371
- | IDEFICS 80B | 0 | FairFace Gender | Male | 95.4 | 96.1 | 94.4 | 96.3 | 97.2 | 94.8 | 94.4 | 95.5 |
372
- | | 0 | | Female | 94.8 | 97.8 | 95.8 | 95.1 | 96.5 | 97.3 | 95.7 | 96.1 |
373
- | | 0 | | Average | 95.1 | 97.0 | 95.1 | 95.7 | 96.8 | 96.1 | 95.1 | 95.8 |
374
- | | 0 | FairFace Race | Male | 73.4 | 71.4 | 87.9 | 73.2 | 52.3 | 35.6 | 63.0 | 65.2 |
375
- | | 0 | | Female | 63.8 | 73.7 | 82.7 | 68.6 | 33.6 | 42.4 | 58.2 | 60.4 |
376
- | | 0 | | Average | 68.6 | 72.6 | 85.3 | 70.9 | 42.9 | 39.0 | 60.6 | 62.8 |
377
- | | 0 | FairFace Age | Male | 53.3 | 51.9 | 50.7 | 54.7 | 48.3 | 48.4 | 52.4 | 51.4 |
378
- | | 0 | | Female | 55.0 | 47.4 | 45.3 | 53.5 | 53.8 | 48.1 | 49.9 | 50.4 |
379
- | | 0 | | Average | 54.1 | 49.7 | 48.0 | 54.1 | 51.1 | 48.2 | 51.1 | 50.9 |
380
- | IDEFICS 9B | 0 | FairFace Gender | Male | 92.9 | 94.6 | 91.5 | 94.7 | 95.2 | 91.6 | 90.2 | 93.0 |
381
- | | 0 | | Female | 94.6 | 97.5 | 95.6 | 94.1 | 96.5 | 97.5 | 96.2 | 96.0 |
382
- | | 0 | | Average | 93.7 | 96.0 | 93.6 | 94.4 | 95.8 | 94.5 | 93.2 | 94.5 |
383
- | | 0 | FairFace Race | Male | 54.2 | 55.5 | 70.0 | 41.3 | 50.3 | 50.6 | 75.0 | 56.7 |
384
- | | 0 | | Female | 44.1 | 65.3 | 67.4 | 42.1 | 29.5 | 56.0 | 73.1 | 53.9 |
385
- | | 0 | | Average | 49.1 | 60.4 | 68.7 | 41.7 | 39.9 | 53.3 | 74.0 | 55.3 |
386
- | | 0 | FairFace Age | Male | 50.1 | 45.6 | 46.8 | 44.5 | 39.2 | 42.2 | 45.4 | 44.8 |
387
- | | 0 | | Female | 48.1 | 43.9 | 41.3 | 48.9 | 46.7 | 42.9 | 45.7 | 45.4 |
388
- | | 0 | | Average | 49.1 | 44.7 | 44.1 | 46.7 | 43.0 | 42.6 | 45.6 | 45.1 |
389
- | IDEFICS 80B Instruct | 0 | FairFace Gender | Male | 94.0 | 95.9 | 92.5 | 93.0 | 96.4 | 91.9 | 92.8 | 93.8 |
390
- | | 0 | | Female | 98.2 | 98.2 | 97.6 | 96.9 | 98.5 | 98.6 | 98.2 | 98.0 |
391
- | | 0 | | Average | 96.1 | 97.0 | 95.1 | 95.0 | 97.5 | 95.2 | 95.5 | 95.9 |
392
- | | 0 | FairFace Race | Male | 85.2 | 72.9 | 93.7 | 82.7 | 43.9 | 19.2 | 38.9 | 62.4 |
393
- | | 0 | | Female | 77.5 | 80.7 | 88.8 | 79.2 | 29.0 | 28.2 | 37.9 | 60.2 |
394
- | | 0 | | Average | 81.3 | 76.8 | 91.3 | 81.0 | 36.5 | 23.7 | 38.4 | 61.3 |
395
- | | 0 | FairFace Age | Male | 51.2 | 48.3 | 47.6 | 50.6 | 45.8 | 44.1 | 46.8 | 47.8 |
396
- | | 0 | | Female | 48.1 | 44.4 | 42.0 | 49.3 | 44.9 | 42.7 | 50.6 | 46.0 |
397
- | | 0 | | Average | 49.7 | 46.4 | 44.8 | 50.0 | 45.4 | 43.4 | 48.7 | 46.9 |
398
- | IDEFICS 9B Instruct | 0 | FairFace Gender | Male | 87.1 | 89.6 | 84.6 | 87.6 | 90.3 | 85.8 | 83.3 | 86.9 |
399
- | | 0 | | Female | 98.7 | 99.6 | 98.8 | 98.7 | 99.0 | 99.8 | 99.0 | 99.1 |
400
- | | 0 | | Average | 92.9 | 94.6 | 91.7 | 93.1 | 94.6 | 92.8 | 91.1 | 93.0 |
401
- | | 0 | FairFace Race | Male | 67.1 | 62.3 | 92.5 | 77.4 | 41.1 | 24.0 | 43.5 | 58.3 |
402
- | | 0 | | Female | 61.1 | 73.0 | 87.2 | 77.9 | 22.5 | 33.1 | 42.5 | 56.7 |
403
- | | 0 | | Average | 64.1 | 67.6 | 89.8 | 77.6 | 31.8 | 28.5 | 43.0 | 57.5 |
404
- | | 0 | FairFace Age | Male | 51.4 | 44.2 | 43.4 | 45.6 | 40.1 | 40.2 | 44.1 | 44.1 |
405
- | | 0 | | Female | 51.2 | 39.3 | 38.2 | 46.7 | 44.9 | 41.1 | 43.4 | 43.6 |
406
- | | 0 | | Average | 51.3 | 41.8 | 40.8 | 46.2 | 42.5 | 40.7 | 43.7 | 43.9 |
407
 
408
  ## Other limitations
409
 
 
366
  | IDEFICS 9B Instruct | 0 | 92.7 | 6.3 | 59.6 | 22.2 | 43.9 | 3.9 |
367
 
368
 
369
+ | Model | Shots | <nobr>Task<br> | <nobr>Gender<br> | <nobr>White<br> | <nobr>Black<br> | <nobr>Latino_Hispanic<br> | <nobr>East<br> | <nobr>Southeast Asian<br> | <nobr>Indian<br> | <nobr>Middle Eastern<br> | <nobr>Average<br> |
370
+ |:---------------------|------------------:|:------------------|:-------------------|------------------:|------------------:|----------------------------:|-----------------:|----------------------------:|-------------------:|---------------------------:|--------------------:|
371
+ | IDEFICS 80B | 0 | FairFace - Gender | Male | 95.4 | 96.1 | 94.4 | 96.3 | 97.2 | 94.8 | 94.4 | 95.5 |
372
+ | | 0 | | Female | 94.8 | 97.8 | 95.8 | 95.1 | 96.5 | 97.3 | 95.7 | 96.1 |
373
+ | | 0 | | Average | 95.1 | 97.0 | 95.1 | 95.7 | 96.8 | 96.1 | 95.1 | 95.8 |
374
+ | | 0 | FairFace - Race | Male | 73.4 | 71.4 | 87.9 | 73.2 | 52.3 | 35.6 | 63.0 | 65.2 |
375
+ | | 0 | | Female | 63.8 | 73.7 | 82.7 | 68.6 | 33.6 | 42.4 | 58.2 | 60.4 |
376
+ | | 0 | | Average | 68.6 | 72.6 | 85.3 | 70.9 | 42.9 | 39.0 | 60.6 | 62.8 |
377
+ | | 0 | FairFace - Age | Male | 53.3 | 51.9 | 50.7 | 54.7 | 48.3 | 48.4 | 52.4 | 51.4 |
378
+ | | 0 | | Female | 55.0 | 47.4 | 45.3 | 53.5 | 53.8 | 48.1 | 49.9 | 50.4 |
379
+ | | 0 | | Average | 54.1 | 49.7 | 48.0 | 54.1 | 51.1 | 48.2 | 51.1 | 50.9 |
380
+ | IDEFICS 9B | 0 | FairFace - Gender | Male | 92.9 | 94.6 | 91.5 | 94.7 | 95.2 | 91.6 | 90.2 | 93.0 |
381
+ | | 0 | | Female | 94.6 | 97.5 | 95.6 | 94.1 | 96.5 | 97.5 | 96.2 | 96.0 |
382
+ | | 0 | | Average | 93.7 | 96.0 | 93.6 | 94.4 | 95.8 | 94.5 | 93.2 | 94.5 |
383
+ | | 0 | FairFace - Race | Male | 54.2 | 55.5 | 70.0 | 41.3 | 50.3 | 50.6 | 75.0 | 56.7 |
384
+ | | 0 | | Female | 44.1 | 65.3 | 67.4 | 42.1 | 29.5 | 56.0 | 73.1 | 53.9 |
385
+ | | 0 | | Average | 49.1 | 60.4 | 68.7 | 41.7 | 39.9 | 53.3 | 74.0 | 55.3 |
386
+ | | 0 | FairFace - Age | Male | 50.1 | 45.6 | 46.8 | 44.5 | 39.2 | 42.2 | 45.4 | 44.8 |
387
+ | | 0 | | Female | 48.1 | 43.9 | 41.3 | 48.9 | 46.7 | 42.9 | 45.7 | 45.4 |
388
+ | | 0 | | Average | 49.1 | 44.7 | 44.1 | 46.7 | 43.0 | 42.6 | 45.6 | 45.1 |
389
+ | IDEFICS 80B Instruct | 0 | FairFace - Gender | Male | 94.0 | 95.9 | 92.5 | 93.0 | 96.4 | 91.9 | 92.8 | 93.8 |
390
+ | | 0 | | Female | 98.2 | 98.2 | 97.6 | 96.9 | 98.5 | 98.6 | 98.2 | 98.0 |
391
+ | | 0 | | Average | 96.1 | 97.0 | 95.1 | 95.0 | 97.5 | 95.2 | 95.5 | 95.9 |
392
+ | | 0 | FairFace - Race | Male | 85.2 | 72.9 | 93.7 | 82.7 | 43.9 | 19.2 | 38.9 | 62.4 |
393
+ | | 0 | | Female | 77.5 | 80.7 | 88.8 | 79.2 | 29.0 | 28.2 | 37.9 | 60.2 |
394
+ | | 0 | | Average | 81.3 | 76.8 | 91.3 | 81.0 | 36.5 | 23.7 | 38.4 | 61.3 |
395
+ | | 0 | FairFace - Age | Male | 51.2 | 48.3 | 47.6 | 50.6 | 45.8 | 44.1 | 46.8 | 47.8 |
396
+ | | 0 | | Female | 48.1 | 44.4 | 42.0 | 49.3 | 44.9 | 42.7 | 50.6 | 46.0 |
397
+ | | 0 | | Average | 49.7 | 46.4 | 44.8 | 50.0 | 45.4 | 43.4 | 48.7 | 46.9 |
398
+ | IDEFICS 9B Instruct | 0 | FairFace - Gender | Male | 87.1 | 89.6 | 84.6 | 87.6 | 90.3 | 85.8 | 83.3 | 86.9 |
399
+ | | 0 | | Female | 98.7 | 99.6 | 98.8 | 98.7 | 99.0 | 99.8 | 99.0 | 99.1 |
400
+ | | 0 | | Average | 92.9 | 94.6 | 91.7 | 93.1 | 94.6 | 92.8 | 91.1 | 93.0 |
401
+ | | 0 | FairFace - Race | Male | 67.1 | 62.3 | 92.5 | 77.4 | 41.1 | 24.0 | 43.5 | 58.3 |
402
+ | | 0 | | Female | 61.1 | 73.0 | 87.2 | 77.9 | 22.5 | 33.1 | 42.5 | 56.7 |
403
+ | | 0 | | Average | 64.1 | 67.6 | 89.8 | 77.6 | 31.8 | 28.5 | 43.0 | 57.5 |
404
+ | | 0 | FairFace - Age | Male | 51.4 | 44.2 | 43.4 | 45.6 | 40.1 | 40.2 | 44.1 | 44.1 |
405
+ | | 0 | | Female | 51.2 | 39.3 | 38.2 | 46.7 | 44.9 | 41.1 | 43.4 | 43.6 |
406
+ | | 0 | | Average | 51.3 | 41.8 | 40.8 | 46.2 | 42.5 | 40.7 |
407
 
408
  ## Other limitations
409