Report for yiyanghkust/finbert-tone
#25
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 9 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.090 | Add typos | 90/1000 tested samples (9.0%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.0% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 357 | ADPnews - Jul 17 , 2009 - Finland-based steel maker Rautaruukki Oyj Ruukki HEL : RTRKS said today it slipped to a net loss of EUR 184 million USD 259.7 m for the first half of 2009 from a net profit of EUR 2 | ADPnews - Jul 17 , 2009 - Finland-bawer tseel maker Rautaruujki Oyj Ruukki HEL : RTRKZ said today it slipped to a jnet loss of EUR 184 million USD 26Y9.7 m or the first half of 2009 fom a net profit of EUR 2 | Negative (p = 0.98) | Neutral (p = 0.94) |
| 781 | After the reporting period , BioTie North American licensing partner Somaxon Pharmaceuticals announced positive results with nalmefene in a pilot Phase 2 clinical trial for smoking cessation . | After the reporting pedriod , BioTie North American lifcensing partner Somaxon Pharmaceuticals announsxed posirtive results with nalmefene in a pilot Phase 2 clinical trial for smokng cessation . | Positive (p = 1.00) | Neutral (p = 1.00) |
| 2216 | Sales by Seppala diminished by 6 per cent . | Saled bt Sepopala diminished by 6 per cent . | Negative (p = 1.00) | Neutral (p = 0.99) |
👉Performance issues (8)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | text contains "said" |
Precision = 0.737 | — | -19.67% than global |
🔍✨Examples
For records in the dataset where `text` contains "said", the Precision is 19.67% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 115 | Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . | Positive | Neutral (p = 1.00) |
| 129 | However , sales returned to growth in April-June 2010 , CEO Pekka Eloholma said . | Positive | Neutral (p = 0.46) |
| 312 | Finnish telecoms software developer Tecnomen Oyj said on December 5 , 2006 it won a $ 3.3 mln ( 2.5 mln euro ) order to supply an expansion of the charging and messaging systems for the mobile and fixed networks of Brasil Telecom . | Positive | Neutral (p = 0.98) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | text contains "period" |
Precision = 0.821 | — | -10.49% than global |
🔍✨Examples
For records in the dataset where `text` contains "period", the Precision is 10.49% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 47 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | Positive | Neutral (p = 0.74) |
| 329 | 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . | Positive | Negative (p = 0.59) |
| 330 | 8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 . | Positive | Negative (p = 0.63) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "finland" |
Balanced Accuracy = 0.810 | — | -8.58% than global |
🔍✨Examples
For records in the dataset where `text` contains "finland", the Balanced Accuracy is 8.58% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 115 | Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . | Positive | Neutral (p = 1.00) |
| 324 | In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years . | Positive | Neutral (p = 1.00) |
| 425 | Citigroup , Inc ( NYSE : C ) has announced that its Global Transaction Services ( GTS ) business has been awarded a new mandate by Finland-based Pohjola Bank Group . | Positive | Neutral (p = 0.75) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_whitespace(text) >= 0.199 AND avg_whitespace(text) < 0.213 |
Precision = 0.843 | — | -8.03% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` >= 0.199 AND `avg_whitespace(text)` < 0.213, the Precision is 8.03% lower than the global Precision.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 10 | Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 . | 0.208 | Positive | Neutral (p = 1.00) |
| 204 | The stock rose for a second day on Wednesday bringing its two-day rise to GBX12 .0 or 2.0 % . | 0.204301 | Positive | Neutral (p = 0.55) |
| 205 | The stock rose for a third day on Tuesday bringing its three-day rise to GBX10 .50 or 1.8 % . | 0.204301 | Positive | Neutral (p = 0.65) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "year" |
Precision = 0.862 | — | -6.02% than global |
🔍✨Examples
For records in the dataset where `text` contains "year", the Precision is 6.02% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 230 | Last year the company raised its turnover to approximately 7 million litas EUR 2 mln , from 6.1 million litas in 2004 . | Positive | Neutral (p = 0.63) |
| 321 | LONDON MarketWatch -- Nokia nok said it 's won a five-year services contract to run Hutchison Essar 's network operations in nine locations in India . | Positive | Neutral (p = 1.00) |
| 324 | In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years . | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "eur" |
Precision = 0.865 | — | -5.68% than global |
🔍✨Examples
For records in the dataset where `text` contains "eur", the Precision is 5.68% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 21 | The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . | Positive | Neutral (p = 1.00) |
| 47 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | Positive | Neutral (p = 0.74) |
| 200 | The fair value of the investment properties totaled EUR 2,534.9 mn , up from EUR 2,455.1 mn in 2006 . | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "quarter" |
Precision = 0.866 | — | -5.57% than global |
🔍✨Examples
For records in the dataset where `text` contains "quarter", the Precision is 5.57% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 328 | In the second quarter of 2010 , the group 's pretax loss narrowed to EUR 400,000 from EUR 600,000 . | Positive | Negative (p = 0.98) |
| 329 | 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . | Positive | Negative (p = 0.59) |
| 330 | 8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 . | Positive | Negative (p = 0.63) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text_length(text) >= 122.500 AND text_length(text) < 133.500 |
Balanced Accuracy = 0.839 | — | -5.28% than global |
🔍✨Examples
For records in the dataset where `text_length(text)` >= 122.500 AND `text_length(text)` < 133.500, the Balanced Accuracy is 5.28% lower than the global Balanced Accuracy.| text | text_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 10 | Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 . | 125 | Positive | Neutral (p = 1.00) |
| 21 | The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . | 132 | Positive | Neutral (p = 1.00) |
| 319 | Bilfinger investors cheered the agreement , pushing shares up 7 % , or & euro ; 3.30 , to & euro ; 50.29 , in afternoon trade . | 127 | Positive | Neutral (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!