Report for soleimanian/financial-roberta-large-sentiment
#36
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 5 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_50agree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.107 | Transform to uppercase | 107/1000 tested samples (10.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 10.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2223 | The Finnish textiles and clothing company Marimekko Corporation ( OMX Helsinki : MMO1V ) reported on Wednesday ( 5 November ) an operating profit of EUR8 .1 m on net sales of EUR59m for the period from January to September 2008 . | THE FINNISH TEXTILES AND CLOTHING COMPANY MARIMEKKO CORPORATION ( OMX HELSINKI : MMO1V ) REPORTED ON WEDNESDAY ( 5 NOVEMBER ) AN OPERATING PROFIT OF EUR8 .1 M ON NET SALES OF EUR59M FOR THE PERIOD FROM JANUARY TO SEPTEMBER 2008 . | neutral (p = 0.93) | positive (p = 1.00) |
| 2310 | The fund at fair value will increase correspondingly . | THE FUND AT FAIR VALUE WILL INCREASE CORRESPONDINGLY . | neutral (p = 0.50) | positive (p = 1.00) |
| 637 | AGJ recorded EUR 43 mln sales in 2006 , most of which was generated by exports to customers in Western Europe , the statement said . | AGJ RECORDED EUR 43 MLN SALES IN 2006 , MOST OF WHICH WAS GENERATED BY EXPORTS TO CUSTOMERS IN WESTERN EUROPE , THE STATEMENT SAID . | neutral (p = 1.00) | positive (p = 0.94) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.099 | Add typos | 99/1000 tested samples (9.9%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.9% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2189 | Solvay S.A. has engaged Poyry to provide project management , engineering , procurement , and site services for a hydrogen peroxide production plant to be built by a Solvay-BASF joint venture at BASF 's Zandvliet site , Belgium . | Solvay .A. has engaged Poyry to provide project management , engineering , procurement , and sute services for a hydrogen peroxide production plant to be buikt by a Solvay-BAZF jont venture at BASF 's Zanrvliet site , Belgium .. | neutral (p = 0.81) | positive (p = 0.95) |
| 4068 | Operating result showed a loss of EUR 2.9 mn , while a year before , it showed a profit of EUR 0.6 mn . | Opderating resjlt showed a lsos of EUR 2.9 mn , while a year before , it showed a profit of EUR 0.6 mn . | negative (p = 1.00) | positive (p = 0.99) |
| 499 | Mr Ashley , deputy executive chairman of Sports Direct , sold a 43pc stake in the company for more than pounds 900m at the time of the float . | Mr Ashpey , deputy executive chairman of Sports Direct , sold a 43pf stske ihn the compamy for more than pounds 900m at the time of the float . | negative (p = 0.91) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.053 | Transform to title case | 53/1000 tested samples (5.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 5.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2310 | The fund at fair value will increase correspondingly . | The Fund At Fair Value Will Increase Correspondingly . | neutral (p = 0.50) | positive (p = 1.00) |
| 4392 | Copper , lead and nickel also dropped ... HBOS ( HBOS ) plummeted 20 % to 70.3 pence after saying this year+ó ?? | Copper , Lead And Nickel Also Dropped ... Hbos ( Hbos ) Plummeted 20 % To 70.3 Pence After Saying This Year+Ó ?? | negative (p = 1.00) | positive (p = 0.98) |
| 1873 | The winners included the Honda Odyssey for minivan and the Nissan Armada for large SUV . | The Winners Included The Honda Odyssey For Minivan And The Nissan Armada For Large Suv . | neutral (p = 0.59) | positive (p = 1.00) |
👉Performance issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_digits(text) < 0.031 AND avg_digits(text) >= 0.014 |
Precision = 0.722 | — | -7.33% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.031 AND `avg_digits(text)` >= 0.014, the Precision is 7.33% lower than the global Precision.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 59 | In Sweden , Gallerix accumulated SEK denominated sales were down 1 % and EUR denominated sales were up 11 % . | 0.0275229 | neutral | negative (p = 0.82) |
| 64 | In June it sold a 30 percent stake to Nordstjernan , and the investment group has now taken up the option to acquire EQT 's remaining shares . | 0.0140845 | neutral | positive (p = 0.99) |
| 75 | On the route between Helsinki in Finland and Tallinn in Estonia , cargo volumes increased by 36 % , while cargo volumes between Finland and Sweden fell by 9 % . | 0.01875 | neutral | positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text_length(text) >= 149.500 AND text_length(text) < 161.500 |
Precision = 0.731 | — | -6.06% than global |
🔍✨Examples
For records in the dataset where `text_length(text)` >= 149.500 AND `text_length(text)` < 161.500, the Precision is 6.06% lower than the global Precision.| text | text_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 60 | The company supports its global customers in developing new technologies and offers a fast route from product development to applications and volume production . | 161 | neutral | positive (p = 1.00) |
| 75 | On the route between Helsinki in Finland and Tallinn in Estonia , cargo volumes increased by 36 % , while cargo volumes between Finland and Sweden fell by 9 % . | 160 | neutral | positive (p = 1.00) |
| 409 | To our members and partners , the use of IT will mostly be apparent in the increased efficiency of the results service , '' observes Perttu Puro from Tradeka . | 159 | positive | neutral (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!