Report for soleimanian/financial-roberta-large-sentiment
#31
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 5 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_66agree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.116 | Transform to uppercase | 116/1000 tested samples (11.6%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 11.6% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1296 | In the next few years , the ICT sector 's share of electricity consumption will be raised by the increase in the popularity of smartphones . | IN THE NEXT FEW YEARS , THE ICT SECTOR 'S SHARE OF ELECTRICITY CONSUMPTION WILL BE RAISED BY THE INCREASE IN THE POPULARITY OF SMARTPHONES . | negative (p = 0.53) | positive (p = 1.00) |
| 2658 | Salcomp Manufacturing Oy will pay EUR 35 million in cash to Salcomp in connection with the implementation of the sale and transfer of the Business . | SALCOMP MANUFACTURING OY WILL PAY EUR 35 MILLION IN CASH TO SALCOMP IN CONNECTION WITH THE IMPLEMENTATION OF THE SALE AND TRANSFER OF THE BUSINESS . | positive (p = 0.95) | neutral (p = 0.96) |
| 2026 | Besides , Pit-Produkt will enter the new segments . | BESIDES , PIT-PRODUKT WILL ENTER THE NEW SEGMENTS . | positive (p = 0.94) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.106 | Add typos | 106/1000 tested samples (10.6%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 10.6% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1117 | RAPALA TOURNAMENT FISHING : With all the major titles already out for the holidays , bargain-priced games such as Rapala aim for a smaller audience . | RAPALA TOURNAMENT FISHING : With all the major titles already out for the holidzys , bargainpriced games such as Rapalx aim for a smaller audinece . | negative (p = 0.94) | neutral (p = 1.00) |
| 1741 | Cymed 's net sales are expected to amount to EUR 3.5 mn in 2006 . | Cyme 's net sales are ecxpected to amoun to EUR 3.5 mn in 2006 . | neutral (p = 0.94) | positive (p = 1.00) |
| 569 | ( ADP News ) - Feb 9 , 2009 - Finnish computer services company Proha Oyj ( HEL : ART1V ) said today its net loss narrowed to EUR 113,000 ( USD 146,000 ) for 2008 from EUR 1.2 million for 2007 . | ( ADP News ) - Feb 9 , 2009 - Finnish computer services company Proha Oyj ( HSL : ARTWV ) waid today its net loss narrowed to EUR 113,000 ( USD 146,000 ) for 2080 from EUR 1.3 milplion for 2007 . | positive (p = 1.00) | negative (p = 0.98) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.067 | Transform to title case | 67/1000 tested samples (6.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 6.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 852 | Sales VAT inclusive expanded by 19 percent , to 351 million euros . | Sales Vat Inclusive Expanded By 19 Percent , To 351 Million Euros . | negative (p = 1.00) | positive (p = 1.00) |
| 1296 | In the next few years , the ICT sector 's share of electricity consumption will be raised by the increase in the popularity of smartphones . | In The Next Few Years , The Ict Sector 'S Share Of Electricity Consumption Will Be Raised By The Increase In The Popularity Of Smartphones . | negative (p = 0.53) | positive (p = 0.94) |
| 3609 | `` We have the most expensive water brand in Finland at the moment . '' | `` We Have The Most Expensive Water Brand In Finland At The Moment . '' | neutral (p = 0.83) | negative (p = 0.99) |
👉Performance issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "said" |
Balanced Accuracy = 0.814 | — | -6.91% than global |
🔍✨Examples
For records in the dataset where `text` contains "said", the Balanced Accuracy is 6.91% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 1 | Technopolis plans to develop in stages an area of no less than 100,000 square meters in order to host companies working in computer technologies and telecommunications , the statement said . | neutral | positive (p = 0.99) |
| 371 | Efore 's CEO Reijo Maihaniemi said that the company has developed and already delivered power supply units to respective systems based on this technology to Chinese sector company ZTE Corp SHE : 000063 . | neutral | positive (p = 1.00) |
| 372 | Finnish Talvivaara Mining Co HEL : TLV1V said Thursday it had picked BofA Merrill Lynch and JPMorgan NYSE : JPM as joint bookrunners of its planned issue of convertible notes worth up to EUR250m USD332m . | neutral | positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "business" |
Precision = 0.763 | — | -6.55% than global |
🔍✨Examples
For records in the dataset where `text` contains "business", the Precision is 6.55% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 74 | The company still expects its turnover in 2010 to slightly increase from the level of 2009 , adding that `` market predictability is still too poor for trustworthy forecasts on the market development of the contract manufacturing business during the current year '' . | positive | negative (p = 1.00) |
| 506 | `` These tests are part of a larger campaign which includes various customer trials and demonstrations to make LTE on 800 MHz commercially viable by this summer , '' Nokia Siemens head of LTE business line , Reino Tammela , said . | neutral | positive (p = 0.96) |
| 617 | Turnover surged to EUR61 .8 m from EUR47 .6 m due to increasing service demand , especially in the third quarter , and the overall growth of its business . | positive | negative (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!