Report for mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis
#27
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 5 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_66agree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.334 | Transform to uppercase | 334/1000 tested samples (33.4%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 33.4% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1573 | `` P&O Ferries now has a very efficient and powerful vessel for its Dover to Calais route , '' head of the shipbuilder 's Rauma yard , Timo Suistio , said . | `` P&O FERRIES NOW HAS A VERY EFFICIENT AND POWERFUL VESSEL FOR ITS DOVER TO CALAIS ROUTE , '' HEAD OF THE SHIPBUILDER 'S RAUMA YARD , TIMO SUISTIO , SAID . | positive (p = 1.00) | neutral (p = 1.00) |
| 256 | Revenue grew 12 percent to ( x20ac ) 3.6 billion ( US$ 4.5 billion ) . | REVENUE GREW 12 PERCENT TO ( X20AC ) 3.6 BILLION ( US$ 4.5 BILLION ) . | positive (p = 1.00) | neutral (p = 1.00) |
| 3959 | `` Beyond the improved voice capabilities , customers now have a streamlined way to comply with recalls and other traceability requirements , providing them with a competitive advantage . | `` BEYOND THE IMPROVED VOICE CAPABILITIES , CUSTOMERS NOW HAVE A STREAMLINED WAY TO COMPLY WITH RECALLS AND OTHER TRACEABILITY REQUIREMENTS , PROVIDING THEM WITH A COMPETITIVE ADVANTAGE . | positive (p = 1.00) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.094 | Transform to title case | 94/1000 tested samples (9.4%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 9.4% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1461 | The GeoSolutions technology will leverage Benefon 's GPS solutions by providing Location Based Search Technology , a Communities Platform , location relevant multimedia content and a new and powerful commercial model . | The Geosolutions Technology Will Leverage Benefon 'S Gps Solutions By Providing Location Based Search Technology , A Communities Platform , Location Relevant Multimedia Content And A New And Powerful Commercial Model . | neutral (p = 0.98) | positive (p = 0.99) |
| 578 | The diluted loss per share narrowed to EUR 0.27 from EUR 0.86 . | The Diluted Loss Per Share Narrowed To Eur 0.27 From Eur 0.86 . | positive (p = 1.00) | negative (p = 1.00) |
| 3015 | 18 March 2010 A leakage in the gypsum pond was detected today at the Talvivaara mine . | 18 March 2010 A Leakage In The Gypsum Pond Was Detected Today At The Talvivaara Mine . | negative (p = 0.86) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.087 | Add typos | 87/1000 tested samples (8.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 8.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 4170 | Finnish Suominen Corporation that makes wipes , nonwovens , and flexible packaging , expects changes in the market situation to reduce sales of Suominen 's nonwovens and wet wipes from the previously estimated volumes . | Finniah Suominen Corporation yhat makes wipdas , nonwoevns , and flexible packaging , ecpects changes in the markwet situation to rexduce sakes kf Xuominen 's nonwovens xand wet wipes from the previouslty estimated volumds . | negative (p = 1.00) | neutral (p = 1.00) |
| 3089 | Earnings per share EPS in 2005 amounted to a loss of EUR1 .26 . | aErnings prer share EPS in 20005 amounted to a kkosx of EUR1 .26 . | negative (p = 1.00) | neutral (p = 1.00) |
| 1387 | The pipeline is strong . | The pipeine is strong . | positive (p = 1.00) | neutral (p = 0.89) |
👉Performance issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_digits(text) < 0.006 |
Balanced Accuracy = 0.811 | — | -7.79% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.006, the Balanced Accuracy is 7.79% lower than the global Balanced Accuracy.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 43 | The agreement was signed with Biohit Healthcare Ltd , the UK-based subsidiary of Biohit Oyj , a Finnish public company which develops , manufactures and markets liquid handling products and diagnostic test systems . | 0 | positive | neutral (p = 1.00) |
| 56 | The company supports its global customers in developing new technologies and offers a fast route from product development to applications and volume production . | 0 | neutral | positive (p = 1.00) |
| 66 | Other measures included increasing synergies and economies of scale within the Grimaldi Group and personnel adjustments , divestments and redelivery of excess tonnage . | 0 | neutral | positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_word_length(text) >= 4.848 AND avg_word_length(text) < 5.291 |
Balanced Accuracy = 0.824 | — | -6.25% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` >= 4.848 AND `avg_word_length(text)` < 5.291, the Balanced Accuracy is 6.25% lower than the global Balanced Accuracy.| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 79 | `` They would invest not only in the physical infrastructure , but would also provide know-how for managing and developing science and technology parks , '' said Sunrise Valley director Andrius Bagdonas . | 5.21212 | positive | neutral (p = 1.00) |
| 138 | After the takeover , Cramo will become the second largest rental services provider in the Latvian market . | 4.94444 | positive | neutral (p = 1.00) |
| 286 | This new representation extends Comptel 's global presence to a total of 18 countries , serving over 250 customers in over 80 countries worldwide . | 4.92 | positive | neutral (p = 0.98) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!