Report for ProsusAI/finbert
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 2 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_75agree, split train).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.093 | Add typos | 93/1000 tested samples (9.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 3221 | Operating result , excluding one-off items , totaled EUR 9.1 mn compared to EUR 10.6 mn in continuing operations , excluding one-off items in 2004 . | Operatng resuly , exckluding one-off oitems , totlaed EUR 9.1 mn ompared to EUR 10.6 mn in continuing opefations , excluding ojne-off itens in 200R . | negative (p = 0.68) | positive (p = 0.89) |
| 3257 | Body The credit falls due February 24 , 2014 . | Body Thd credit falls due Fbruary 24 , 2014 . | neutral (p = 0.77) | negative (p = 0.95) |
| 2646 | The 3C Expo is a signature show in Dongguan , which is supported by the Dongguan Municipal Government every year , featuring computer accessories , software , communication and network products . | The 3C Expo s a singature show in Dongguan , whic his supported by the Dongguan Municipal Govrenment evwery year , featuring computer accessories , software , communication and network products . | neutral (p = 0.90) | positive (p = 0.52) |
👉Performance issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "quarter" |
Balanced Accuracy = 0.901 | — | -6.20% than global |
🔍✨Examples
For records in the dataset where `text` contains "quarter", the Balanced Accuracy is 6.2% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 480 | The loss for the third quarter of 2007 was EUR 0.3 mn smaller than the loss of the second quarter of 2007 . | positive | negative (p = 0.93) |
| 506 | Finnish power supply solutions and systems provider Efore Oyj said its net loss widened to 3.2 mln euro $ 4.2 mln for the first quarter of fiscal 2006-2007 ending October 31 , 2007 from 900,000 euro $ 1.2 mln for the same period of fiscal 2005-06 . | negative | positive (p = 0.93) |
| 507 | ADP News - Apr 22 , 2009 - Finnish business information systems developer Solteq Oyj HEL : STQ1V said today its net loss widened to EUR 189,000 USD 245,000 for the first quarter of 2009 from EUR 10,000 for the same peri | negative | positive (p = 0.93) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!