Report for yiyanghkust/finbert-tone
#40
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 7 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_75agree, split train).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.096 | Add typos | 96/1000 tested samples (9.6%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.6% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 198 | Net sales of the Vegetable Oil Business rose to EUR 10.6 mn from EUR 10.1 mn . | Nt sale of hyw Vegetable Oil Busibess rose to EUR 10.6 mn ffrom EUR 10.1 mn . | Positive (p = 1.00) | Neutral (p = 1.00) |
| 474 | In the second quarter of 2010 , Raute 's net loss narrowed to EUR 123,000 from EUR 1.5 million in the same period of 2009 . | In the second quarte rof 2010 , Raute 's net loss narroqed to EUR 23,000 from EUR 1.5 million in tge ame peiod of 2009 . | Positive (p = 0.55) | Negative (p = 0.98) |
| 1848 | According to Saarioinen 's Managing Director Ilkka M+ñkel+ñ , the food industry sector has a significant excess of production machinery . | According to aarioinen 's Managing XDirectoe Ilkka M+ñkel+ñ , the food ondustyr sector has a significant excess of production machinery . | Neutral (p = 0.53) | Negative (p = 1.00) |
👉Performance issues (6)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | text contains "said" |
Precision = 0.746 | — | -14.92% than global |
🔍✨Examples
For records in the dataset where `text` contains "said", the Precision is 14.92% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 156 | Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . | Positive | Neutral (p = 1.00) |
| 172 | However , sales returned to growth in April-June 2010 , CEO Pekka Eloholma said . | Positive | Neutral (p = 0.46) |
| 261 | With the acquisition , the company will expand its offering to North , Central and South America , it said . | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | avg_digits(text) >= 0.002 AND avg_digits(text) < 0.019 |
Balanced Accuracy = 0.729 | — | -12.38% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` >= 0.002 AND `avg_digits(text)` < 0.019, the Balanced Accuracy is 12.38% lower than the global Balanced Accuracy.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 8 | A purchase agreement for 7,200 tons of gasoline with delivery at the Hamina terminal , Finland , was signed with Neste Oil OYj at the average Platts index for this September plus eight US dollars per month . | 0.0193237 | Positive | Neutral (p = 1.00) |
| 56 | On the route between Helsinki in Finland and Tallinn in Estonia , cargo volumes increased by 36 % , while cargo volumes between Finland and Sweden fell by 9 % . | 0.01875 | Neutral | Positive (p = 1.00) |
| 280 | HELSINKI AFX - Outokumpu Technology said it has signed a 3.5 mln eur agreement with Mongolia 's Erdenet Mining Corporation for the engineering of the first HydroCopper plant to be built at the Erdenet mine site . | 0.00943396 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "period" |
Precision = 0.793 | — | -9.55% than global |
🔍✨Examples
For records in the dataset where `text` contains "period", the Precision is 9.55% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 36 | Nyrstar has also agreed to supply to Talvivaara up to 150,000 tonnes of sulphuric acid per annum for use in Talvivaara 's leaching process during the period of supply of the zinc in concentrate . | Positive | Neutral (p = 1.00) |
| 68 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | Positive | Neutral (p = 0.74) |
| 460 | 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . | Positive | Negative (p = 0.59) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_whitespace(text) < 0.211 AND avg_whitespace(text) >= 0.199 |
Precision = 0.825 | — | -6.00% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` < 0.211 AND `avg_whitespace(text)` >= 0.199, the Precision is 6.0% lower than the global Precision.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 15 | Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 . | 0.208 | Positive | Neutral (p = 1.00) |
| 252 | The stock rose for a second day on Wednesday bringing its two-day rise to GBX12 .0 or 2.0 % . | 0.204301 | Positive | Neutral (p = 0.55) |
| 253 | The stock rose for a third day on Tuesday bringing its three-day rise to GBX10 .50 or 1.8 % . | 0.204301 | Positive | Neutral (p = 0.65) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_word_length(text) >= 5.203 |
Balanced Accuracy = 0.783 | — | -5.88% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` >= 5.203, the Balanced Accuracy is 5.88% lower than the global Balanced Accuracy.| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 114 | Finnish Okmetic that manufactures and processes silicon wafers for the semiconductor and sensor industries and Norwegian solar wafer company NorSun have signed a contract under which Okmetic will supply NorSun mono silicon crystals for use in solar cell manufacturing . | 5.75 | Positive | Neutral (p = 1.00) |
| 116 | Finnish Ponsse has signed an agreement with Babcock Africa for the distribution and support of Ponsse forest machines , harvester heads , and information systems in South Africa . | 5.2069 | Positive | Neutral (p = 1.00) |
| 117 | South African Sappi will become the largest foreign forest industry company operating in Finland as a result of the acquisition Finnish M-real Corporation 's Graphic Papers Business unit . | 5.51724 | Positive | Neutral (p = 0.99) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "year" |
Precision = 0.826 | — | -5.79% than global |
🔍✨Examples
For records in the dataset where `text` contains "year", the Precision is 5.79% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 60 | The company still expects its turnover in 2010 to slightly increase from the level of 2009 , adding that `` market predictability is still too poor for trustworthy forecasts on the market development of the contract manufacturing business during the current year '' . | Positive | Negative (p = 1.00) |
| 228 | Previously , the company had guided for EBIT and sales growth of 20 pct and 10 pct respectively for this year . | Positive | Neutral (p = 0.68) |
| 288 | Last year the company raised its turnover to approximately 7 million litas EUR 2 mln , from 6.1 million litas in 2004 . | Positive | Neutral (p = 0.63) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!