Report for yiyanghkust/finbert-tone
#35
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 7 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_50agree, split train).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.113 | Add typos | 113/1000 tested samples (11.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 11.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 297 | The Helsinki-based company , which also owns the Salomon , Atomic and Suunto brands , said net profit rose 15 percent in the three months through Dec. 31 to ( x20ac ) 47 million ( $ 61US million ) , from ( x20ac ) 40.8 million a year earlier . | The Helsinki-based company , which lsio owns the Salomon , Atomic and Suunto brands , said et rofit rose 15 percent in the three months through Dec. 31 to ( x20ac ) 47 million ( $ 61UDS million ) , from ( x20ac ) 40.8 million a yea earlier . | Positive (p = 1.00) | Neutral (p = 0.98) |
| 1524 | Finnish retailer Stockmann has won approval from the board of Swedish rival Lindex for a public tender offer with the aim of expanding the companies ' presence in Russia and other CEE countries , Stockmann said Monday . | Finnish retailer Stokmann has won approval ftrom tnhe aord f Swedisbh rival Lindex fir a public tender offer with teg aim of expanding the companies ' perzsence in Russia and other CEE coyntries , Stockmann said Mlondwy . | Positive (p = 1.00) | Neutral (p = 0.94) |
| 500 | One of the challenges in the oil production in the North Sea is scale formation that can plug pipelines and halt production . | One of the cijallenges in he oil prkoduction in the North Sea is scale formation that can plug pipelines and halt production . | Negative (p = 1.00) | Neutral (p = 1.00) |
👉Performance issues (6)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_word_length(text) >= 4.947 |
Balanced Accuracy = 0.660 | — | -9.11% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` >= 4.947, the Balanced Accuracy is 9.11% lower than the global Balanced Accuracy.| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 2 | The international electronic industry company Elcoteq has laid off tens of employees from its Tallinn facility ; contrary to earlier layoffs the company contracted the ranks of its office workers , the daily Postimees reported . | 5.36111 | Negative | Neutral (p = 1.00) |
| 18 | Incap Contract Manufacturing Services Pvt Ltd , a subsidiary of Incap Corporation of Finland , plans to double its revenues by 2007-2008 . | 5.04348 | Positive | Neutral (p = 0.70) |
| 30 | The Brazilian unit of Finnish security solutions provider F-Secure signed up 1,500 new clients last year , online news source Reseller Web quoted the division 's commercial director , Vladimir Brand+úo , as saying . | 5.17143 | Positive | Neutral (p = 0.99) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_whitespace(text) < 0.166 AND avg_whitespace(text) >= 0.158 |
Balanced Accuracy = 0.666 | — | -8.27% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` < 0.166 AND `avg_whitespace(text)` >= 0.158, the Balanced Accuracy is 8.27% lower than the global Balanced Accuracy.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 18 | Incap Contract Manufacturing Services Pvt Ltd , a subsidiary of Incap Corporation of Finland , plans to double its revenues by 2007-2008 . | 0.15942 | Positive | Neutral (p = 0.70) |
| 34 | The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . | 0.159091 | Positive | Neutral (p = 1.00) |
| 66 | Sponda Plc Stock Exchange Release 5 December 2006 at 15.45 EUR 1.5 BILLION SYNDICATED CREDIT FACILITY FOR SPONDA Sponda has signed an agreement for a one-year syndicated credit facility for a total of EUR 1.5 billion . | 0.165138 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_digits(text) < 0.004 |
Balanced Accuracy = 0.667 | — | -8.02% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.004, the Balanced Accuracy is 8.02% lower than the global Balanced Accuracy.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 2 | The international electronic industry company Elcoteq has laid off tens of employees from its Tallinn facility ; contrary to earlier layoffs the company contracted the ranks of its office workers , the daily Postimees reported . | 0 | Negative | Neutral (p = 1.00) |
| 11 | STORA ENSO , NORSKE SKOG , M-REAL , UPM-KYMMENE Credit Suisse First Boston ( CFSB ) raised the fair value for shares in four of the largest Nordic forestry groups . | 0 | Positive | Neutral (p = 0.87) |
| 21 | ( Filippova ) A trilateral agreement on investment in the construction of a technology park in St Petersburg was to have been signed in the course of the forum , Days of the Russian Economy , that opened in Helsinki today . | 0 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_whitespace(text) < 0.158 AND avg_whitespace(text) >= 0.135 |
Balanced Accuracy = 0.670 | — | -7.72% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` < 0.158 AND `avg_whitespace(text)` >= 0.135, the Balanced Accuracy is 7.72% lower than the global Balanced Accuracy.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 2 | The international electronic industry company Elcoteq has laid off tens of employees from its Tallinn facility ; contrary to earlier layoffs the company contracted the ranks of its office workers , the daily Postimees reported . | 0.153509 | Negative | Neutral (p = 1.00) |
| 47 | The agreement was signed with Biohit Healthcare Ltd , the UK-based subsidiary of Biohit Oyj , a Finnish public company which develops , manufactures and markets liquid handling products and diagnostic test systems . | 0.153488 | Positive | Neutral (p = 1.00) |
| 52 | Within the framework of the partnership , Nokia Siemens Networks has signed an initial framework purchase agreement with Sitronics subsidiary , JSC Mikron , which is the largest manufacturer and exporter of microelectronic components in Russia . | 0.146939 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_digits(text) < 0.047 AND avg_digits(text) >= 0.016 |
Balanced Accuracy = 0.671 | — | -7.54% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.047 AND `avg_digits(text)` >= 0.016, the Balanced Accuracy is 7.54% lower than the global Balanced Accuracy.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 12 | A purchase agreement for 7,200 tons of gasoline with delivery at the Hamina terminal , Finland , was signed with Neste Oil OYj at the average Platts index for this September plus eight US dollars per month . | 0.0193237 | Positive | Neutral (p = 1.00) |
| 30 | The Brazilian unit of Finnish security solutions provider F-Secure signed up 1,500 new clients last year , online news source Reseller Web quoted the division 's commercial director , Vladimir Brand+úo , as saying . | 0.0186047 | Positive | Neutral (p = 0.99) |
| 37 | Aspo Plc STOCK EXCHANGE RELEASE February 11 , 2011 at8 .45 a.m. ESL Shipping Ltd , part of Aspo Group , has signed a new , long-term contract with Rautaruukki Corporation for the marine transport of raw materials on the Baltic Sea . | 0.0387931 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "group" |
Balanced Accuracy = 0.679 | — | -6.40% than global |
🔍✨Examples
For records in the dataset where `text` contains "group", the Balanced Accuracy is 6.4% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 37 | Aspo Plc STOCK EXCHANGE RELEASE February 11 , 2011 at8 .45 a.m. ESL Shipping Ltd , part of Aspo Group , has signed a new , long-term contract with Rautaruukki Corporation for the marine transport of raw materials on the Baltic Sea . | Positive | Neutral (p = 1.00) |
| 76 | Other measures included increasing synergies and economies of scale within the Grimaldi Group and personnel adjustments , divestments and redelivery of excess tonnage . | Neutral | Positive (p = 1.00) |
| 79 | TELECOMWORLDWIRE-7 April 2006-TJ Group Plc sells stake in Morning Digital Design Oy Finnish IT company TJ Group Plc said on Friday 7 April that it had signed an agreement on selling its shares of Morning Digital Design Oy to Edita Oyj . | Positive | Neutral (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!