Report for yiyanghkust/finbert-tone
#30
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 13 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_66agree, split train).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.099 | Add typos | 99/1000 tested samples (9.9%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.9% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1444 | Aspocomp has repaid its interest bearing liability to Standard Chartered Bank and will use the rest of the consideration to partially repay its interest bearing liabilities in Finland and to improve its liquidity . | Aspocojp hqs repzid its interest bearing liability to Stznard Chartered Bank nd will use the rest of the consideration to partially repay its intedrest bearing liabilities in Finland and to improve its liquidkity . | Positive (p = 0.99) | Neutral (p = 1.00) |
| 669 | Demand was brisk as expected and order levels have remained high . | Dwmand was brisk as expected and order levels have reamined high . | Negative (p = 0.99) | Positive (p = 1.00) |
| 763 | Nokia controls more than 50 percent of phone sales in India and Africa , and has a strong prosition on most other emerging markets . | Nkokia fontrols more than 50 percent of phone sales in India and Africa , and has a stront prositon on most other emerging markets . | Positive (p = 1.00) | Neutral (p = 1.00) |
👉Performance issues (12)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | avg_digits(text) < 0.019 AND avg_digits(text) >= 0.002 |
Balanced Accuracy = 0.609 | — | -21.49% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.019 AND `avg_digits(text)` >= 0.002, the Balanced Accuracy is 21.49% lower than the global Balanced Accuracy.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 10 | A purchase agreement for 7,200 tons of gasoline with delivery at the Hamina terminal , Finland , was signed with Neste Oil OYj at the average Platts index for this September plus eight US dollars per month . | 0.0193237 | Positive | Neutral (p = 1.00) |
| 27 | The Brazilian unit of Finnish security solutions provider F-Secure signed up 1,500 new clients last year , online news source Reseller Web quoted the division 's commercial director , Vladimir Brand+úo , as saying . | 0.0186047 | Positive | Neutral (p = 0.99) |
| 65 | On the route between Helsinki in Finland and Tallinn in Estonia , cargo volumes increased by 36 % , while cargo volumes between Finland and Sweden fell by 9 % . | 0.01875 | Neutral | Positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | text contains "said" |
Precision = 0.717 | — | -13.85% than global |
🔍✨Examples
For records in the dataset where `text` contains "said", the Precision is 13.85% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 68 | TELECOMWORLDWIRE-7 April 2006-TJ Group Plc sells stake in Morning Digital Design Oy Finnish IT company TJ Group Plc said on Friday 7 April that it had signed an agreement on selling its shares of Morning Digital Design Oy to Edita Oyj . | Positive | Neutral (p = 1.00) |
| 79 | `` They would invest not only in the physical infrastructure , but would also provide know-how for managing and developing science and technology parks , '' said Sunrise Valley director Andrius Bagdonas . | Positive | Neutral (p = 1.00) |
| 175 | Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | avg_whitespace(text) < 0.165 AND avg_whitespace(text) >= 0.161 |
Balanced Accuracy = 0.685 | — | -11.73% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` < 0.165 AND `avg_whitespace(text)` >= 0.161, the Balanced Accuracy is 11.73% lower than the global Balanced Accuracy.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 122 | `` After this purchase , Cramo will become the second largest rental services provider in the Latvian market . | 0.163636 | Positive | Neutral (p = 0.90) |
| 126 | Finnish Aldata Solution has signed a contract of supply its G.O.L.D. system to two French retail chains . | 0.161905 | Positive | Neutral (p = 1.00) |
| 413 | - BEIJING XFN-ASIA - Hong Kong-listed Standard Chartered Bank said it has signed a China mobile phone dealer financing agreement with Nokia , making it the first foreign bank to offer financing to the country 's small and medium enterprise -LR | 0.164609 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | major 🔴 | avg_word_length(text) >= 5.702 |
Balanced Accuracy = 0.690 | — | -11.09% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` >= 5.702, the Balanced Accuracy is 11.09% lower than the global Balanced Accuracy.| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 56 | The company supports its global customers in developing new technologies and offers a fast route from product development to applications and volume production . | 5.75 | Neutral | Positive (p = 1.00) |
| 66 | Other measures included increasing synergies and economies of scale within the Grimaldi Group and personnel adjustments , divestments and redelivery of excess tonnage . | 6.04167 | Neutral | Positive (p = 1.00) |
| 131 | Finnish Okmetic that manufactures and processes silicon wafers for the semiconductor and sensor industries and Norwegian solar wafer company NorSun have signed a contract under which Okmetic will supply NorSun mono silicon crystals for use in solar cell manufacturing . | 5.75 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_whitespace(text) < 0.171 AND avg_whitespace(text) >= 0.166 |
Balanced Accuracy = 0.702 | — | -9.54% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` < 0.171 AND `avg_whitespace(text)` >= 0.166, the Balanced Accuracy is 9.54% lower than the global Balanced Accuracy.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 59 | KESKO CORPORATION STOCK EXCHANGE RELEASE 28.02.2008 AT 09.30 1 ( 1 ) Kesko Corporation and Aspo plc today signed an agreement by which Aspo acquires the share capital of Kauko-Telko Ltd , a subsidiary wholly owned by Kesko . | 0.169643 | Positive | Neutral (p = 1.00) |
| 83 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | 0.167742 | Positive | Neutral (p = 0.74) |
| 111 | Ruukki announced that it has signed an agreement with Aker Solutions of Norway to supply 24 suction anchors in September 2010 from its Kalajoki unit in Finland . | 0.167702 | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_whitespace(text) < 0.151 |
Balanced Accuracy = 0.709 | — | -8.61% than global |
🔍✨Examples
For records in the dataset where `avg_whitespace(text)` < 0.151, the Balanced Accuracy is 8.61% lower than the global Balanced Accuracy.| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 48 | Within the framework of the partnership , Nokia Siemens Networks has signed an initial framework purchase agreement with Sitronics subsidiary , JSC Mikron , which is the largest manufacturer and exporter of microelectronic components in Russia . | 0.146939 | Positive | Neutral (p = 1.00) |
| 56 | The company supports its global customers in developing new technologies and offers a fast route from product development to applications and volume production . | 0.142857 | Neutral | Positive (p = 1.00) |
| 66 | Other measures included increasing synergies and economies of scale within the Grimaldi Group and personnel adjustments , divestments and redelivery of excess tonnage . | 0.136905 | Neutral | Positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_word_length(text) >= 4.949 AND avg_word_length(text) < 5.095 |
Balanced Accuracy = 0.713 | — | -8.12% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` >= 4.949 AND `avg_word_length(text)` < 5.095, the Balanced Accuracy is 8.12% lower than the global Balanced Accuracy.| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 16 | Incap Contract Manufacturing Services Pvt Ltd , a subsidiary of Incap Corporation of Finland , plans to double its revenues by 2007-2008 . | 5.04348 | Positive | Neutral (p = 0.70) |
| 31 | The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . | 5.04545 | Positive | Neutral (p = 1.00) |
| 74 | The company still expects its turnover in 2010 to slightly increase from the level of 2009 , adding that `` market predictability is still too poor for trustworthy forecasts on the market development of the contract manufacturing business during the current year '' . | 5.09091 | Positive | Negative (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "business" |
Balanced Accuracy = 0.722 | — | -6.94% than global |
🔍✨Examples
For records in the dataset where `text` contains "business", the Balanced Accuracy is 6.94% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 74 | The company still expects its turnover in 2010 to slightly increase from the level of 2009 , adding that `` market predictability is still too poor for trustworthy forecasts on the market development of the contract manufacturing business during the current year '' . | Positive | Negative (p = 1.00) |
| 134 | South African Sappi will become the largest foreign forest industry company operating in Finland as a result of the acquisition Finnish M-real Corporation 's Graphic Papers Business unit . | Positive | Neutral (p = 0.99) |
| 374 | Separately , YIT Corporation and Austrian firm E AG last week signed an agreement on the sale of E 's building system service business in Germany , Austria , Poland , the Czech Republic , Hungary and Romania for EUR 55 mln . | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "group" |
Balanced Accuracy = 0.725 | — | -6.49% than global |
🔍✨Examples
For records in the dataset where `text` contains "group", the Balanced Accuracy is 6.49% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 34 | Aspo Plc STOCK EXCHANGE RELEASE February 11 , 2011 at8 .45 a.m. ESL Shipping Ltd , part of Aspo Group , has signed a new , long-term contract with Rautaruukki Corporation for the marine transport of raw materials on the Baltic Sea . | Positive | Neutral (p = 1.00) |
| 66 | Other measures included increasing synergies and economies of scale within the Grimaldi Group and personnel adjustments , divestments and redelivery of excess tonnage . | Neutral | Positive (p = 1.00) |
| 68 | TELECOMWORLDWIRE-7 April 2006-TJ Group Plc sells stake in Morning Digital Design Oy Finnish IT company TJ Group Plc said on Friday 7 April that it had signed an agreement on selling its shares of Morning Digital Design Oy to Edita Oyj . | Positive | Neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "period" |
Precision = 0.780 | — | -6.37% than global |
🔍✨Examples
For records in the dataset where `text` contains "period", the Precision is 6.37% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 39 | Nyrstar has also agreed to supply to Talvivaara up to 150,000 tonnes of sulphuric acid per annum for use in Talvivaara 's leaching process during the period of supply of the zinc in concentrate . | Positive | Neutral (p = 1.00) |
| 83 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | Positive | Neutral (p = 0.74) |
| 560 | 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . | Positive | Negative (p = 0.59) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_digits(text) < 0.047 AND avg_digits(text) >= 0.023 |
Balanced Accuracy = 0.727 | — | -6.33% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.047 AND `avg_digits(text)` >= 0.023, the Balanced Accuracy is 6.33% lower than the global Balanced Accuracy.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 34 | Aspo Plc STOCK EXCHANGE RELEASE February 11 , 2011 at8 .45 a.m. ESL Shipping Ltd , part of Aspo Group , has signed a new , long-term contract with Rautaruukki Corporation for the marine transport of raw materials on the Baltic Sea . | 0.0387931 | Positive | Neutral (p = 1.00) |
| 39 | Nyrstar has also agreed to supply to Talvivaara up to 150,000 tonnes of sulphuric acid per annum for use in Talvivaara 's leaching process during the period of supply of the zinc in concentrate . | 0.0307692 | Positive | Neutral (p = 1.00) |
| 55 | In Sweden , Gallerix accumulated SEK denominated sales were down 1 % and EUR denominated sales were up 11 % . | 0.0275229 | Neutral | Negative (p = 0.84) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_word_length(text) >= 5.095 AND avg_word_length(text) < 5.627 |
Balanced Accuracy = 0.737 | — | -5.03% than global |
🔍✨Examples
For records in the dataset where `avg_word_length(text)` >= 5.095 AND `avg_word_length(text)` < 5.627, the Balanced Accuracy is 5.03% lower than the global Balanced Accuracy.| text | avg_word_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 27 | The Brazilian unit of Finnish security solutions provider F-Secure signed up 1,500 new clients last year , online news source Reseller Web quoted the division 's commercial director , Vladimir Brand+úo , as saying . | 5.17143 | Positive | Neutral (p = 0.99) |
| 43 | The agreement was signed with Biohit Healthcare Ltd , the UK-based subsidiary of Biohit Oyj , a Finnish public company which develops , manufactures and markets liquid handling products and diagnostic test systems . | 5.35294 | Positive | Neutral (p = 1.00) |
| 71 | Under this agreement Biohit becomes a focus supplier of pipettors and disposable pipettor tips to VWR customers throughout Europe . | 5.6 | Positive | Neutral (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!