inoki-giskard/scan-report-temp · Report for ahmedrachid/FinancialBERT-Sentiment-Analysis

Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊

We have identified 3 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).

👉Robustness issues (3)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	major 🔴	—	Fail rate = 0.416	Transform to title case	416/1000 tested samples (41.6%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 41.6% of the cases. We expected the predictions not to be affected by this transformation.

	text	Transform to title case(text)	Original prediction	Prediction after perturbation
408	Ragutis , which is based in Lithuania 's second-largest city Kaunas , boosted its sales last year 22.3 per cent to 36.4 million liters .	Ragutis , Which Is Based In Lithuania 'S Second-Largest City Kaunas , Boosted Its Sales Last Year 22.3 Per Cent To 36.4 Million Liters .	positive (p = 1.00)	neutral (p = 1.00)
785	The agreement must be approved by the Russian competition authorities before it enters into force .	The Agreement Must Be Approved By The Russian Competition Authorities Before It Enters Into Force .	positive (p = 0.98)	neutral (p = 1.00)
117	Finnish department store chain Stockmann Oyj Abp net profit rose to 39.8 mln euro ( $ 56.8 mln ) for the first nine months of 2007 from 37.4 mln euro ( $ 53.4 mln ) for the same period of 2006 .	Finnish Department Store Chain Stockmann Oyj Abp Net Profit Rose To 39.8 Mln Euro ( $ 56.8 Mln ) For The First Nine Months Of 2007 From 37.4 Mln Euro ( $ 53.4 Mln ) For The Same Period Of 2006 .	positive (p = 1.00)	neutral (p = 1.00)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	major 🔴	—	Fail rate = 0.408	Transform to uppercase	408/1000 tested samples (40.8%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 40.8% of the cases. We expected the predictions not to be affected by this transformation.

	text	Transform to uppercase(text)	Original prediction	Prediction after perturbation
785	The agreement must be approved by the Russian competition authorities before it enters into force .	THE AGREEMENT MUST BE APPROVED BY THE RUSSIAN COMPETITION AUTHORITIES BEFORE IT ENTERS INTO FORCE .	positive (p = 0.98)	neutral (p = 0.99)
118	Finnish lifting equipment maker Konecranes Oyj said on July 30 , 2008 that its net profit rose to 71.2 mln euro ( $ 111.1 mln ) for the first half of 2008 from 57.1 mln euro ( $ 89.1 mln ) for the same period of 2007 .	FINNISH LIFTING EQUIPMENT MAKER KONECRANES OYJ SAID ON JULY 30 , 2008 THAT ITS NET PROFIT ROSE TO 71.2 MLN EURO ( $ 111.1 MLN ) FOR THE FIRST HALF OF 2008 FROM 57.1 MLN EURO ( $ 89.1 MLN ) FOR THE SAME PERIOD OF 2007 .	positive (p = 1.00)	neutral (p = 0.99)
875	Net sales will , however , increase from 2005 .	NET SALES WILL , HOWEVER , INCREASE FROM 2005 .	positive (p = 1.00)	neutral (p = 1.00)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	medium 🟡	—	Fail rate = 0.073	Add typos	73/1000 tested samples (7.3%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 7.3% of the cases. We expected the predictions not to be affected by this transformation.

	text	Add typos(text)	Original prediction	Prediction after perturbation
2073	Based on the first quarter result , existing order backlog and new order prospects , the company expects that full-year sales will contract by 25 % from 2008 , the gross margin will stay at a healthy level , and the operating profit margin will be lower than in 2008 due to lower sales volume .	VBased on yhe first quarter result , existing ordser backlog and nw order prospects , the company expects that full-year sales will contract by 25 % from 2008 , the gross margin will stay at a healthy level ,a nd the operatonh profit margin will be klower than in 2008 due to lower sales volume .	negative (p = 1.00)	positive (p = 0.99)
253	The disposal of Autotank will also strengthen Aspo 's capital structure , '' commented Gustav Nyberg , CEO of Aspo .	The disposal of Autotank will also sfrengthen Aspo 's capital structurs , '' commente dGustav Nyberg , CEO of Aspo .	positive (p = 1.00)	neutral (p = 1.00)
99	Both operating profit and sales for the three-month period increased , respectively from EUR0 .3 m and EUR13 .1 m , as compared to the corresponding period in 2005 .	Both operating profit and sales for the three-month period nicreased , fespectively frkm EUR0 .3 m and EUR13 .1 m , as cpompared to the corresponding petriod in 2005 .	positive (p = 1.00)	neutral (p = 1.00)

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

💡 What's Next?

Checkout the Giskard Space and improve your model.
The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.

🙌 Big Thanks!

We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!