inoki-giskard/scan-report-temp · Report for ahmedrachid/FinancialBERT-Sentiment-Analysis

Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊

We have identified 4 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_75agree, split train).

👉Robustness issues (3)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	major 🔴	—	Fail rate = 0.397	Transform to uppercase	397/1000 tested samples (39.7%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 39.7% of the cases. We expected the predictions not to be affected by this transformation.

	text	Transform to uppercase(text)	Original prediction	Prediction after perturbation
679	fi is developing cooperation in keyword advertising with Microsoft .	FI IS DEVELOPING COOPERATION IN KEYWORD ADVERTISING WITH MICROSOFT .	positive (p = 1.00)	neutral (p = 1.00)
202	Operating profit rose from EUR 1.94 mn to EUR 2.45 mn .	OPERATING PROFIT ROSE FROM EUR 1.94 MN TO EUR 2.45 MN .	positive (p = 1.00)	neutral (p = 1.00)
3223	Making matters more difficult , the company said it has been grappling with higher oil and gas prices , which have pushed up the cost of energy , raw materials and transportation .	MAKING MATTERS MORE DIFFICULT , THE COMPANY SAID IT HAS BEEN GRAPPLING WITH HIGHER OIL AND GAS PRICES , WHICH HAVE PUSHED UP THE COST OF ENERGY , RAW MATERIALS AND TRANSPORTATION .	negative (p = 1.00)	neutral (p = 1.00)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	major 🔴	—	Fail rate = 0.397	Transform to title case	397/1000 tested samples (39.7%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 39.7% of the cases. We expected the predictions not to be affected by this transformation.

	text	Transform to title case(text)	Original prediction	Prediction after perturbation
2985	It is a disappointment to see the plan folded .	It Is A Disappointment To See The Plan Folded .	negative (p = 1.00)	neutral (p = 1.00)
201	Operating profit of Kauppalehti group rose to EUR 1.5 mn from EUR 1.3 mn , and that of Marketplaces to EUR 1.3 mn from EUR 1.0 mn in the third quarter of 2006 .	Operating Profit Of Kauppalehti Group Rose To Eur 1.5 Mn From Eur 1.3 Mn , And That Of Marketplaces To Eur 1.3 Mn From Eur 1.0 Mn In The Third Quarter Of 2006 .	positive (p = 1.00)	neutral (p = 1.00)
3223	Making matters more difficult , the company said it has been grappling with higher oil and gas prices , which have pushed up the cost of energy , raw materials and transportation .	Making Matters More Difficult , The Company Said It Has Been Grappling With Higher Oil And Gas Prices , Which Have Pushed Up The Cost Of Energy , Raw Materials And Transportation .	negative (p = 1.00)	neutral (p = 1.00)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	medium 🟡	—	Fail rate = 0.072	Add typos	72/1000 tested samples (7.2%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 7.2% of the cases. We expected the predictions not to be affected by this transformation.

	text	Add typos(text)	Original prediction	Prediction after perturbation
1393	Net profit was 35.5 mln compared with 29.8 mln .	Net prifit was 35.5 mln ckompadred with 29.8 mln .	negative (p = 0.64)	neutral (p = 1.00)
1140	Several large stocks tacked lower , however .	Severao large stocks tacked lowee , hoeever .	negative (p = 1.00)	neutral (p = 1.00)
1382	Juha Jordan , chief engineer at Glaston , said one of the reasons for choosing Vacon as a global AC drives supplier is that it has service and support centres in the same countries where Glaston operates .	Juha Jordan , chief engineer at Glaston , saido ne kf the reasons for chowsing Vacon as a gkobal AC rrives supplier iz that ut haz service and support centrwes in the same ountries where Glaston operates .	positive (p = 1.00)	neutral (p = 0.88)

👉Performance issues (1)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	medium 🟡	`text_length(text)` < 112.500 AND `text_length(text)` >= 104.500	Balanced Accuracy = 0.895	—	-8.41% than global

🔍✨Examples

For records in the dataset where `text_length(text)` < 112.500 AND `text_length(text)` >= 104.500, the Balanced Accuracy is 8.41% lower than the global Balanced Accuracy.

	text	text_length(text)	label	Predicted `label`
49	In Sweden , Gallerix accumulated SEK denominated sales were down 1 % and EUR denominated sales were up 11 % .	109	neutral	positive (p = 1.00)
370	Following the increase the company+óEUR TM s capital totals 5.5 mln Romanian lei $ 1.98 mln-1 .56 mln euro .	108	neutral	positive (p = 1.00)
811	Amanda said that it had already made a USD5 .0 m investment commitment in Russia Partners II fund in July 2005 .	112	neutral	positive (p = 0.84)

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

💡 What's Next?

Checkout the Giskard Space and improve your model.
The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.

🙌 Big Thanks!

We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!