Intellya Data Science Team

AI & ML interests

None defined yet.

IntellyaDS's activity

StopwolfĀ 
posted an update 6 months ago
view post
Post
1061
šŸ‡·šŸ‡ø New Benchmark for Serbian Language šŸ‡·šŸ‡ø

@DjMel and I recently released a new benchmark for Serbian language that measures General Knowledge of LLMs. We had to parse over 20 years of university entrance exams for University of Belgrade, so the dataset is of high quality.

šŸ„‡ OAI models still hold the podium places with a significant gap compared to open-source models
šŸ¤” Qwen/Qwen2-7B-Instruct and VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct models show promising results considering they weren't trained on Serbian language
šŸ“ˆ Best open-source model seems to be Stopwolf/Mustra-7B-Instruct-v0.2, a merge between gordicaleksa/YugoGPT and mistralai/Mistral-7B-Instruct-v0.2
šŸ“‰ Some models like google/gemma-2-9b-it turned out to be a disappointment with random guessing-like accuracy

Take a look at the whole results at the dataset page:
DjMel/oz-eval

P.S. If you have any constructive criticism or ideas for improvement, feel free to use dataset's Discussions page!