Update README.md

README.md

@@ -40,7 +40,7 @@ In this work, we take a step toward realizing such an approach. Specifically, we
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
-model_id="empirischtech/Llama-3.1-10b-instruct"
+model_id="empirischtech/Llama-3.1-10B-Instruct"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
     model_id,
@@ -64,23 +64,57 @@ output_text = tokenizer.decode(output[0], skip_special_tokens=True)
 
 ## Evaluation Results
 
-### Overview
+The following two different evaluations are performed.
+
+
+### Preplexity as Evaluation Metric
+
+Perplexity (PPL) is a metric used to evaluate the performance of language models. It measures how well a probability distribution or a language model predicts a sample. A **lower perplexity** score indicates better performance (i.e., the model is more confident in its predictions).
+
+
+
+```python
+from evaluate import load
+import datasets
+
+
+perplexity = load("perplexity", module_type="metric")
+input_texts = datasets.load_dataset("wikitext",
+                                    "wikitext-2-raw-v1",
+                                    split="test")["text"]
+
+input_texts = [s for s in input_texts if s!='']
+
+model_path='empirischtech/Llama-3.1-10B-Instruct'
+results = perplexity.compute(model_id=model_name_or_path,
+                             add_start_token=False,
+                             predictions=input_texts)
+
+
+print(round(results["mean_perplexity"], 2))
+```
+
+
+
+#### Main Results
+
+| Model |  Perplexity Score |
+|---------------------------------------------|----------|
+| **Llama-3.1-8B-Instruct** | 842611366.59 |
+| **Llama-3.1-10B-Instruct** | 2890.31 |
+
+
+### Harness Evaluation
+
 - The performance evaluation is based on the tasks being evaluated on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
 The model is evaluated on three benchmark datasets, which include `ARC-Challenge`, `HellaSwag` and `MMLU`.
 The library used is [lm-evaluation-harness repository](https://github.com/EleutherAI/lm-evaluation-harness)
 
 
-### Main Results
-| Model |  ARC | HellaSwag | MMLU | TruthfulQA | | MT_Bench |
-|--------------------------------------------------------------------|----------|----------|----------|------|----------|-|-------------|
-| **[Llama-2-70b-instruct-v2](https://huggingface.co/upstage/Llama-2-70b-instruct-v2)**(Ours, Open LLM Leaderboard) | **73** | **71.1** | **87.9** | **70.6** | **62.2** | | **7.44063** |
-| [Llama-2-70b-instruct](https://huggingface.co/upstage/Llama-2-70b-instruct) (Ours, Open LLM Leaderboard) | 72.3 | 70.9 | 87.5 | 69.8 | 61 | | 7.24375  |
-| [llama-65b-instruct](https://huggingface.co/upstage/llama-65b-instruct) (***Ours***, ***Open LLM Leaderboard***) | 69.4 | 67.6 | 86.5 | 64.9 | 58.8 | | |
-| Llama-2-70b-hf | 67.3 | 67.3 | 87.3 | 69.8 | 44.9 | | |
-| [llama-30b-instruct-2048](https://huggingface.co/upstage/llama-30b-instruct-2048) (Ours, Open LLM Leaderboard) | 67.0 | 64.9 | 84.9 | 61.9 | 56.3 | | |
-| [llama-30b-instruct](https://huggingface.co/upstage/llama-30b-instruct) (Ours, Open LLM Leaderboard) | 65.2 | 62.5 | 86.2 | 59.4 | 52.8 | | |
-| llama-65b | 64.2 | 63.5 | 86.1 | 63.9 | 43.4 | | |
-| falcon-40b-instruct | 63.4 | 61.6 | 84.3 | 55.4 | 52.5 | | |
+#### Main Results
+| Model |  ARC | HellaSwag | MMLU |
+|------------------------|----------|--------|------|
+| **Llama-3.1-8B-Instruct** | **73** | **71.1** | **87.9** |
 
 
 ### Scripts to generate evalution results
@@ -94,6 +128,9 @@ from lm_eval import evaluator
 tasks_list = ["arc_challenge", "gpqa", "ifeval", "mmlu_pro", "hellaswag"]  # Benchmark dataset
 
 model_path='rwmasood/llama-3.1-10b-instruct'
+model_name_or_path = "./output/checkpoint-2800"
+
+```
 
 # Run evaluation
 results = evaluator.simple_evaluate(