--- language: - en license: apache-2.0 library_name: transformers tags: - transformers datasets: - mwitiderrick/AlpacaCode base_model: mwitiderrick/open_llama_3b_code_instruct_0.1 inference: true model_type: llama prompt_template: "[INST] \n{prompt}\n[/INST]\n" created_by: mwitiderrick pipeline_tag: text-generation model-index: - name: mwitiderrick/open_llama_3b_instruct_v_0.2 results: - task: type: text-generation dataset: name: hellaswag type: hellaswag metrics: - type: hellaswag (0-Shot) value: 0.66 name: hellaswag(0-Shot) - task: type: text-generation dataset: name: winogrande type: winogrande metrics: - type: winogrande (0-Shot) value: 0.6322 name: winogrande(0-Shot) - task: type: text-generation dataset: name: arc_challenge type: arc_challenge metrics: - type: arc_challenge (0-Shot) value: 0.3447 name: arc_challenge(0-Shot) source: url: https://huggingface.co/mwitiderrick/open_llama_3b_instruct_v_0.2 name: open_llama_3b_instruct_v_0.2 model card - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 40.7 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_v0.1 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 67.45 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_v0.1 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 27.74 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_v0.1 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 35.86 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_v0.1 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 64.72 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_v0.1 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 1.97 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_glaive_v0.1 name: Open LLM Leaderboard --- # OpenLLaMA Glaive: An Open Reproduction of LLaMA This is an [OpenLlama model Code Instruct](https://huggingface.co/mwitiderrick/open_llama_3b_code_instruct_0.1) that has been fine-tuned on 1 epoch of the [Glaive Assistsnt](https://huggingface.co/datasets/mwitiderrick/glaive-code-assistant) dataset. ## Prompt Template ``` [INST] {{ user_msg }} [/INST] ``` ## Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM,pipeline tokenizer = AutoTokenizer.from_pretrained("mwitiderrick/open_llama_3b_glaive_code_v0.1") model = AutoModelForCausalLM.from_pretrained("mwitiderrick/open_llama_3b_glaive_v0.1") query = "Write a quick sort algorithm in Python" text_gen = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200) output = text_gen(f"[INST]{query}[/INST]") print(output[0]['generated_text']) """ [INST]Write a quick sort algorithm in Python[/INST] Quick sort is a divide and conquer algorithm that sorts an array in-place. It works by repeatedly dividing the array into two sub-arrays, sorting them, and then merging them back together. Here's a Python implementation of the quick sort algorithm: def quick_sort(arr): if len(arr) <= 1: return arr else: pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + [pivot] + quick_sort """ ``` ## Metrics [Detailed metrics](https://huggingface.co/datasets/open-llm-leaderboard/details_mwitiderrick__open_llama_3b_glaive_assistant_v0.1) ``` | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |---------|-------|------|-----:|--------|-----:|---|-----:| |hellaswag|Yaml |none | 0|acc |0.4974|± |0.0050| | | |none | 0|acc_norm|0.6600|± |0.0047| | Groups |Version|Filter|n-shot| Metric | Value | |Stderr| |----------|-------|------|-----:|-----------|-------:|---|-----:| |truthfulqa|N/A |none | 0|bleu_max | 23.5771|± |0.5407| | | |none | 0|bleu_acc | 0.2754|± |0.0002| | | |none | 0|bleu_diff | -8.1019|± |0.5137| | | |none | 0|rouge1_max | 49.5707|± |0.6501| | | |none | 0|rouge1_acc | 0.2607|± |0.0002| | | |none | 0|rouge1_diff| -9.8962|± |0.5492| | | |none | 0|rouge2_max | 33.0399|± |0.8237| | | |none | 0|rouge2_acc | 0.2313|± |0.0002| | | |none | 0|rouge2_diff|-11.9054|± |0.7963| | | |none | 0|rougeL_max | 46.3168|± |0.6705| | | |none | 0|rougeL_acc | 0.2521|± |0.0002| | | |none | 0|rougeL_diff|-10.1301|± |0.5669| | | |none | 0|acc | 0.3191|± |0.0405| | Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |----------|-------|------|-----:|------|-----:|---|-----:| |winogrande|Yaml |none | 0|acc |0.6322|± |0.0136| | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |-------------|-------|------|-----:|--------|-----:|---|-----:| |arc_challenge|Yaml |none | 0|acc |0.3234|± |0.0137| | | |none | 0|acc_norm|0.3447|± |0.0139| ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_mwitiderrick__open_llama_3b_glaive_v0.1) | Metric |Value| |---------------------------------|----:| |Avg. |39.74| |AI2 Reasoning Challenge (25-Shot)|40.70| |HellaSwag (10-Shot) |67.45| |MMLU (5-Shot) |27.74| |TruthfulQA (0-shot) |35.86| |Winogrande (5-shot) |64.72| |GSM8k (5-shot) | 1.97|