---
license: apache-2.0
---
<!--
 * @Author: qiang gao gaoqiang_mx@163.com
 * @Date: 2024-05-04 21:09:13
 * @LastEditors: qiang gao gaoqiang_mx@163.com
 * @LastEditTime: 2024-05-05 08:43:45
 * @FilePath: \llama3\hf\Llama3-8x8b-MoE-Instruct\README.md
 * @Description:
-->
<p align="center">
    <br>
    <img src="./figures/llama3-MoE.jpg" width="800"/>
    <br>
</p>
<!-- <p align="center">
    <img alt="GitHub" src="https://img.shields.io/github/license/cooper12121/Llama3-8×8b-MoE .svg?color=blue&style=flat-square">
    <img alt="GitHub release (latest by date)" src="https://img.shields.io/github/v/release/cooper12121/llama3-Chinese">
    <img alt="GitHub top language" src="https://img.shields.io/github/languages/top/cooper12121/llama3-Chinese">
    <a href="https://app.codacy.com/gh/cooper12121/llama3-Chinese/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade"><img src="https://app.codacy.com/project/badge/Grade/142d688425494644b5b156068f55370d"/></a>
</p> -->

---
本项目基于Meta发布的[llama3-8B-Instruct模型](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Chat)进行开发。即将MLP复制8份，创建一个随机初始化的router，其余参数权重保持不变，搭建一个热启动的MoE模型。这种方式能够极大地降低从头开始训练一个MoE模型的成本，便于快速的在下游任务中微调使用。
---

> 其中 router_warmboot表示使用chines-mixtral-Instruct版本中的router参数进行llama3-MoE——Instruct参数的初始化，router_random是router随机初始化的版本。

**详情请见github仓库[https://github.com/cooper12121/llama3-8x8b-MoE](https://github.com/cooper12121/llama3-8x8b-MoE)**

**generate**
```python
  import sys
  sys.path.append("/apdcephfs_qy3/share_301372554/share_info/qianggao/")

  from modeling_file.llama3_moe.modeling_llama_moe import LlamaMoEForCausalLM
  from modeling_file.llama3_moe.tokenization_llama_fast import LlamaTokenizerFast
  model_ckpt = "/apdcephfs_qy3/share_301372554/share_info/qianggao/ckpt/llama3-8x8b-MoE-base"
  tokenizer = LlamaTokenizerFast.from_pretrained(model_ckpt)
  # print(tokenizer)
  model = LlamaMoEForCausalLM.from_pretrained(model_ckpt,device_map="auto",use_cache=False)
  text_list = ["hello,what is your name?","你好，你叫什么名字"]
  
  tokenizer.pad_token = tokenizer.eos_token
  tokenizer.pad_token_id = tokenizer.eos_token_id

  inputs = tokenizer(text_list,return_tensors="pt", padding=True).to("cuda")
 
  output = model.generate(**inputs,pad_token_id=tokenizer.eos_token_id,max_new_tokens=100)
  print(tokenizer.batch_decode(output))
```
**其中modeling_file文件可从github仓库获取**