Upload folder using huggingface_hub

.gitattributes

@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 *.psd filter=lfs diff=lfs merge=lfs -text
 *.txt filter=lfs diff=lfs merge=lfs -text
+img/gpt2_bridge.png filter=lfs diff=lfs merge=lfs -text

.ipynb_checkpoints/lecture_intro_cn-checkpoint.ipynb

@@ -5,7 +5,7 @@
    "id": "2365faf7-39fb-4e53-a810-2e28c4f6b4c1",
    "metadata": {},
    "source": [
-    "# DNAGTP2-基因序列大模型最佳入门1\n",
+    "# DNAGTP2-基因序列大模型最佳入门\n",
     "\n",
     "## 1 概要\n",
     "自然语言大模型早已超出NLP研究领域，正在成为AI for science的基石。生物信息学中的基因序列，则是和自然语言最类似的，把大模型应用于生物序列研究，就成了最近一两年的热门研究方向，特别是2024年预测蛋白质结构的alphaFold获得诺贝尔化学奖，更是为生物学的研究指明了未来的方向。\n",
@@ -18,6 +18,9 @@
     "\n",
     "DNAGTP2就是这样的梯子，仅望能抛砖引玉，让更多的生物学工作者能够越过大模型的门槛，戴上大模型的翅膀，卷过同行。\n",
     "\n",
+    "\n",
+    "<<img src='img/gpt2_bridge.png'  width=\"600px\" />\n",
+    "\n",
     "## 2 教程特色\n",
     "本教程主要有以下特色：\n",
     "\n",
@@ -39,7 +42,17 @@
     "\n",
     "2 大模型学习入门。不仅是生物学领域的，都可以看看，和一般大模型入门没啥差别，只是数据不同。\n",
     "\n",
+    "\n",
+    "huggingface: https://huggingface.co/dnagpt/dnagpt2\n",
+    "\n",
+    "github: https://github.com/maris205/dnagpt2\n",
+    "\n",
+    "\n",
     "## 3 教程大纲\n",
+    "\n",
+    "<img src='img/DNAGPT2.png'  width=\"600px\" />\n",
+    "\n",
+    "\n",
     "1 数据和环境\n",
     "\n",
     "1.1 大模型运行环境简介\n",

01-data_env/.ipynb_checkpoints/3-dataset-use-checkpoint.ipynb

@@ -134,6 +134,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "17a1fa7c-ff4b-419f-8a82-e58cc5777cd4",
    "metadata": {},
@@ -142,7 +143,9 @@
     "\n",
     "当然，数据也可以直接从huggingface的线上仓库读取，这时候需要注意科学上网问题。\n",
     "\n",
-    "具体使用函数也是load_dataset"
+    "具体使用函数也是load_dataset\n",
+    "\n",
+    "<img src='img/datasets_dnagpt.png' width='800px' />"
    ]
   },
   {

01-data_env/1-env-intro.ipynb

@@ -62,7 +62,48 @@
    "id": "444adc87-78c8-4209-8260-0c5c4a668ea0",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "import os\n",
+    "\n",
+    "# 设置环境变量, autodl专区 其他idc\n",
+    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
+    "\n",
+    "# 打印环境变量以确认设置成功\n",
+    "print(os.environ.get('HF_ENDPOINT'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "06d9dc67-dbd4-4d37-bbdb-ccf59c8fdbf9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "import os\n",
+    "# 设置环境变量, autodl一般区域\n",
+    "result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
+    "output = result.stdout\n",
+    "for line in output.splitlines():\n",
+    "    if '=' in line:\n",
+    "        var, value = line.split('=', 1)\n",
+    "        os.environ[var] = value"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2168e365-8254-4063-98bd-27afdbdb2f32",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#lfs 支持\n",
+    "!apt-get update\n",
+    "\n",
+    "!apt-get install git-lfs\n",
+    "\n",
+    "!git lfs install"
+   ]
   }
  ],
  "metadata": {

01-data_env/2-data-intro.ipynb

@@ -15,9 +15,9 @@
    "source": [
     "本教程主要关注基因相关的生物序列数据，包括主要的DNA和蛋白质序列，data目录下数据如下：\n",
     "\n",
-    "* dna_1g.txt DNA序列数据，大小1G，从glue数据集中抽取，具体可参考dnabert2的论文，包括多个模式生物的数据\n",
-    "* potein_1g.txt 蛋白质序列数据，大小1G，从pdb数据库中抽取\n",
-    "* english_500m.txt  英文数据，大小500M，就是英文百科"
+    "* dna_1g.txt DNA序列数据，大小1G，从GUE数据集中抽取，具体可参考dnabert2的论文，包括多个模式生物的数据(https://github.com/MAGICS-LAB/DNABERT_2)\n",
+    "* potein_1g.txt 蛋白质序列数据，大小1G，从pdb/uniprot数据库中抽取(https://www.uniprot.org/help/downloads)\n",
+    "* english_500m.txt  英文数据，大小500M，就是英文百科(https://huggingface.co/datasets/Salesforce/wikitext, https://huggingface.co/datasets/iohadrubin/wikitext-103-raw-v1)"
    ]
   },
   {

01-data_env/3-dataset-use.ipynb

@@ -117,7 +117,9 @@
     "dna_dataset_sample =  DatasetDict(\n",
     "    {\n",
     "        \"train\": dna_dataset[\"train\"].shuffle().select(range(50000)), \n",
-    "        \"valid\": dna_dataset[\"test\"].shuffle().select(range(500))\n",
+    "        \"valid\": dna_dataset[\"test\"].shuffle().select(range(500)),\n",
+    "        \"evla\": dna_dataset[\"test\"].shuffle().select(range(500))\n",
+    "\n",
     "    }\n",
     ")\n",
     "dna_dataset_sample"
@@ -134,6 +136,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "17a1fa7c-ff4b-419f-8a82-e58cc5777cd4",
    "metadata": {},
@@ -142,7 +145,9 @@
     "\n",
     "当然，数据也可以直接从huggingface的线上仓库读取，这时候需要注意科学上网问题。\n",
     "\n",
-    "具体使用函数也是load_dataset"
+    "具体使用函数也是load_dataset\n",
+    "\n",
+    "<img src='img/datasets_dnagpt.png' width='800px' />"
    ]
   },
   {

02-gpt2_bert/.ipynb_checkpoints/1-dna-bpe-checkpoint.ipynb

@@ -0,0 +1,528 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a9fffce5-83e3-4838-8335-acb2e3b50c35",
+   "metadata": {},
+   "source": [
+    "# 2.1 DNA分词器构建"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f28b0950-37dc-4f78-ae6c-9fca33d513fc",
+   "metadata": {},
+   "source": [
+    "## **分词算法**\n",
+    "\n",
+    "### **什么是分词**\n",
+    "分词就是把一个文本序列，分成一个一个的token/词，对于英文这种天生带空格的语言，一般使用空格和标点分词就行了，而对于中文等语言，并没有特殊的符号来分词，因此，一般需要设计专门的分词算法，对于大模型而言，一般需要处理多种语言，因此，也需要专门的分词算法。\n",
+    "\n",
+    "在大模型（如 BERT、GPT 系列、T5 等）中，分词器（tokenizer）扮演着至关重要的角色。它负责将原始文本转换为模型可以处理的格式，即将文本分解成 token 序列，并将这些 token 映射到模型词汇表中的唯一 ID。分词器的选择和配置直接影响模型的性能和效果。以下是几种常见的分词器及其特点，特别关注它们在大型语言模型中的应用。\n",
+    "\n",
+    "### 1. **WordPiece 分词器**\n",
+    "\n",
+    "- **使用场景**：广泛应用于 BERT 及其变体。\n",
+    "- **工作原理**：基于频率统计，从语料库中学习最有效的词汇表。它根据子词（subword）在文本中的出现频率来决定如何分割单词。例如，“playing” 可能被分为 “play” 和 “##ing”，其中“##”表示该部分是前一个 token 的延续。\n",
+    "- **优点**：\n",
+    "  - 处理未知词汇能力强，能够将未见过的词汇分解为已知的子词。\n",
+    "  - 兼容性好，适合多种语言任务。\n",
+    "- **缺点**：\n",
+    "  - 需要额外的标记（如 `##`）来指示子词，可能影响某些应用场景下的可读性。\n",
+    "\n",
+    "### 2. **Byte Pair Encoding (BPE)**\n",
+    "\n",
+    "- **使用场景**：广泛应用于 GPT 系列、RoBERTa、XLM-R 等模型。\n",
+    "- **工作原理**：通过迭代地合并最常见的字符对来构建词汇表。BPE 是一种无监督的学习方法，能够在不依赖于预先定义的词汇表的情况下进行分词。\n",
+    "- **优点**：\n",
+    "  - 灵活性高，适应性强，尤其适用于多语言模型。\n",
+    "  - 不需要特殊标记，生成的词汇表更简洁。\n",
+    "- **缺点**：\n",
+    "  - 对于某些语言或领域特定的词汇，可能会产生较短的子词，导致信息丢失。\n",
+    "\n",
+    "### 3. **SentencePiece**\n",
+    "\n",
+    "- **使用场景**：常见于 T5、mBART 等多语言模型。\n",
+    "- **工作原理**：结合了 BPE 和 WordPiece 的优点，同时支持字符级和词汇级分词。它可以在没有空格的语言（如中文、日文）中表现良好。\n",
+    "- **优点**：\n",
+    "  - 支持无空格语言，适合多语言处理。\n",
+    "  - 学习速度快，适应性强。\n",
+    "- **缺点**：\n",
+    "  - 对于某些特定领域的专业术语，可能需要额外的预处理步骤。\n",
+    "\n",
+    "### 4. **Character-Level Tokenizer**\n",
+    "\n",
+    "- **使用场景**：较少用于大型语言模型，但在某些特定任务（如拼写检查、手写识别）中有应用。\n",
+    "- **工作原理**：直接将每个字符视为一个 token。这种方式简单直接，但通常会导致较大的词汇表。\n",
+    "- **优点**：\n",
+    "  - 简单易实现，不需要复杂的训练过程。\n",
+    "  - 对于字符级别的任务非常有效。\n",
+    "- **缺点**：\n",
+    "  - 词汇表较大，计算资源消耗较多。\n",
+    "  - 捕捉上下文信息的能力较弱。\n",
+    "\n",
+    "### 5. **Unigram Language Model**\n",
+    "\n",
+    "- **使用场景**：主要用于 SentencePiece 中。\n",
+    "- **工作原理**：基于概率分布，选择最优的分词方案以最大化似然函数。这种方法类似于 BPE，但在构建词汇表时考虑了更多的统计信息。\n",
+    "- **优点**：\n",
+    "  - 统计基础强，优化效果好。\n",
+    "  - 适应性强，适用于多种语言和任务。\n",
+    "- **缺点**：\n",
+    "  - 计算复杂度较高，训练时间较长。\n",
+    "\n",
+    "### 分词器的关键特性\n",
+    "\n",
+    "无论选择哪种分词器，以下几个关键特性都是设计和应用中需要考虑的：\n",
+    "\n",
+    "- **词汇表大小**：决定了模型所能识别的词汇量。较大的词汇表可以捕捉更多细节，但也增加了内存和计算需求。\n",
+    "- **处理未知词汇的能力**：好的分词器应该能够有效地处理未登录词（OOV, Out-Of-Vocabulary），将其分解为已知的子词。\n",
+    "- **多语言支持**：对于多语言模型，分词器应能处理不同语言的文本，尤其是那些没有明显分隔符的语言。\n",
+    "- **效率和速度**：分词器的执行速度直接影响整个数据处理管道的效率，尤其是在大规模数据集上。\n",
+    "- **兼容性和灵活性**：分词器应与目标模型架构兼容，并且能够灵活适应不同的任务需求。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "165e2594-277d-44d0-b582-77859a0bc0b2",
+   "metadata": {},
+   "source": [
+    "## DNA等生物序列分词\n",
+    "在生物信息学中，DNA 和蛋白质序列的处理与自然语言处理（NLP）有相似之处，但也有其独特性。为了提取这些生物序列的特征并用于机器学习或深度学习模型，通常需要将长序列分解成更小的片段（类似于 NLP 中的“分词”），以便更好地捕捉局部和全局特征。以下是几种常见的方法，用于对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。\n",
+    "\n",
+    "### 1. **K-mer 分解**\n",
+    "\n",
+    "**定义**：K-mer 是指长度为 k 的连续子序列。例如，在 DNA 序列中，一个 3-mer 可能是 \"ATG\" 或 \"CGA\"。\n",
+    "\n",
+    "**应用**：\n",
+    "- **DNA 序列**：常用的 k 值范围从 3 到 6。较小的 k 值可以捕捉到更细粒度的信息，而较大的 k 值则有助于识别更长的模式。\n",
+    "- **蛋白质序列**：k 值通常较大，因为氨基酸的数量较多（20 种），较长的 k-mer 可以捕捉到重要的结构域或功能区域。\n",
+    "\n",
+    "**优点**：\n",
+    "- 简单且直观，易于实现。\n",
+    "- 可以捕捉到短序列中的局部特征。\n",
+    "\n",
+    "**缺点**：\n",
+    "- 对于非常长的序列，生成的 k-mer 数量会非常大，导致维度爆炸问题。\n",
+    "- 不同位置的 k-mer 之间缺乏上下文关系。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "29c390ef-2e9d-493e-9991-69ecb835b52b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "DNA 3-mers: ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
+      "Protein 4-mers: ['MKQH', 'KQHK', 'QHKA', 'HKAM', 'KAMI', 'AMIV', 'MIVA', 'IVAL', 'VALI', 'ALIV', 'LIVL', 'IVLI', 'VLIT', 'LITA', 'ITAY']\n"
+     ]
+    }
+   ],
+   "source": [
+    "#示例代码（Python）\n",
+    "\n",
+    "def k_mer(seq, k):\n",
+    "    return [seq[i:i+k] for i in range(len(seq) - k + 1)]\n",
+    "\n",
+    "dna_sequence = \"ATGCGTACGTA\"\n",
+    "protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
+    "\n",
+    "print(\"DNA 3-mers:\", k_mer(dna_sequence, 3))\n",
+    "print(\"Protein 4-mers:\", k_mer(protein_sequence, 4))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7ced2bfb-bd42-425a-a3ad-54c9573609c5",
+   "metadata": {},
+   "source": [
+    "### 2. **滑动窗口**\n",
+    "\n",
+    "**定义**：滑动窗口方法通过设定一个固定大小的窗口沿着序列移动，并在每个位置提取窗口内的子序列。这与 K-mer 类似，但允许重叠。\n",
+    "\n",
+    "**应用**：\n",
+    "- **DNA 和蛋白质序列**：窗口大小可以根据具体任务调整，如基因预测、蛋白质结构预测等。\n",
+    "\n",
+    "**优点**：\n",
+    "- 提供了更多的灵活性，可以控制窗口的步长和大小。\n",
+    "- 有助于捕捉局部和全局特征。\n",
+    "\n",
+    "**缺点**：\n",
+    "- 计算复杂度较高，尤其是当窗口大小较大时。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "82cecf91-0076-4c12-b11c-b35120581ef9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Sliding window (DNA, size=3, step=1): ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
+      "Sliding window (Protein, size=4, step=2): ['MKQH', 'QHKA', 'KAMI', 'MIVA', 'VALI', 'LIVL', 'VLIT', 'ITAY']\n"
+     ]
+    }
+   ],
+   "source": [
+    "def sliding_window(seq, window_size, step=1):\n",
+    "    return [seq[i:i+window_size] for i in range(0, len(seq) - window_size + 1, step)]\n",
+    "\n",
+    "dna_sequence = \"ATGCGTACGTA\"\n",
+    "protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
+    "\n",
+    "print(\"Sliding window (DNA, size=3, step=1):\", sliding_window(dna_sequence, 3))\n",
+    "print(\"Sliding window (Protein, size=4, step=2):\", sliding_window(protein_sequence, 4, step=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c33ab920-b451-4846-93d4-20da5a4e1001",
+   "metadata": {},
+   "source": [
+    "### 3. **词表分词和嵌入式表示**\n",
+    "\n",
+    "**定义**：使用预训练的嵌入模型（如 Word2Vec、BERT 等）来将每个 token 映射到高维向量空间中。对于生物序列，可以使用专门设计的嵌入模型，如 ProtTrans、ESM 等。\n",
+    "\n",
+    "**应用**：\n",
+    "- **DNA 和蛋白质序列**：嵌入模型可以捕捉到序列中的语义信息和上下文依赖关系。\n",
+    "\n",
+    "**优点**：\n",
+    "- 捕捉到丰富的语义信息，适合复杂的下游任务。\n",
+    "- 可以利用大规模预训练模型的优势。\n",
+    "\n",
+    "**缺点**：\n",
+    "- 需要大量的计算资源来进行预训练。\n",
+    "- 模型复杂度较高，解释性较差。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "02bf2af0-6077-4b27-8822-f1c3f22914fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "import os\n",
+    "# 设置环境变量, autodl一般区域\n",
+    "result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
+    "output = result.stdout\n",
+    "for line in output.splitlines():\n",
+    "    if '=' in line:\n",
+    "        var, value = line.split('=', 1)\n",
+    "        os.environ[var] = value\n",
+    "\n",
+    "\"\"\"\n",
+    "import os\n",
+    "\n",
+    "# 设置环境变量, autodl专区 其他idc\n",
+    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
+    "\n",
+    "# 打印环境变量以确认设置成功\n",
+    "print(os.environ.get('HF_ENDPOINT'))\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "d43b60ee-67f2-4d06-95ea-966c01084fc4",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
+      "To disable this warning, you can either:\n",
+      "\t- Avoid using `tokenizers` before the fork if possible\n",
+      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['ATGCG', 'TACG', 'T', 'A']\n",
+      "Embeddings shape: torch.Size([1, 4, 768])\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "import torch\n",
+    "\n",
+    "# 加载预训练的蛋白质嵌入模型\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
+    "model = AutoModel.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
+    "\n",
+    "dna_sequence = \"ATGCGTACGTA\"\n",
+    "print(tokenizer.tokenize(dna_sequence))\n",
+    "\n",
+    "# 编码序列\n",
+    "inputs = tokenizer(dna_sequence, return_tensors=\"pt\")\n",
+    "\n",
+    "# 获取嵌入\n",
+    "with torch.no_grad():\n",
+    "    outputs = model(**inputs)\n",
+    "    embeddings = outputs.last_hidden_state\n",
+    "\n",
+    "print(\"Embeddings shape:\", embeddings.shape)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c24f10dc-1117-4493-9333-5ed6d898f44a",
+   "metadata": {},
+   "source": [
+    "### 训练DNA BPE分词器\n",
+    "\n",
+    "以上方法展示了如何对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。选择哪种方法取决于具体的任务需求和数据特性。对于简单的分类或回归任务，K-mer 分解或滑动窗口可能是足够的；而对于更复杂的任务，如序列标注或结构预测，基于词汇表的方法或嵌入表示可能会提供更好的性能。\n",
+    "\n",
+    "目前大部分生物序列大模型的论文中，使用最多的依然是传统的K-mer，但一些SOTA的论文则以BEP为主。而BEP分词也是目前GPT、llama等主流自然语言大模型使用的基础分词器。\n",
+    "\n",
+    "因此，我们也演示下从头训练一个DNA BPE分词器的方法。\n",
+    "\n",
+    "我们首先看下GPT2模型，默认的分词器，对DNA序列分词的结果："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "43f1eb8b-1cc2-4ab5-aa8e-2a63132be98c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tokenizers import (\n",
+    "    decoders,\n",
+    "    models,\n",
+    "    normalizers,\n",
+    "    pre_tokenizers,\n",
+    "    processors,\n",
+    "    trainers,\n",
+    "    Tokenizer,\n",
+    ")\n",
+    "from transformers import AutoTokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "27e88f7b-1399-418b-9b91-f970762fac0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gpt2_tokenizer = AutoTokenizer.from_pretrained('gpt2')\n",
+    "gpt2_tokenizer.pad_token = gpt2_tokenizer.eos_token"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "4b015db7-63ba-4909-b02f-07634b3d5584",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['T', 'GG', 'C', 'GT', 'GA', 'AC', 'CC', 'GG', 'G', 'AT', 'C', 'GG', 'G']"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gpt2_tokenizer.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a246fbc9-9e29-4b63-bdf7-f80635d06d1e",
+   "metadata": {},
+   "source": [
+    "可以看到，gpt2模型因为是以英文为主的BPE分词模型，分解的都是1到2个字母的结果，这样显然很难充分表达生物语义，因此，我们使用DNA序列来训练1个BPE分词器，代码也非常简单："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "8357a695-1c29-4b5c-8099-d2e337189410",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = Tokenizer(models.BPE())\n",
+    "tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串\n",
+    "trainer = trainers.BpeTrainer(vocab_size=30000, special_tokens=[\"<|endoftext|>\"]) #3w words"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "32c95888-1498-45cf-8453-421219cc7d45",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "tokenizer.train([\"../01-data_env/data/dna_1g.txt\"], trainer=trainer) #all file list, take 10-20 min"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "5ffdd717-72ed-4a37-bafc-b4a0f61f8ff1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']\n"
+     ]
+    }
+   ],
+   "source": [
+    "encoding = tokenizer.encode(\"TGGCGTGAACCCGGGATCGGG\")\n",
+    "print(encoding.tokens)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a96e7838-6c23-4446-bf86-b098cd93214a",
+   "metadata": {},
+   "source": [
+    "可以看到，以DNA数据训练的分词器，分词效果明显要好的多，各种长度的词都有。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "f1d757c1-702b-4147-9207-471f422f67b2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.save(\"dna_bpe_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "caf8ecea-359e-487b-b456-fab546b9da0d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#然后我们可以使用from_file() 方法从该文件里重新加载 Tokenizer 对象：\n",
+    "new_tokenizer = Tokenizer.from_file(\"dna_bpe_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "8ec6f045-bc30-4012-8027-a879df8def3a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "('dna_bpe_dict/tokenizer_config.json',\n",
+       " 'dna_bpe_dict/special_tokens_map.json',\n",
+       " 'dna_bpe_dict/vocab.json',\n",
+       " 'dna_bpe_dict/merges.txt',\n",
+       " 'dna_bpe_dict/added_tokens.json',\n",
+       " 'dna_bpe_dict/tokenizer.json')"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "#要在 🤗 Transformers 中使用这个标记器，我们必须将它包裹在一个 PreTrainedTokenizerFast 类中\n",
+    "from transformers import GPT2TokenizerFast\n",
+    "dna_tokenizer = GPT2TokenizerFast(tokenizer_object=new_tokenizer)\n",
+    "dna_tokenizer.save_pretrained(\"dna_bpe_dict\")\n",
+    "#dna_tokenizer.push_to_hub(\"dna_bpe_dict_1g\", organization=\"dnagpt\", use_auth_token=\"hf_*****\") # push to huggingface"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "f84506d8-6208-4027-aad7-2b68a1bc16d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer_new = AutoTokenizer.from_pretrained('dna_bpe_dict')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "d40d4d53-6fed-445c-afb5-c0346ab854c8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tokenizer_new.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "640302f6-f740-41a4-ae92-ca4c43d97493",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/.ipynb_checkpoints/3-dna-bert-checkpoint.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a3ec4b86-2029-4d50-9bbf-64b208249165",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tokenizers import Tokenizer\n",
+    "from tokenizers.models import WordPiece\n",
+    "from tokenizers.trainers import WordPieceTrainer\n",
+    "from tokenizers.pre_tokenizers import Whitespace"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "47b3fc92-df22-4e4b-bdf9-671bda924c49",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 初始化一个空的 WordPiece 模型\n",
+    "tokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "73f59aa6-8cce-4124-a3ee-7a5617b91ea7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 设置训练参数\n",
+    "trainer = WordPieceTrainer(\n",
+    "    vocab_size=90000,        # 词汇表大小\n",
+    "    min_frequency=2,         # 最小词频\n",
+    "    special_tokens=[\n",
+    "        \"[PAD]\", \"[UNK]\", \"[CLS]\", \"[SEP]\", \"[MASK]\"\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "tokenizer.train(files=[\"../01-data_env/data/dna_1g.txt\"], trainer=trainer)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7a0ccd64-5172-4f40-9868-cdf02687ae10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.save(\"dna_wordpiece_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eea3c48a-2245-478e-a2ce-f5d1af399d83",
+   "metadata": {},
+   "source": [
+    "## GPT2和bert配置的关键区别\n",
+    "* 最大长度：\n",
+    "在 GPT-2 中，n_ctx 参数指定了模型的最大上下文窗口大小。\n",
+    "在 BERT 中，你应该设置 max_position_embeddings 来指定最大位置嵌入数，这限制了输入序列的最大长度。\n",
+    "* 特殊 token ID：\n",
+    "GPT-2 使用 bos_token_id 和 eos_token_id 分别表示句子的开始和结束。\n",
+    "BERT 使用 [CLS] (cls_token_id) 表示句子的开始，用 [SEP] (sep_token_id) 表示句子的结束。BERT 还有专门的填充 token [PAD] (pad_token_id)。\n",
+    "* 模型类选择：\n",
+    "对于 GPT-2，你使用了 GPT2LMHeadModel，它适合生成任务或语言建模。\n",
+    "对于 BERT，如果你打算进行预训练（例如 Masked Language Modeling），应该使用 BertForMaskedLM。\n",
+    "* 预训练权重：\n",
+    "如果你想从头开始训练，像上面的例子中那样直接从配置创建模型即可。\n",
+    "如果你希望基于现有预训练模型微调，则可以使用 from_pretrained 方法加载预训练权重。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "48e1f20b-cd1a-49fa-be2b-aba30a24e706",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "new_tokenizer = Tokenizer.from_file(\"dna_wordpiece_dict.json\")\n",
+    "\n",
+    "wrapped_tokenizer = PreTrainedTokenizerFast(\n",
+    "    tokenizer_object=new_tokenizer,\n",
+    "    unk_token=\"[UNK]\",\n",
+    "    pad_token=\"[PAD]\",\n",
+    "    cls_token=\"[CLS]\",\n",
+    "    sep_token=\"[SEP]\",\n",
+    "    mask_token=\"[MASK]\",\n",
+    ")\n",
+    "wrapped_tokenizer.save_pretrained(\"dna_wordpiece_dict\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c94dc601-86ec-421c-8638-c8d8b5078682",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer, GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
+    "from transformers import GPT2Tokenizer,GPT2Model,AutoModel\n",
+    "from transformers import DataCollatorForLanguageModeling\n",
+    "from transformers import Trainer, TrainingArguments\n",
+    "from transformers import LineByLineTextDataset\n",
+    "from tokenizers import Tokenizer\n",
+    "from datasets import load_dataset\n",
+    "from transformers import BertConfig, BertModel"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b2658cd2-0ac5-483e-b04d-2716993770e3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = AutoTokenizer.from_pretrained(\"dna_wordpiece_dict\")\n",
+    "#tokenizer.pad_token = tokenizer.eos_token"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a7d0b7b8-b6dc-422a-9133-1d51ec40adbe",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "max_length = 256 #最大输入长度\n",
+    "\n",
+    "# Building the config\n",
+    "#config = BertConfig()\n",
+    "\n",
+    "\n",
+    "# 构建配置\n",
+    "config = AutoConfig.from_pretrained(\n",
+    "    \"bert-base-uncased\",  # 或者其他预训练 BERT 模型名称，这里只是为了获取默认配置\n",
+    "    vocab_size=len(tokenizer),\n",
+    "    max_position_embeddings=max_length,  # 对应于最大位置嵌入数\n",
+    "    pad_token_id=tokenizer.pad_token_id,\n",
+    "    bos_token_id=tokenizer.cls_token_id,  # BERT 使用 [CLS] 作为句子开始标记\n",
+    "    eos_token_id=tokenizer.sep_token_id   # BERT 使用 [SEP] 作为句子结束标记\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Building the model from the config\n",
+    "model = AutoModelForMaskedLM.from_config(config)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "afc2cdd1-228e-4ee7-95f5-07718f00723d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 1. load dna dataset\n",
+    "raw_dataset = load_dataset('text',  data_files=\"../01-data_env/data/dna_1g.txt\")\n",
+    "dataset = raw_dataset[\"train\"].train_test_split(test_size=0.1, shuffle=True)\n",
+    "\n",
+    "# 2. tokenize\n",
+    "def tokenize_function(examples):\n",
+    "    return tokenizer(examples['text'], truncation=True, padding='max_length', max_length=max_length)\n",
+    "\n",
+    "# 3. 对数据集应用分词函数\n",
+    "tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=['text'], num_proc=15)  # 设置为你的 CPU 核心数或根据需要调整\n",
+    "\n",
+    "# 4. 创建一个数据收集器，用于动态填充和遮蔽,注意mlm=true\n",
+    "data_collator = DataCollatorForLanguageModeling(\n",
+    "    tokenizer=tokenizer, mlm=True,mlm_probability=0.15\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "604491f9-2ee7-4722-aad6-02e98457b5ee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run_path = \"bert_run\"\n",
+    "train_epoches = 5\n",
+    "batch_size = 10\n",
+    "\n",
+    "\n",
+    "training_args = TrainingArguments(\n",
+    "        output_dir=run_path,\n",
+    "        overwrite_output_dir=True,\n",
+    "        num_train_epochs=train_epoches,\n",
+    "        per_device_train_batch_size=batch_size,\n",
+    "        save_steps=2000,\n",
+    "        save_total_limit=2,\n",
+    "        prediction_loss_only=True,\n",
+    "        fp16=True, #v100没法用\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    train_dataset=tokenized_datasets[\"train\"],\n",
+    "    eval_dataset=tokenized_datasets[\"test\"],\n",
+    "    data_collator=data_collator,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d91a8bfb-f3ff-4031-a0d7-ebedc200d65a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.train()\n",
+    "trainer.save_model(\"dna_bert_v0\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fc4ad6ad-6433-471f-8510-1ae46558d4ce",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#upload model\n",
+    "#model.push_to_hub(\"dna_bert_v0\", organization=\"dnagpt\", use_auth_token=\"hf_*******\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/.ipynb_checkpoints/4-gene-feature-checkpoint.ipynb

@@ -0,0 +1,489 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "b1b37ca8-25a3-440c-9b68-7f72ce670ade",
+   "metadata": {},
+   "source": [
+    "# 2.4 基因大模型的生物序列特征提取"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d3d04215-2b6c-41fb-92a4-90c82d322ba4",
+   "metadata": {},
+   "source": [
+    "使用 GPT-2 模型获取文本的特征向量是一个常见的需求，尤其是在进行文本分类、相似度计算或其他下游任务时。Hugging Face 的 transformers 库提供了简单易用的接口来实现这一点。以下是详细的步骤和代码示例，帮助你从 GPT-2 模型中提取文本的特征向量。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3ff5b7c6-e57c-4839-8510-f764154faa65",
+   "metadata": {},
+   "source": [
+    "使用 GPT-2 模型获取文本的特征向量是一个常见的需求，尤其是在进行文本分类、相似度计算或其他下游任务时。Hugging Face 的 `transformers` 库提供了简单易用的接口来实现这一点。以下是详细的步骤和代码示例，帮助你从 GPT-2 模型中提取文本的特征向量。\n",
+    "\n",
+    "### 方法 1: 使用隐藏状态（Hidden States）\n",
+    "\n",
+    "GPT-2 是一个基于 Transformer 的语言模型，它在每一层都有隐藏状态（hidden states），这些隐藏状态可以作为文本的特征表示。你可以选择最后一层的隐藏状态作为最终的特征向量，或者对多层的隐藏状态进行平均或拼接。\n",
+    "\n",
+    "\n",
+    "### 方法 2: 使用池化策略\n",
+    "\n",
+    "另一种方法是通过对所有 token 的隐藏状态进行池化操作来获得句子级别的特征向量。常见的池化方法包括：\n",
+    "\n",
+    "- **均值池化**（Mean Pooling）：对所有 token 的隐藏状态求平均。\n",
+    "- **最大池化**（Max Pooling）：对每个维度取最大值。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "e7fe053b-d6da-488a-9c62-24e4b40a992d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'input_ids': tensor([[    1,   191,    29,   753,  1241,  2104, 12297,   357,    85,  4395,\n",
+      "         26392,    16]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "tokenizer = AutoTokenizer.from_pretrained('dna_bpe_dict')\n",
+    "tokenizer.tokenize(\"GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG\")\n",
+    "#result: [G','AGCAC','ATTCGCC',....]\n",
+    "\n",
+    "model = AutoModel.from_pretrained('dna_gpt2_v0')\n",
+    "import torch\n",
+    "dna = \"ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC\"\n",
+    "inputs = tokenizer(dna, return_tensors = 'pt')\n",
+    "print(inputs)\n",
+    "\n",
+    "outputs = model(inputs[\"input_ids\"])\n",
+    "#outputs = model(**inputs)\n",
+    "\n",
+    "hidden_states = outputs.last_hidden_state # [1, sequence_length, 768]  outputs.last_hidden_state or outputs[0]\n",
+    "\n",
+    "# embedding with mean pooling\n",
+    "embedding_mean = torch.mean(hidden_states[0], dim=0)\n",
+    "print(embedding_mean.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with max pooling\n",
+    "embedding_max = torch.max(hidden_states[0], dim=0)[0]\n",
+    "print(embedding_max.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with first token\n",
+    "embedding_first_token = hidden_states[0][0]\n",
+    "print(embedding_first_token.shape) # expect to be 768"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "id": "a1f2b545-283a-4613-a953-beb82f427826",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Some weights of BertModel were not initialized from the model checkpoint at dna_bert_v0 and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
+      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'input_ids': tensor([[    6,   200, 16057,    10,  1256,  2123, 12294,   366, 13138,  7826,\n",
+      "            82,    25]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "import torch\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained('dna_wordpiece_dict')\n",
+    "tokenizer.tokenize(\"GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG\")\n",
+    "#result: [G','AGCAC','ATTCGCC',....]\n",
+    "\n",
+    "model = AutoModel.from_pretrained('dna_bert_v0')\n",
+    "dna = \"ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC\"\n",
+    "inputs = tokenizer(dna, return_tensors = 'pt')\n",
+    "print(inputs)\n",
+    "\n",
+    "outputs = model(inputs[\"input_ids\"])\n",
+    "#outputs = model(**inputs)\n",
+    "\n",
+    "hidden_states = outputs.last_hidden_state # [1, sequence_length, 768]  outputs.last_hidden_state or outputs[0]\n",
+    "\n",
+    "# embedding with mean pooling\n",
+    "embedding_mean = torch.mean(hidden_states[0], dim=0)\n",
+    "print(embedding_mean.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with max pooling\n",
+    "embedding_max = torch.max(hidden_states[0], dim=0)[0]\n",
+    "print(embedding_max.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with first token\n",
+    "embedding_first_token = hidden_states[0][0]\n",
+    "print(embedding_first_token.shape) # expect to be 768"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56761874-9af7-4b90-aa8b-131e5b8c69b6",
+   "metadata": {},
+   "source": [
+    "## 特征提取并分类\n",
+    "\n",
+    "我们使用第一章中的\"dnagpt/dna_core_promoter\"数据集，演示下使用我们训练的DNA GPT2或者DNA bert模型，提取序列特征，然使用最基础的逻辑回归分类方法，对序列进行分类。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "id": "f1ca177c-a80f-48a1-b2f9-16c13b3350db",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
+      ]
+     },
+     "execution_count": 45,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import subprocess\n",
+    "import os\n",
+    "# 设置环境变量, autodl一般区域\n",
+    "result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
+    "output = result.stdout\n",
+    "for line in output.splitlines():\n",
+    "    if '=' in line:\n",
+    "        var, value = line.split('=', 1)\n",
+    "        os.environ[var] = value\n",
+    "\n",
+    "#或者\n",
+    "\"\"\"\n",
+    "import os\n",
+    "\n",
+    "# 设置环境变量, autodl专区 其他idc\n",
+    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
+    "\n",
+    "# 打印环境变量以确认设置成功\n",
+    "print(os.environ.get('HF_ENDPOINT'))\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "2295739c-e80a-47be-9400-88bfab4b0bb6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "DatasetDict({\n",
+       "    train: Dataset({\n",
+       "        features: ['sequence', 'label'],\n",
+       "        num_rows: 59196\n",
+       "    })\n",
+       "})"
+      ]
+     },
+     "execution_count": 46,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from datasets import load_dataset\n",
+    "dna_data = load_dataset(\"dnagpt/dna_core_promoter\")\n",
+    "dna_data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c804bced-f151-43a7-8a95-156db358da3e",
+   "metadata": {},
+   "source": [
+    "这里，我们不需要关注这个数据的具体生物学含义，只需知道sequence是具体的DNA序列，label是分类标签，有两个类别0和1即可"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "id": "9a47a1b1-21f2-4d71-801c-50f88e326ed3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'sequence': 'CATGCGGGTCGATATCCTATCTGAATCTCTCAGCCCAAGAGGGAGTCCGCTCATCTATTCGGCAGTACTG',\n",
+       " 'label': 0}"
+      ]
+     },
+     "execution_count": 47,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dna_data[\"train\"][0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cde7986d-a225-41ca-8f11-614d079fd2bf",
+   "metadata": {},
+   "source": [
+    "这里使用scikit-learn库来构建逻辑回归分类器。首先是特征提取："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 52,
+   "id": "4010d991-056a-43ce-8cca-30eeec8678f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "from sklearn.datasets import load_iris\n",
+    "from sklearn.metrics import accuracy_score\n",
+    "\n",
+    "\n",
+    "def get_gpt2_feature(sequence):\n",
+    "    return \n",
+    "\n",
+    "# 加载数据集\n",
+    "data = load_iris()\n",
+    "X = data.data[data.target < 2]  # 只选择前两个类别\n",
+    "y = data.target[data.target < 2]\n",
+    "\n",
+    "X = []\n",
+    "Y = []\n",
+    "\n",
+    "for item in dna_data[\"train\"]:\n",
+    "    sequence = item[\"sequence\"]\n",
+    "    label = item[\"label\"]\n",
+    "    x_v = get_gpt2_feature(sequence)\n",
+    "    y_v = label\n",
+    "    X.append(x_v)\n",
+    "    Y.append(y_v)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 49,
+   "id": "8af0effa-b2b6-4e49-9256-cead146d848c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[5.1, 3.5, 1.4, 0.2],\n",
+       "       [4.9, 3. , 1.4, 0.2],\n",
+       "       [4.7, 3.2, 1.3, 0.2],\n",
+       "       [4.6, 3.1, 1.5, 0.2],\n",
+       "       [5. , 3.6, 1.4, 0.2],\n",
+       "       [5.4, 3.9, 1.7, 0.4],\n",
+       "       [4.6, 3.4, 1.4, 0.3],\n",
+       "       [5. , 3.4, 1.5, 0.2],\n",
+       "       [4.4, 2.9, 1.4, 0.2],\n",
+       "       [4.9, 3.1, 1.5, 0.1],\n",
+       "       [5.4, 3.7, 1.5, 0.2],\n",
+       "       [4.8, 3.4, 1.6, 0.2],\n",
+       "       [4.8, 3. , 1.4, 0.1],\n",
+       "       [4.3, 3. , 1.1, 0.1],\n",
+       "       [5.8, 4. , 1.2, 0.2],\n",
+       "       [5.7, 4.4, 1.5, 0.4],\n",
+       "       [5.4, 3.9, 1.3, 0.4],\n",
+       "       [5.1, 3.5, 1.4, 0.3],\n",
+       "       [5.7, 3.8, 1.7, 0.3],\n",
+       "       [5.1, 3.8, 1.5, 0.3],\n",
+       "       [5.4, 3.4, 1.7, 0.2],\n",
+       "       [5.1, 3.7, 1.5, 0.4],\n",
+       "       [4.6, 3.6, 1. , 0.2],\n",
+       "       [5.1, 3.3, 1.7, 0.5],\n",
+       "       [4.8, 3.4, 1.9, 0.2],\n",
+       "       [5. , 3. , 1.6, 0.2],\n",
+       "       [5. , 3.4, 1.6, 0.4],\n",
+       "       [5.2, 3.5, 1.5, 0.2],\n",
+       "       [5.2, 3.4, 1.4, 0.2],\n",
+       "       [4.7, 3.2, 1.6, 0.2],\n",
+       "       [4.8, 3.1, 1.6, 0.2],\n",
+       "       [5.4, 3.4, 1.5, 0.4],\n",
+       "       [5.2, 4.1, 1.5, 0.1],\n",
+       "       [5.5, 4.2, 1.4, 0.2],\n",
+       "       [4.9, 3.1, 1.5, 0.2],\n",
+       "       [5. , 3.2, 1.2, 0.2],\n",
+       "       [5.5, 3.5, 1.3, 0.2],\n",
+       "       [4.9, 3.6, 1.4, 0.1],\n",
+       "       [4.4, 3. , 1.3, 0.2],\n",
+       "       [5.1, 3.4, 1.5, 0.2],\n",
+       "       [5. , 3.5, 1.3, 0.3],\n",
+       "       [4.5, 2.3, 1.3, 0.3],\n",
+       "       [4.4, 3.2, 1.3, 0.2],\n",
+       "       [5. , 3.5, 1.6, 0.6],\n",
+       "       [5.1, 3.8, 1.9, 0.4],\n",
+       "       [4.8, 3. , 1.4, 0.3],\n",
+       "       [5.1, 3.8, 1.6, 0.2],\n",
+       "       [4.6, 3.2, 1.4, 0.2],\n",
+       "       [5.3, 3.7, 1.5, 0.2],\n",
+       "       [5. , 3.3, 1.4, 0.2],\n",
+       "       [7. , 3.2, 4.7, 1.4],\n",
+       "       [6.4, 3.2, 4.5, 1.5],\n",
+       "       [6.9, 3.1, 4.9, 1.5],\n",
+       "       [5.5, 2.3, 4. , 1.3],\n",
+       "       [6.5, 2.8, 4.6, 1.5],\n",
+       "       [5.7, 2.8, 4.5, 1.3],\n",
+       "       [6.3, 3.3, 4.7, 1.6],\n",
+       "       [4.9, 2.4, 3.3, 1. ],\n",
+       "       [6.6, 2.9, 4.6, 1.3],\n",
+       "       [5.2, 2.7, 3.9, 1.4],\n",
+       "       [5. , 2. , 3.5, 1. ],\n",
+       "       [5.9, 3. , 4.2, 1.5],\n",
+       "       [6. , 2.2, 4. , 1. ],\n",
+       "       [6.1, 2.9, 4.7, 1.4],\n",
+       "       [5.6, 2.9, 3.6, 1.3],\n",
+       "       [6.7, 3.1, 4.4, 1.4],\n",
+       "       [5.6, 3. , 4.5, 1.5],\n",
+       "       [5.8, 2.7, 4.1, 1. ],\n",
+       "       [6.2, 2.2, 4.5, 1.5],\n",
+       "       [5.6, 2.5, 3.9, 1.1],\n",
+       "       [5.9, 3.2, 4.8, 1.8],\n",
+       "       [6.1, 2.8, 4. , 1.3],\n",
+       "       [6.3, 2.5, 4.9, 1.5],\n",
+       "       [6.1, 2.8, 4.7, 1.2],\n",
+       "       [6.4, 2.9, 4.3, 1.3],\n",
+       "       [6.6, 3. , 4.4, 1.4],\n",
+       "       [6.8, 2.8, 4.8, 1.4],\n",
+       "       [6.7, 3. , 5. , 1.7],\n",
+       "       [6. , 2.9, 4.5, 1.5],\n",
+       "       [5.7, 2.6, 3.5, 1. ],\n",
+       "       [5.5, 2.4, 3.8, 1.1],\n",
+       "       [5.5, 2.4, 3.7, 1. ],\n",
+       "       [5.8, 2.7, 3.9, 1.2],\n",
+       "       [6. , 2.7, 5.1, 1.6],\n",
+       "       [5.4, 3. , 4.5, 1.5],\n",
+       "       [6. , 3.4, 4.5, 1.6],\n",
+       "       [6.7, 3.1, 4.7, 1.5],\n",
+       "       [6.3, 2.3, 4.4, 1.3],\n",
+       "       [5.6, 3. , 4.1, 1.3],\n",
+       "       [5.5, 2.5, 4. , 1.3],\n",
+       "       [5.5, 2.6, 4.4, 1.2],\n",
+       "       [6.1, 3. , 4.6, 1.4],\n",
+       "       [5.8, 2.6, 4. , 1.2],\n",
+       "       [5. , 2.3, 3.3, 1. ],\n",
+       "       [5.6, 2.7, 4.2, 1.3],\n",
+       "       [5.7, 3. , 4.2, 1.2],\n",
+       "       [5.7, 2.9, 4.2, 1.3],\n",
+       "       [6.2, 2.9, 4.3, 1.3],\n",
+       "       [5.1, 2.5, 3. , 1.1],\n",
+       "       [5.7, 2.8, 4.1, 1.3]])"
+      ]
+     },
+     "execution_count": 49,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "X"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 51,
+   "id": "868a3cab-e991-4990-9ec5-3e632a41a599",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+       "       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
+       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
+       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])"
+      ]
+     },
+     "execution_count": 51,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "y"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5ab0c188-6476-43c4-b361-a2bfe0ec7a8a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 将数据分为训练集和测试集\n",
+    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
+    "\n",
+    "# 创建逻辑回归模型\n",
+    "model = LogisticRegression()\n",
+    "\n",
+    "# 训练模型\n",
+    "model.fit(X_train, y_train)\n",
+    "\n",
+    "# 在测试集上进行预测\n",
+    "y_pred = model.predict(X_test)\n",
+    "\n",
+    "# 计算准确率\n",
+    "accuracy = accuracy_score(y_test, y_pred)\n",
+    "print(f\"Accuracy: {accuracy * 100:.2f}%\")\n",
+    "\n",
+    "# 输出部分预测结果与真实标签对比\n",
+    "for i in range(5):\n",
+    "    print(f\"True: {y_test[i]}, Predicted: {y_pred[i]}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/.ipynb_checkpoints/5-multi-seq-gpt-checkpoint.ipynb

@@ -0,0 +1,261 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9131f25f-227b-4dbe-b28d-c5006df092c6",
+   "metadata": {},
+   "source": [
+    "# 2.5 基于多模态数据构建大模型"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a30b35c-1f5f-41e6-8fe1-5f522c700e9e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tokenizers import (\n",
+    "    decoders,\n",
+    "    models,\n",
+    "    normalizers,\n",
+    "    pre_tokenizers,\n",
+    "    processors,\n",
+    "    trainers,\n",
+    "    Tokenizer,\n",
+    ")\n",
+    "from transformers import AutoTokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "688fa3b1-f2ca-457a-abde-117c79b54fa9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = Tokenizer(models.BPE())\n",
+    "tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串\n",
+    "trainer = trainers.BpeTrainer(vocab_size=90000, special_tokens=[\"<|endoftext|>\"]) #9w words"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7d680700-1051-4af4-94d6-2ce3071a5979",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.train([\"../01-data_env/data/dna_1g.txt\",\"../01-data_env/data/protein_1g.txt\",\"../01-data_env/data/english_500m.txt\"]\n",
+    "                , trainer=trainer) #all file list, take 10-20 min"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74434ece-2f6e-46fa-9a9e-ff88e9364de8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.save(\"gene_eng_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8ea34e18-6cee-40b9-ba96-d8734153eb9f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#然后我们可以使用from_file() 方法从该文件里重新加载 Tokenizer 对象：\n",
+    "new_tokenizer = Tokenizer.from_file(\"gene_eng_dict.json\")\n",
+    "\n",
+    "#要在 🤗 Transformers 中使用这个标记器，我们必须将它包裹在一个 PreTrainedTokenizerFast 类中\n",
+    "from transformers import GPT2TokenizerFast\n",
+    "gene_eng_tokenizer = GPT2TokenizerFast(tokenizer_object=new_tokenizer)\n",
+    "gene_eng_tokenizer.save_pretrained(\"gene_eng_dict\")\n",
+    "#dna_tokenizer.push_to_hub(\"dna_bpe_dict_1g\", organization=\"dnagpt\", use_auth_token=\"hf_*****\") # push to huggingface"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16c7a3ef-c924-4fbb-b8ab-c12fab43f019",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer_new = AutoTokenizer.from_pretrained('gene_eng_dict')\n",
+    "tokenizer_new.tokenize(\"TGGCGTGAACCCGGGATCGGG,hello world hello gene, MANITWMANHTGWSDFILLGLFRQSKHPALLCVVIFVVFLMAL\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0ca0b2e3-f270-4645-abbb-cb8535e97a0a",
+   "metadata": {},
+   "source": [
+    "## 训练混合模型"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c9b1c9b4-57a8-4711-912d-307e55481f8a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer, GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
+    "from transformers import GPT2Tokenizer,GPT2Model,AutoModel\n",
+    "from transformers import DataCollatorForLanguageModeling\n",
+    "from transformers import Trainer, TrainingArguments\n",
+    "from transformers import LineByLineTextDataset\n",
+    "from tokenizers import Tokenizer\n",
+    "from datasets import load_dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3926a959-4224-4d78-9413-dc47a58087e0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = GPT2Tokenizer.from_pretrained(\"gene_eng_dict\")\n",
+    "tokenizer.pad_token = tokenizer.eos_token"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1c2f5a6d-d405-40dc-a802-f0c1dff50a1e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "max_length = 256 #最大输入长度\n",
+    "\n",
+    "config = AutoConfig.from_pretrained(\n",
+    "    \"gpt2\",\n",
+    "    vocab_size=len(tokenizer),\n",
+    "    n_ctx=max_length, #最大长度\n",
+    "    bos_token_id=tokenizer.bos_token_id,\n",
+    "    eos_token_id=tokenizer.eos_token_id,\n",
+    ")\n",
+    "\n",
+    "model = GPT2LMHeadModel(config) #for pretrain,从头预训练"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c8a47141-56a7-4e41-8cfd-1b381a64e2c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 1. load dna dataset\n",
+    "raw_dataset = load_dataset('text',  \n",
+    "                           data_files=[\"../01-data_env/data/dna_1g.txt\",\"../01-data_env/data/protein_1g.txt\",\"../01-data_env/data/english_500m.txt\"])\n",
+    "\n",
+    "dataset = raw_dataset[\"train\"].train_test_split(test_size=0.05, shuffle=True)\n",
+    "\n",
+    "# 2. tokenize\n",
+    "def tokenize_function(examples):\n",
+    "    return tokenizer(examples['text'], truncation=True, padding='max_length', max_length=max_length)\n",
+    "\n",
+    "# 3. 对数据集应用分词函数\n",
+    "tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=['text'], num_proc=15)  # 设置为你的 CPU 核心数或根据需要调整\n",
+    "\n",
+    "# 4. 创建一个数据收集器，用于动态填充和遮蔽\n",
+    "data_collator = DataCollatorForLanguageModeling(\n",
+    "    tokenizer=tokenizer, mlm=False\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f4f802a2-88e2-49c2-a654-9d6e0996433a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run_path = \"gpt2_run\"\n",
+    "train_epoches = 5\n",
+    "batch_size = 10\n",
+    "\n",
+    "\n",
+    "training_args = TrainingArguments(\n",
+    "        output_dir=run_path,\n",
+    "        overwrite_output_dir=True,\n",
+    "        num_train_epochs=train_epoches,\n",
+    "        per_device_train_batch_size=batch_size,\n",
+    "        save_steps=2000,\n",
+    "        save_total_limit=2,\n",
+    "        prediction_loss_only=True,\n",
+    "        fp16=True, #v100没法用\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    train_dataset=tokenized_datasets[\"train\"],\n",
+    "    eval_dataset=tokenized_datasets[\"test\"],\n",
+    "    data_collator=data_collator,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13fa4a99-ee7c-4d6a-853f-4be04a4ee43c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.train()\n",
+    "trainer.save_model(\"gene_eng_gpt2_v0\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ca452721-3914-49be-a577-d4c257946578",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import math\n",
+    "eval_results = trainer.evaluate()\n",
+    "print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b7e7a455-0e08-4a75-87c1-0f909829b1c1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#upload model\n",
+    "#model.push_to_hub(\"gene_eng_gpt2_v0\", organization=\"dnagpt\", use_auth_token=\"hf_*******\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/1-dna-bpe.ipynb

@@ -0,0 +1,528 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a9fffce5-83e3-4838-8335-acb2e3b50c35",
+   "metadata": {},
+   "source": [
+    "# 2.1 DNA分词器构建"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f28b0950-37dc-4f78-ae6c-9fca33d513fc",
+   "metadata": {},
+   "source": [
+    "## **分词算法**\n",
+    "\n",
+    "### **什么是分词**\n",
+    "分词就是把一个文本序列，分成一个一个的token/词，对于英文这种天生带空格的语言，一般使用空格和标点分词就行了，而对于中文等语言，并没有特殊的符号来分词，因此，一般需要设计专门的分词算法，对于大模型而言，一般需要处理多种语言，因此，也需要专门的分词算法。\n",
+    "\n",
+    "在大模型（如 BERT、GPT 系列、T5 等）中，分词器（tokenizer）扮演着至关重要的角色。它负责将原始文本转换为模型可以处理的格式，即将文本分解成 token 序列，并将这些 token 映射到模型词汇表中的唯一 ID。分词器的选择和配置直接影响模型的性能和效果。以下是几种常见的分词器及其特点，特别关注它们在大型语言模型中的应用。\n",
+    "\n",
+    "### 1. **WordPiece 分词器**\n",
+    "\n",
+    "- **使用场景**：广泛应用于 BERT 及其变体。\n",
+    "- **工作原理**：基于频率统计，从语料库中学习最有效的词汇表。它根据子词（subword）在文本中的出现频率来决定如何分割单词。例如，“playing” 可能被分为 “play” 和 “##ing”，其中“##”表示该部分是前一个 token 的延续。\n",
+    "- **优点**：\n",
+    "  - 处理未知词汇能力强，能够将未见过的词汇分解为已知的子词。\n",
+    "  - 兼容性好，适合多种语言任务。\n",
+    "- **缺点**：\n",
+    "  - 需要额外的标记（如 `##`）来指示子词，可能影响某些应用场景下的可读性。\n",
+    "\n",
+    "### 2. **Byte Pair Encoding (BPE)**\n",
+    "\n",
+    "- **使用场景**：广泛应用于 GPT 系列、RoBERTa、XLM-R 等模型。\n",
+    "- **工作原理**：通过迭代地合并最常见的字符对来构建词汇表。BPE 是一种无监督的学习方法，能够在不依赖于预先定义的词汇表的情况下进行分词。\n",
+    "- **优点**：\n",
+    "  - 灵活性高，适应性强，尤其适用于多语言模型。\n",
+    "  - 不需要特殊标记，生成的词汇表更简洁。\n",
+    "- **缺点**：\n",
+    "  - 对于某些语言或领域特定的词汇，可能会产生较短的子词，导致信息丢失。\n",
+    "\n",
+    "### 3. **SentencePiece**\n",
+    "\n",
+    "- **使用场景**：常见于 T5、mBART 等多语言模型。\n",
+    "- **工作原理**：结合了 BPE 和 WordPiece 的优点，同时支持字符级和词汇级分词。它可以在没有空格的语言（如中文、日文）中表现良好。\n",
+    "- **优点**：\n",
+    "  - 支持无空格语言，适合多语言处理。\n",
+    "  - 学习速度快，适应性强。\n",
+    "- **缺点**：\n",
+    "  - 对于某些特定领域的专业术语，可能需要额外的预处理步骤。\n",
+    "\n",
+    "### 4. **Character-Level Tokenizer**\n",
+    "\n",
+    "- **使用场景**：较少用于大型语言模型，但在某些特定任务（如拼写检查、手写识别）中有应用。\n",
+    "- **工作原理**：直接将每个字符视为一个 token。这种方式简单直接，但通常会导致较大的词汇表。\n",
+    "- **优点**：\n",
+    "  - 简单易实现，不需要复杂的训练过程。\n",
+    "  - 对于字符级别的任务非常有效。\n",
+    "- **缺点**：\n",
+    "  - 词汇表较大，计算资源消耗较多。\n",
+    "  - 捕捉上下文信息的能力较弱。\n",
+    "\n",
+    "### 5. **Unigram Language Model**\n",
+    "\n",
+    "- **使用场景**：主要用于 SentencePiece 中。\n",
+    "- **工作原理**：基于概率分布，选择最优的分词方案以最大化似然函数。这种方法类似于 BPE，但在构建词汇表时考虑了更多的统计信息。\n",
+    "- **优点**：\n",
+    "  - 统计基础强，优化效果好。\n",
+    "  - 适应性强，适用于多种语言和任务。\n",
+    "- **缺点**：\n",
+    "  - 计算复杂度较高，训练时间较长。\n",
+    "\n",
+    "### 分词器的关键特性\n",
+    "\n",
+    "无论选择哪种分词器，以下几个关键特性都是设计和应用中需要考虑的：\n",
+    "\n",
+    "- **词汇表大小**：决定了模型所能识别的词汇量。较大的词汇表可以捕捉更多细节，但也增加了内存和计算需求。\n",
+    "- **处理未知词汇的能力**：好的分词器应该能够有效地处理未登录词（OOV, Out-Of-Vocabulary），将其分解为已知的子词。\n",
+    "- **多语言支持**：对于多语言模型，分词器应能处理不同语言的文本，尤其是那些没有明显分隔符的语言。\n",
+    "- **效率和速度**：分词器的执行速度直接影响整个数据处理管道的效率，尤其是在大规模数据集上。\n",
+    "- **兼容性和灵活性**：分词器应与目标模型架构兼容，并且能够灵活适应不同的任务需求。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "165e2594-277d-44d0-b582-77859a0bc0b2",
+   "metadata": {},
+   "source": [
+    "## DNA等生物序列分词\n",
+    "在生物信息学中，DNA 和蛋白质序列的处理与自然语言处理（NLP）有相似之处，但也有其独特性。为了提取这些生物序列的特征并用于机器学习或深度学习模型，通常需要将长序列分解成更小的片段（类似于 NLP 中的“分词”），以便更好地捕捉局部和全局特征。以下是几种常见的方法，用于对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。\n",
+    "\n",
+    "### 1. **K-mer 分解**\n",
+    "\n",
+    "**定义**：K-mer 是指长度为 k 的连续子序列。例如，在 DNA 序列中，一个 3-mer 可能是 \"ATG\" 或 \"CGA\"。\n",
+    "\n",
+    "**应用**：\n",
+    "- **DNA 序列**：常用的 k 值范围从 3 到 6。较小的 k 值可以捕捉到更细粒度的信息，而较大的 k 值则有助于识别更长的模式。\n",
+    "- **蛋白质序列**：k 值通常较大，因为氨基酸的数量较多（20 种），较长的 k-mer 可以捕捉到重要的结构域或功能区域。\n",
+    "\n",
+    "**优点**：\n",
+    "- 简单且直观，易于实现。\n",
+    "- 可以捕捉到短序列中的局部特征。\n",
+    "\n",
+    "**缺点**：\n",
+    "- 对于非常长的序列，生成的 k-mer 数量会非常大，导致维度爆炸问题。\n",
+    "- 不同位置的 k-mer 之间缺乏上下文关系。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "29c390ef-2e9d-493e-9991-69ecb835b52b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "DNA 3-mers: ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
+      "Protein 4-mers: ['MKQH', 'KQHK', 'QHKA', 'HKAM', 'KAMI', 'AMIV', 'MIVA', 'IVAL', 'VALI', 'ALIV', 'LIVL', 'IVLI', 'VLIT', 'LITA', 'ITAY']\n"
+     ]
+    }
+   ],
+   "source": [
+    "#示例代码（Python）\n",
+    "\n",
+    "def k_mer(seq, k):\n",
+    "    return [seq[i:i+k] for i in range(len(seq) - k + 1)]\n",
+    "\n",
+    "dna_sequence = \"ATGCGTACGTA\"\n",
+    "protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
+    "\n",
+    "print(\"DNA 3-mers:\", k_mer(dna_sequence, 3))\n",
+    "print(\"Protein 4-mers:\", k_mer(protein_sequence, 4))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7ced2bfb-bd42-425a-a3ad-54c9573609c5",
+   "metadata": {},
+   "source": [
+    "### 2. **滑动窗口**\n",
+    "\n",
+    "**定义**：滑动窗口方法通过设定一个固定大小的窗口沿着序列移动，并在每个位置提取窗口内的子序列。这与 K-mer 类似，但允许重叠。\n",
+    "\n",
+    "**应用**：\n",
+    "- **DNA 和蛋白质序列**：窗口大小可以根据具体任务调整，如基因预测、蛋白质结构预测等。\n",
+    "\n",
+    "**优点**：\n",
+    "- 提供了更多的灵活性，可以控制窗口的步长和大小。\n",
+    "- 有助于捕捉局部和全局特征。\n",
+    "\n",
+    "**缺点**：\n",
+    "- 计算复杂度较高，尤其是当窗口大小较大时。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "82cecf91-0076-4c12-b11c-b35120581ef9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Sliding window (DNA, size=3, step=1): ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
+      "Sliding window (Protein, size=4, step=2): ['MKQH', 'QHKA', 'KAMI', 'MIVA', 'VALI', 'LIVL', 'VLIT', 'ITAY']\n"
+     ]
+    }
+   ],
+   "source": [
+    "def sliding_window(seq, window_size, step=1):\n",
+    "    return [seq[i:i+window_size] for i in range(0, len(seq) - window_size + 1, step)]\n",
+    "\n",
+    "dna_sequence = \"ATGCGTACGTA\"\n",
+    "protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
+    "\n",
+    "print(\"Sliding window (DNA, size=3, step=1):\", sliding_window(dna_sequence, 3))\n",
+    "print(\"Sliding window (Protein, size=4, step=2):\", sliding_window(protein_sequence, 4, step=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c33ab920-b451-4846-93d4-20da5a4e1001",
+   "metadata": {},
+   "source": [
+    "### 3. **词表分词和嵌入式表示**\n",
+    "\n",
+    "**定义**：使用预训练的嵌入模型（如 Word2Vec、BERT 等）来将每个 token 映射到高维向量空间中。对于生物序列，可以使用专门设计的嵌入模型，如 ProtTrans、ESM 等。\n",
+    "\n",
+    "**应用**：\n",
+    "- **DNA 和蛋白质序列**：嵌入模型可以捕捉到序列中的语义信息和上下文依赖关系。\n",
+    "\n",
+    "**优点**：\n",
+    "- 捕捉到丰富的语义信息，适合复杂的下游任务。\n",
+    "- 可以利用大规模预训练模型的优势。\n",
+    "\n",
+    "**缺点**：\n",
+    "- 需要大量的计算资源来进行预训练。\n",
+    "- 模型复杂度较高，解释性较差。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "02bf2af0-6077-4b27-8822-f1c3f22914fa",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "import os\n",
+    "# 设置环境变量, autodl一般区域\n",
+    "result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
+    "output = result.stdout\n",
+    "for line in output.splitlines():\n",
+    "    if '=' in line:\n",
+    "        var, value = line.split('=', 1)\n",
+    "        os.environ[var] = value\n",
+    "\n",
+    "\"\"\"\n",
+    "import os\n",
+    "\n",
+    "# 设置环境变量, autodl专区 其他idc\n",
+    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
+    "\n",
+    "# 打印环境变量以确认设置成功\n",
+    "print(os.environ.get('HF_ENDPOINT'))\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "d43b60ee-67f2-4d06-95ea-966c01084fc4",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
+      "To disable this warning, you can either:\n",
+      "\t- Avoid using `tokenizers` before the fork if possible\n",
+      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['ATGCG', 'TACG', 'T', 'A']\n",
+      "Embeddings shape: torch.Size([1, 4, 768])\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "import torch\n",
+    "\n",
+    "# 加载预训练的蛋白质嵌入模型\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
+    "model = AutoModel.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
+    "\n",
+    "dna_sequence = \"ATGCGTACGTA\"\n",
+    "print(tokenizer.tokenize(dna_sequence))\n",
+    "\n",
+    "# 编码序列\n",
+    "inputs = tokenizer(dna_sequence, return_tensors=\"pt\")\n",
+    "\n",
+    "# 获取嵌入\n",
+    "with torch.no_grad():\n",
+    "    outputs = model(**inputs)\n",
+    "    embeddings = outputs.last_hidden_state\n",
+    "\n",
+    "print(\"Embeddings shape:\", embeddings.shape)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c24f10dc-1117-4493-9333-5ed6d898f44a",
+   "metadata": {},
+   "source": [
+    "### 训练DNA BPE分词器\n",
+    "\n",
+    "以上方法展示了如何对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。选择哪种方法取决于具体的任务需求和数据特性。对于简单的分类或回归任务，K-mer 分解或滑动窗口可能是足够的；而对于更复杂的任务，如序列标注或结构预测，基于词汇表的方法或嵌入表示可能会提供更好的性能。\n",
+    "\n",
+    "目前大部分生物序列大模型的论文中，使用最多的依然是传统的K-mer，但一些SOTA的论文则以BEP为主。而BEP分词也是目前GPT、llama等主流自然语言大模型使用的基础分词器。\n",
+    "\n",
+    "因此，我们也演示下从头训练一个DNA BPE分词器的方法。\n",
+    "\n",
+    "我们首先看下GPT2模型，默认的分词器，对DNA序列分词的结果："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "43f1eb8b-1cc2-4ab5-aa8e-2a63132be98c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tokenizers import (\n",
+    "    decoders,\n",
+    "    models,\n",
+    "    normalizers,\n",
+    "    pre_tokenizers,\n",
+    "    processors,\n",
+    "    trainers,\n",
+    "    Tokenizer,\n",
+    ")\n",
+    "from transformers import AutoTokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "27e88f7b-1399-418b-9b91-f970762fac0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gpt2_tokenizer = AutoTokenizer.from_pretrained('gpt2')\n",
+    "gpt2_tokenizer.pad_token = gpt2_tokenizer.eos_token"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "4b015db7-63ba-4909-b02f-07634b3d5584",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['T', 'GG', 'C', 'GT', 'GA', 'AC', 'CC', 'GG', 'G', 'AT', 'C', 'GG', 'G']"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gpt2_tokenizer.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a246fbc9-9e29-4b63-bdf7-f80635d06d1e",
+   "metadata": {},
+   "source": [
+    "可以看到，gpt2模型因为是以英文为主的BPE分词模型，分解的都是1到2个字母的结果，这样显然很难充分表达生物语义，因此，我们使用DNA序列来训练1个BPE分词器，代码也非常简单："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "8357a695-1c29-4b5c-8099-d2e337189410",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = Tokenizer(models.BPE())\n",
+    "tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串\n",
+    "trainer = trainers.BpeTrainer(vocab_size=30000, special_tokens=[\"<|endoftext|>\"]) #3w words"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "32c95888-1498-45cf-8453-421219cc7d45",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "tokenizer.train([\"../01-data_env/data/dna_1g.txt\"], trainer=trainer) #all file list, take 10-20 min"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "5ffdd717-72ed-4a37-bafc-b4a0f61f8ff1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']\n"
+     ]
+    }
+   ],
+   "source": [
+    "encoding = tokenizer.encode(\"TGGCGTGAACCCGGGATCGGG\")\n",
+    "print(encoding.tokens)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a96e7838-6c23-4446-bf86-b098cd93214a",
+   "metadata": {},
+   "source": [
+    "可以看到，以DNA数据训练的分词器，分词效果明显要好的多，各种长度的词都有。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "f1d757c1-702b-4147-9207-471f422f67b2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.save(\"dna_bpe_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "caf8ecea-359e-487b-b456-fab546b9da0d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#然后我们可以使用from_file() 方法从该文件里重新加载 Tokenizer 对象：\n",
+    "new_tokenizer = Tokenizer.from_file(\"dna_bpe_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "8ec6f045-bc30-4012-8027-a879df8def3a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "('dna_bpe_dict/tokenizer_config.json',\n",
+       " 'dna_bpe_dict/special_tokens_map.json',\n",
+       " 'dna_bpe_dict/vocab.json',\n",
+       " 'dna_bpe_dict/merges.txt',\n",
+       " 'dna_bpe_dict/added_tokens.json',\n",
+       " 'dna_bpe_dict/tokenizer.json')"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "#要在 🤗 Transformers 中使用这个标记器，我们必须将它包裹在一个 PreTrainedTokenizerFast 类中\n",
+    "from transformers import GPT2TokenizerFast\n",
+    "dna_tokenizer = GPT2TokenizerFast(tokenizer_object=new_tokenizer)\n",
+    "dna_tokenizer.save_pretrained(\"dna_bpe_dict\")\n",
+    "#dna_tokenizer.push_to_hub(\"dna_bpe_dict_1g\", organization=\"dnagpt\", use_auth_token=\"hf_*****\") # push to huggingface"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "f84506d8-6208-4027-aad7-2b68a1bc16d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer_new = AutoTokenizer.from_pretrained('dna_bpe_dict')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "d40d4d53-6fed-445c-afb5-c0346ab854c8",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tokenizer_new.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "640302f6-f740-41a4-ae92-ca4c43d97493",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/4-gene-feature.ipynb

@@ -0,0 +1,489 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "b1b37ca8-25a3-440c-9b68-7f72ce670ade",
+   "metadata": {},
+   "source": [
+    "# 2.4 基因大模型的生物序列特征提取"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d3d04215-2b6c-41fb-92a4-90c82d322ba4",
+   "metadata": {},
+   "source": [
+    "使用 GPT-2 模型获取文本的特征向量是一个常见的需求，尤其是在进行文本分类、相似度计算或其他下游任务时。Hugging Face 的 transformers 库提供了简单易用的接口来实现这一点。以下是详细的步骤和代码示例，帮助你从 GPT-2 模型中提取文本的特征向量。"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3ff5b7c6-e57c-4839-8510-f764154faa65",
+   "metadata": {},
+   "source": [
+    "使用 GPT-2 模型获取文本的特征向量是一个常见的需求，尤其是在进行文本分类、相似度计算或其他下游任务时。Hugging Face 的 `transformers` 库提供了简单易用的接口来实现这一点。以下是详细的步骤和代码示例，帮助你从 GPT-2 模型中提取文本的特征向量。\n",
+    "\n",
+    "### 方法 1: 使用隐藏状态（Hidden States）\n",
+    "\n",
+    "GPT-2 是一个基于 Transformer 的语言模型，它在每一层都有隐藏状态（hidden states），这些隐藏状态可以作为文本的特征表示。你可以选择最后一层的隐藏状态作为最终的特征向量，或者对多层的隐藏状态进行平均或拼接。\n",
+    "\n",
+    "\n",
+    "### 方法 2: 使用池化策略\n",
+    "\n",
+    "另一种方法是通过对所有 token 的隐藏状态进行池化操作来获得句子级别的特征向量。常见的池化方法包括：\n",
+    "\n",
+    "- **均值池化**（Mean Pooling）：对所有 token 的隐藏状态求平均。\n",
+    "- **最大池化**（Max Pooling）：对每个维度取最大值。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "e7fe053b-d6da-488a-9c62-24e4b40a992d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'input_ids': tensor([[    1,   191,    29,   753,  1241,  2104, 12297,   357,    85,  4395,\n",
+      "         26392,    16]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "tokenizer = AutoTokenizer.from_pretrained('dna_bpe_dict')\n",
+    "tokenizer.tokenize(\"GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG\")\n",
+    "#result: [G','AGCAC','ATTCGCC',....]\n",
+    "\n",
+    "model = AutoModel.from_pretrained('dna_gpt2_v0')\n",
+    "import torch\n",
+    "dna = \"ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC\"\n",
+    "inputs = tokenizer(dna, return_tensors = 'pt')\n",
+    "print(inputs)\n",
+    "\n",
+    "outputs = model(inputs[\"input_ids\"])\n",
+    "#outputs = model(**inputs)\n",
+    "\n",
+    "hidden_states = outputs.last_hidden_state # [1, sequence_length, 768]  outputs.last_hidden_state or outputs[0]\n",
+    "\n",
+    "# embedding with mean pooling\n",
+    "embedding_mean = torch.mean(hidden_states[0], dim=0)\n",
+    "print(embedding_mean.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with max pooling\n",
+    "embedding_max = torch.max(hidden_states[0], dim=0)[0]\n",
+    "print(embedding_max.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with first token\n",
+    "embedding_first_token = hidden_states[0][0]\n",
+    "print(embedding_first_token.shape) # expect to be 768"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "id": "a1f2b545-283a-4613-a953-beb82f427826",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Some weights of BertModel were not initialized from the model checkpoint at dna_bert_v0 and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
+      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'input_ids': tensor([[    6,   200, 16057,    10,  1256,  2123, 12294,   366, 13138,  7826,\n",
+      "            82,    25]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n",
+      "torch.Size([768])\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoTokenizer, AutoModel\n",
+    "import torch\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained('dna_wordpiece_dict')\n",
+    "tokenizer.tokenize(\"GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG\")\n",
+    "#result: [G','AGCAC','ATTCGCC',....]\n",
+    "\n",
+    "model = AutoModel.from_pretrained('dna_bert_v0')\n",
+    "dna = \"ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC\"\n",
+    "inputs = tokenizer(dna, return_tensors = 'pt')\n",
+    "print(inputs)\n",
+    "\n",
+    "outputs = model(inputs[\"input_ids\"])\n",
+    "#outputs = model(**inputs)\n",
+    "\n",
+    "hidden_states = outputs.last_hidden_state # [1, sequence_length, 768]  outputs.last_hidden_state or outputs[0]\n",
+    "\n",
+    "# embedding with mean pooling\n",
+    "embedding_mean = torch.mean(hidden_states[0], dim=0)\n",
+    "print(embedding_mean.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with max pooling\n",
+    "embedding_max = torch.max(hidden_states[0], dim=0)[0]\n",
+    "print(embedding_max.shape) # expect to be 768\n",
+    "\n",
+    "# embedding with first token\n",
+    "embedding_first_token = hidden_states[0][0]\n",
+    "print(embedding_first_token.shape) # expect to be 768"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56761874-9af7-4b90-aa8b-131e5b8c69b6",
+   "metadata": {},
+   "source": [
+    "## 特征提取并分类\n",
+    "\n",
+    "我们使用第一章中的\"dnagpt/dna_core_promoter\"数据集，演示下使用我们训练的DNA GPT2或者DNA bert模型，提取序列特征，然使用最基础的逻辑回归分类方法，对序列进行分类。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "id": "f1ca177c-a80f-48a1-b2f9-16c13b3350db",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
+      ]
+     },
+     "execution_count": 45,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import subprocess\n",
+    "import os\n",
+    "# 设置环境变量, autodl一般区域\n",
+    "result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
+    "output = result.stdout\n",
+    "for line in output.splitlines():\n",
+    "    if '=' in line:\n",
+    "        var, value = line.split('=', 1)\n",
+    "        os.environ[var] = value\n",
+    "\n",
+    "#或者\n",
+    "\"\"\"\n",
+    "import os\n",
+    "\n",
+    "# 设置环境变量, autodl专区 其他idc\n",
+    "os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
+    "\n",
+    "# 打印环境变量以确认设置成功\n",
+    "print(os.environ.get('HF_ENDPOINT'))\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "id": "2295739c-e80a-47be-9400-88bfab4b0bb6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "DatasetDict({\n",
+       "    train: Dataset({\n",
+       "        features: ['sequence', 'label'],\n",
+       "        num_rows: 59196\n",
+       "    })\n",
+       "})"
+      ]
+     },
+     "execution_count": 46,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from datasets import load_dataset\n",
+    "dna_data = load_dataset(\"dnagpt/dna_core_promoter\")\n",
+    "dna_data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c804bced-f151-43a7-8a95-156db358da3e",
+   "metadata": {},
+   "source": [
+    "这里，我们不需要关注这个数据的具体生物学含义，只需知道sequence是具体的DNA序列，label是分类标签，有两个类别0和1即可"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "id": "9a47a1b1-21f2-4d71-801c-50f88e326ed3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'sequence': 'CATGCGGGTCGATATCCTATCTGAATCTCTCAGCCCAAGAGGGAGTCCGCTCATCTATTCGGCAGTACTG',\n",
+       " 'label': 0}"
+      ]
+     },
+     "execution_count": 47,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dna_data[\"train\"][0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cde7986d-a225-41ca-8f11-614d079fd2bf",
+   "metadata": {},
+   "source": [
+    "这里使用scikit-learn库来构建逻辑回归分类器。首先是特征提取："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 52,
+   "id": "4010d991-056a-43ce-8cca-30eeec8678f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "from sklearn.datasets import load_iris\n",
+    "from sklearn.metrics import accuracy_score\n",
+    "\n",
+    "\n",
+    "def get_gpt2_feature(sequence):\n",
+    "    return \n",
+    "\n",
+    "# 加载数据集\n",
+    "data = load_iris()\n",
+    "X = data.data[data.target < 2]  # 只选择前两个类别\n",
+    "y = data.target[data.target < 2]\n",
+    "\n",
+    "X = []\n",
+    "Y = []\n",
+    "\n",
+    "for item in dna_data[\"train\"]:\n",
+    "    sequence = item[\"sequence\"]\n",
+    "    label = item[\"label\"]\n",
+    "    x_v = get_gpt2_feature(sequence)\n",
+    "    y_v = label\n",
+    "    X.append(x_v)\n",
+    "    Y.append(y_v)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 49,
+   "id": "8af0effa-b2b6-4e49-9256-cead146d848c",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[5.1, 3.5, 1.4, 0.2],\n",
+       "       [4.9, 3. , 1.4, 0.2],\n",
+       "       [4.7, 3.2, 1.3, 0.2],\n",
+       "       [4.6, 3.1, 1.5, 0.2],\n",
+       "       [5. , 3.6, 1.4, 0.2],\n",
+       "       [5.4, 3.9, 1.7, 0.4],\n",
+       "       [4.6, 3.4, 1.4, 0.3],\n",
+       "       [5. , 3.4, 1.5, 0.2],\n",
+       "       [4.4, 2.9, 1.4, 0.2],\n",
+       "       [4.9, 3.1, 1.5, 0.1],\n",
+       "       [5.4, 3.7, 1.5, 0.2],\n",
+       "       [4.8, 3.4, 1.6, 0.2],\n",
+       "       [4.8, 3. , 1.4, 0.1],\n",
+       "       [4.3, 3. , 1.1, 0.1],\n",
+       "       [5.8, 4. , 1.2, 0.2],\n",
+       "       [5.7, 4.4, 1.5, 0.4],\n",
+       "       [5.4, 3.9, 1.3, 0.4],\n",
+       "       [5.1, 3.5, 1.4, 0.3],\n",
+       "       [5.7, 3.8, 1.7, 0.3],\n",
+       "       [5.1, 3.8, 1.5, 0.3],\n",
+       "       [5.4, 3.4, 1.7, 0.2],\n",
+       "       [5.1, 3.7, 1.5, 0.4],\n",
+       "       [4.6, 3.6, 1. , 0.2],\n",
+       "       [5.1, 3.3, 1.7, 0.5],\n",
+       "       [4.8, 3.4, 1.9, 0.2],\n",
+       "       [5. , 3. , 1.6, 0.2],\n",
+       "       [5. , 3.4, 1.6, 0.4],\n",
+       "       [5.2, 3.5, 1.5, 0.2],\n",
+       "       [5.2, 3.4, 1.4, 0.2],\n",
+       "       [4.7, 3.2, 1.6, 0.2],\n",
+       "       [4.8, 3.1, 1.6, 0.2],\n",
+       "       [5.4, 3.4, 1.5, 0.4],\n",
+       "       [5.2, 4.1, 1.5, 0.1],\n",
+       "       [5.5, 4.2, 1.4, 0.2],\n",
+       "       [4.9, 3.1, 1.5, 0.2],\n",
+       "       [5. , 3.2, 1.2, 0.2],\n",
+       "       [5.5, 3.5, 1.3, 0.2],\n",
+       "       [4.9, 3.6, 1.4, 0.1],\n",
+       "       [4.4, 3. , 1.3, 0.2],\n",
+       "       [5.1, 3.4, 1.5, 0.2],\n",
+       "       [5. , 3.5, 1.3, 0.3],\n",
+       "       [4.5, 2.3, 1.3, 0.3],\n",
+       "       [4.4, 3.2, 1.3, 0.2],\n",
+       "       [5. , 3.5, 1.6, 0.6],\n",
+       "       [5.1, 3.8, 1.9, 0.4],\n",
+       "       [4.8, 3. , 1.4, 0.3],\n",
+       "       [5.1, 3.8, 1.6, 0.2],\n",
+       "       [4.6, 3.2, 1.4, 0.2],\n",
+       "       [5.3, 3.7, 1.5, 0.2],\n",
+       "       [5. , 3.3, 1.4, 0.2],\n",
+       "       [7. , 3.2, 4.7, 1.4],\n",
+       "       [6.4, 3.2, 4.5, 1.5],\n",
+       "       [6.9, 3.1, 4.9, 1.5],\n",
+       "       [5.5, 2.3, 4. , 1.3],\n",
+       "       [6.5, 2.8, 4.6, 1.5],\n",
+       "       [5.7, 2.8, 4.5, 1.3],\n",
+       "       [6.3, 3.3, 4.7, 1.6],\n",
+       "       [4.9, 2.4, 3.3, 1. ],\n",
+       "       [6.6, 2.9, 4.6, 1.3],\n",
+       "       [5.2, 2.7, 3.9, 1.4],\n",
+       "       [5. , 2. , 3.5, 1. ],\n",
+       "       [5.9, 3. , 4.2, 1.5],\n",
+       "       [6. , 2.2, 4. , 1. ],\n",
+       "       [6.1, 2.9, 4.7, 1.4],\n",
+       "       [5.6, 2.9, 3.6, 1.3],\n",
+       "       [6.7, 3.1, 4.4, 1.4],\n",
+       "       [5.6, 3. , 4.5, 1.5],\n",
+       "       [5.8, 2.7, 4.1, 1. ],\n",
+       "       [6.2, 2.2, 4.5, 1.5],\n",
+       "       [5.6, 2.5, 3.9, 1.1],\n",
+       "       [5.9, 3.2, 4.8, 1.8],\n",
+       "       [6.1, 2.8, 4. , 1.3],\n",
+       "       [6.3, 2.5, 4.9, 1.5],\n",
+       "       [6.1, 2.8, 4.7, 1.2],\n",
+       "       [6.4, 2.9, 4.3, 1.3],\n",
+       "       [6.6, 3. , 4.4, 1.4],\n",
+       "       [6.8, 2.8, 4.8, 1.4],\n",
+       "       [6.7, 3. , 5. , 1.7],\n",
+       "       [6. , 2.9, 4.5, 1.5],\n",
+       "       [5.7, 2.6, 3.5, 1. ],\n",
+       "       [5.5, 2.4, 3.8, 1.1],\n",
+       "       [5.5, 2.4, 3.7, 1. ],\n",
+       "       [5.8, 2.7, 3.9, 1.2],\n",
+       "       [6. , 2.7, 5.1, 1.6],\n",
+       "       [5.4, 3. , 4.5, 1.5],\n",
+       "       [6. , 3.4, 4.5, 1.6],\n",
+       "       [6.7, 3.1, 4.7, 1.5],\n",
+       "       [6.3, 2.3, 4.4, 1.3],\n",
+       "       [5.6, 3. , 4.1, 1.3],\n",
+       "       [5.5, 2.5, 4. , 1.3],\n",
+       "       [5.5, 2.6, 4.4, 1.2],\n",
+       "       [6.1, 3. , 4.6, 1.4],\n",
+       "       [5.8, 2.6, 4. , 1.2],\n",
+       "       [5. , 2.3, 3.3, 1. ],\n",
+       "       [5.6, 2.7, 4.2, 1.3],\n",
+       "       [5.7, 3. , 4.2, 1.2],\n",
+       "       [5.7, 2.9, 4.2, 1.3],\n",
+       "       [6.2, 2.9, 4.3, 1.3],\n",
+       "       [5.1, 2.5, 3. , 1.1],\n",
+       "       [5.7, 2.8, 4.1, 1.3]])"
+      ]
+     },
+     "execution_count": 49,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "X"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 51,
+   "id": "868a3cab-e991-4990-9ec5-3e632a41a599",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+       "       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
+       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
+       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])"
+      ]
+     },
+     "execution_count": 51,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "y"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5ab0c188-6476-43c4-b361-a2bfe0ec7a8a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 将数据分为训练集和测试集\n",
+    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
+    "\n",
+    "# 创建逻辑回归模型\n",
+    "model = LogisticRegression()\n",
+    "\n",
+    "# 训练模型\n",
+    "model.fit(X_train, y_train)\n",
+    "\n",
+    "# 在测试集上进行预测\n",
+    "y_pred = model.predict(X_test)\n",
+    "\n",
+    "# 计算准确率\n",
+    "accuracy = accuracy_score(y_test, y_pred)\n",
+    "print(f\"Accuracy: {accuracy * 100:.2f}%\")\n",
+    "\n",
+    "# 输出部分预测结果与真实标签对比\n",
+    "for i in range(5):\n",
+    "    print(f\"True: {y_test[i]}, Predicted: {y_pred[i]}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/5-multi-seq-gpt.ipynb

@@ -0,0 +1,261 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9131f25f-227b-4dbe-b28d-c5006df092c6",
+   "metadata": {},
+   "source": [
+    "# 2.5 基于多模态数据构建大模型"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a30b35c-1f5f-41e6-8fe1-5f522c700e9e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tokenizers import (\n",
+    "    decoders,\n",
+    "    models,\n",
+    "    normalizers,\n",
+    "    pre_tokenizers,\n",
+    "    processors,\n",
+    "    trainers,\n",
+    "    Tokenizer,\n",
+    ")\n",
+    "from transformers import AutoTokenizer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "688fa3b1-f2ca-457a-abde-117c79b54fa9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = Tokenizer(models.BPE())\n",
+    "tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串\n",
+    "trainer = trainers.BpeTrainer(vocab_size=90000, special_tokens=[\"<|endoftext|>\"]) #9w words"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7d680700-1051-4af4-94d6-2ce3071a5979",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.train([\"../01-data_env/data/dna_1g.txt\",\"../01-data_env/data/protein_1g.txt\",\"../01-data_env/data/english_500m.txt\"]\n",
+    "                , trainer=trainer) #all file list, take 10-20 min"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74434ece-2f6e-46fa-9a9e-ff88e9364de8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer.save(\"gene_eng_dict.json\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8ea34e18-6cee-40b9-ba96-d8734153eb9f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#然后我们可以使用from_file() 方法从该文件里重新加载 Tokenizer 对象：\n",
+    "new_tokenizer = Tokenizer.from_file(\"gene_eng_dict.json\")\n",
+    "\n",
+    "#要在 🤗 Transformers 中使用这个标记器，我们必须将它包裹在一个 PreTrainedTokenizerFast 类中\n",
+    "from transformers import GPT2TokenizerFast\n",
+    "gene_eng_tokenizer = GPT2TokenizerFast(tokenizer_object=new_tokenizer)\n",
+    "gene_eng_tokenizer.save_pretrained(\"gene_eng_dict\")\n",
+    "#dna_tokenizer.push_to_hub(\"dna_bpe_dict_1g\", organization=\"dnagpt\", use_auth_token=\"hf_*****\") # push to huggingface"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16c7a3ef-c924-4fbb-b8ab-c12fab43f019",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer_new = AutoTokenizer.from_pretrained('gene_eng_dict')\n",
+    "tokenizer_new.tokenize(\"TGGCGTGAACCCGGGATCGGG,hello world hello gene, MANITWMANHTGWSDFILLGLFRQSKHPALLCVVIFVVFLMAL\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0ca0b2e3-f270-4645-abbb-cb8535e97a0a",
+   "metadata": {},
+   "source": [
+    "## 训练混合模型"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c9b1c9b4-57a8-4711-912d-307e55481f8a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer, GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
+    "from transformers import GPT2Tokenizer,GPT2Model,AutoModel\n",
+    "from transformers import DataCollatorForLanguageModeling\n",
+    "from transformers import Trainer, TrainingArguments\n",
+    "from transformers import LineByLineTextDataset\n",
+    "from tokenizers import Tokenizer\n",
+    "from datasets import load_dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3926a959-4224-4d78-9413-dc47a58087e0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = GPT2Tokenizer.from_pretrained(\"gene_eng_dict\")\n",
+    "tokenizer.pad_token = tokenizer.eos_token"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1c2f5a6d-d405-40dc-a802-f0c1dff50a1e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "max_length = 256 #最大输入长度\n",
+    "\n",
+    "config = AutoConfig.from_pretrained(\n",
+    "    \"gpt2\",\n",
+    "    vocab_size=len(tokenizer),\n",
+    "    n_ctx=max_length, #最大长度\n",
+    "    bos_token_id=tokenizer.bos_token_id,\n",
+    "    eos_token_id=tokenizer.eos_token_id,\n",
+    ")\n",
+    "\n",
+    "model = GPT2LMHeadModel(config) #for pretrain,从头预训练"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c8a47141-56a7-4e41-8cfd-1b381a64e2c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 1. load dna dataset\n",
+    "raw_dataset = load_dataset('text',  \n",
+    "                           data_files=[\"../01-data_env/data/dna_1g.txt\",\"../01-data_env/data/protein_1g.txt\",\"../01-data_env/data/english_500m.txt\"])\n",
+    "\n",
+    "dataset = raw_dataset[\"train\"].train_test_split(test_size=0.05, shuffle=True)\n",
+    "\n",
+    "# 2. tokenize\n",
+    "def tokenize_function(examples):\n",
+    "    return tokenizer(examples['text'], truncation=True, padding='max_length', max_length=max_length)\n",
+    "\n",
+    "# 3. 对数据集应用分词函数\n",
+    "tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=['text'], num_proc=15)  # 设置为你的 CPU 核心数或根据需要调整\n",
+    "\n",
+    "# 4. 创建一个数据收集器，用于动态填充和遮蔽\n",
+    "data_collator = DataCollatorForLanguageModeling(\n",
+    "    tokenizer=tokenizer, mlm=False\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f4f802a2-88e2-49c2-a654-9d6e0996433a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run_path = \"gpt2_run\"\n",
+    "train_epoches = 5\n",
+    "batch_size = 10\n",
+    "\n",
+    "\n",
+    "training_args = TrainingArguments(\n",
+    "        output_dir=run_path,\n",
+    "        overwrite_output_dir=True,\n",
+    "        num_train_epochs=train_epoches,\n",
+    "        per_device_train_batch_size=batch_size,\n",
+    "        save_steps=2000,\n",
+    "        save_total_limit=2,\n",
+    "        prediction_loss_only=True,\n",
+    "        fp16=True, #v100没法用\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    train_dataset=tokenized_datasets[\"train\"],\n",
+    "    eval_dataset=tokenized_datasets[\"test\"],\n",
+    "    data_collator=data_collator,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13fa4a99-ee7c-4d6a-853f-4be04a4ee43c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.train()\n",
+    "trainer.save_model(\"gene_eng_gpt2_v0\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ca452721-3914-49be-a577-d4c257946578",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import math\n",
+    "eval_results = trainer.evaluate()\n",
+    "print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b7e7a455-0e08-4a75-87c1-0f909829b1c1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#upload model\n",
+    "#model.push_to_hub(\"gene_eng_gpt2_v0\", organization=\"dnagpt\", use_auth_token=\"hf_*******\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

02-gpt2_bert/dna_bert_v0/config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "bert-base-uncased",
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 2,
+  "classifier_dropout": null,
+  "eos_token_id": 3,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 256,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.47.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30000
+}

02-gpt2_bert/dna_bert_v0/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "pad_token_id": 0,
+  "transformers_version": "4.47.1"
+}

02-gpt2_bert/dna_bert_v0/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4ddcda268020f781f23a80f4c0e6ab047189dd5f1cf61418429b08ad7ede68c
+size 435688784

02-gpt2_bert/dna_bert_v0/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c59e1911412cf624ecf026e2f285ee02c53693e2822a16004df0254bd63561d
+size 5304

02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/merges-checkpoint.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c79d89982c6ffe11f99a9830590377eba204aa277e9e00da4b44db9a758babd
+size 323115

02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/special_tokens_map-checkpoint.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/tokenizer_config-checkpoint.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

02-gpt2_bert/dna_bpe_dict/merges.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c79d89982c6ffe11f99a9830590377eba204aa277e9e00da4b44db9a758babd
+size 323115

02-gpt2_bert/dna_bpe_dict/special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

02-gpt2_bert/dna_bpe_dict/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

02-gpt2_bert/dna_gpt2_v0/config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 0,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 256,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.47.1",
+  "use_cache": true,
+  "vocab_size": 30000
+}

02-gpt2_bert/dna_gpt2_v0/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.47.1"
+}

02-gpt2_bert/dna_gpt2_v0/merges.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c79d89982c6ffe11f99a9830590377eba204aa277e9e00da4b44db9a758babd
+size 323115

02-gpt2_bert/dna_gpt2_v0/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4df0047ff1933c1cae760e550e58e593b9da1b9ad3cd9461bc665e9c17bc0416
+size 435544704

02-gpt2_bert/dna_gpt2_v0/special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

02-gpt2_bert/dna_gpt2_v0/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

02-gpt2_bert/dna_gpt2_v0/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bcbeec6a01e5f602600f5a54d50cadadebc1e9685dec8b7b5998b4798129b24b
+size 5304

02-gpt2_bert/dna_wordpiece_dict/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

02-gpt2_bert/dna_wordpiece_dict/tokenizer_config.json

@@ -0,0 +1,53 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}