Upload folder using huggingface_hub

README.md

@@ -1,3 +1,112 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- lmms-lab/LLaVA-OneVision-Data
+- BAAI/Infinity-MM
+language:
+- en
+- zh
+base_model:
+- apple/aimv2-huge-patch14-448
+- Qwen/Qwen2-1.5B-Instruct
+pipeline_tag: image-text-to-text
+library_name: transformers
+---
+
+# FlashVL-2B-Dynamic
+[\[📜 FlashVL\]](https://www.arxiv.org/abs/2505.09498)
+
+![image/png](https://s3plus.meituan.net/automl-datasets/mlm/logo.jpg)
+
+## Introduction
+
+We are excited to introduce **FlashVL**, a novel approach to optimizing Vision-Language Models (VLMs) for real-time applications, targeting ultra-low latency and high throughput without sacrificing accuracy. Leveraging advanced architectural enhancements and efficient computational strategies, Flash-VL 2B is designed to maximize throughput by reducing processing time while maintaining competitive performance across multiple vision-language benchmarks. Our approach includes tailored architectural choices, token compression mechanisms, data curation, training schemes, and a novel image processing technique called implicit semantic stitching that effectively balances computational load and model performance. Through extensive evaluations on 11 standard VLM benchmarks, we demonstrate that Flash-VL 2B achieves state-of-the-art results in both speed and accuracy, making it a promising solution for deployment in resource-constrained environments and large-scale real-time applications.
+
+
+### Environment Setup
+
+```bash
+pip install torch==2.1.2
+pip install transformers==4.50.0.dev0
+```
+
+
+### How to use it?
+
+```python
+import torch
+from PIL import Image
+import requests
+from io import BytesIO
+from transformers import AutoModel, AutoTokenizer, CLIPImageProcessor
+
+model_path = "FlashVL/FlashVL-2B-Dynamic"
+model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,trust_remote_code=True,device_map='cuda')
+model.tokenizer = AutoTokenizer.from_pretrained(model_path,device_map='cuda')
+model.im_trans = CLIPImageProcessor.from_pretrained(model_path)
+
+# single-image single-round conversation (单图单轮对话)
+image_url ="https://s3plus.meituan.net/automl-datasets/mlm/0516.png"
+response = requests.get(image_url)
+image_data = BytesIO(response.content)
+pil_image = Image.open(image_data).convert('RGB')   
+messages = [{'role': 'user', 'content': "生成图中菜品的菜谱"}] # answer: EXTRA
+answer = model.chat(pil_image, messages, do_sample=False, max_new_tokens=256)
+print(answer)
+
+# single-image multi-round conversation (单图多轮对话)
+messages = [
+    {'role': 'user', 'content': '这是什么'},
+    {"role": "assistant", "content": '这是一道看起来像是银耳莲子汤的甜品。\
+     银耳是一种常见的食材，通常用于制作甜品和汤品，具有软糯的口感和清润的口感。莲 \
+     子是莲子的干燥部分，常用于中医和食疗中，具有补脾止泻的功效。图片中还可以看到 \
+     一些枸杞和核桃，枸杞富含维生素和抗氧化物质，核桃则提供丰富的蛋白质和健康脂肪。 \
+     整体来看，这道甜品不仅美味，还具有一定的营养价值。'},
+    {'role': 'user', 'content': '对图中菜品卡路里分析'}
+    ]
+answer = model.chat(pil_image, messages, do_sample=False, max_new_tokens=512)
+print(answer)
+
+# pure-text single-round conversation (纯文本对话）
+messages = [{'role': 'user', 'content': "who are you"}]
+answer = model.chat(None, messages, do_sample=False, max_new_tokens=256)
+print(answer)
+
+```
+
+### Evaluation
+
+| Benchmark | Qwen2-VL-2B | Aquila-VL-2B | InternVL2.5-2B | Flash-VL-2B<sub>s<sub> | Flash-VL-2B<sub>d<sub> | Flash-VL-2B<sub>d-ISS<sub> |
+| :-------------: | :-------------: | :-------------: | :-------------: |:-------------: |:-------------: |:-------------: |
+| MMMU<sub>val<sub>                | 41.9 | 44.4 | 41.8 | 43.6 | 42.9 | 42.9 |
+| MMBench<sup>en<sup>              | 74.9 | 78.6 | 74.7 | 78.4 | 78.4 | 79.1 |
+| MMBench<sup>cn<sup>              | 73.5 | 76.3 | 71.6 | 74.7 | 74.9 | 76.7 |
+| MMStar                           | 48.0 | 54.9 | 54.1 | 53.8 | 54.4 | 54.1 |
+| MathVista<sub>testmini<sub>      | 43.0 | 59.4 | 50.9 | 59.3 | 58.1 | 61.5 |
+| AI2D<sub>test<sub>               | 74.1 | 75.0 | 75.1 | 74.2 | 74.1 | 74.4 |
+| MMVet                            | 49.5 | 40.9 | 61.7 | 47.3 | 52.7 | 50.7 |
+| HallusionBench                   | 39.2 | 38.5 | 42.7 | 43.5 | 45.5 | 49.0 |
+| OCRBench                         |  794 |  773 | 800  | 764  | 831  | 843  |
+| MME                              | 1872 | 1813 | 2091 | 1715 | 1866 | 1850 |
+| SEEDBench                        | 71.5 | 78.9 | 73.2 | 73.6 | 73.6 | 74.5 |
+| Average                          | 60.2 | 62.6 | 63.6 | 62.4 | 64.0 | 64.8 |
+
+
+We use [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) to evaluate FlashVL-2B-Static. 
+
+
+
+## Citation
+If you find this project useful in your research, please consider citing:
+
+```BibTeX
+@misc{zhang2025flashvl2boptimizingvisionlanguage,
+      title={Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughput}, 
+      author={Bo Zhang and Shuo Li and Runhe Tian and Yang Yang and Jixin Tang and Jinhao Zhou and Lin Ma},
+      year={2025},
+      eprint={2505.09498},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2505.09498}, 
+}
+```

adapters.py

@@ -0,0 +1,61 @@
+import os
+import math
+import torch
+from torch import nn
+from functools import partial
+import torch.nn.functional as F
+
+
+class Adapter_Template(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.gradient_checkpointing = False
+    
+    def freeze_module(self, module):
+        for p in module.parameters():
+            p.requires_grad = False
+
+    def forward(self, inputs, add_start_end=True):
+        input_ids, hidden_states, targets, attn_mask, loss_mask = inputs
+        image_features = self.forward_adapter_modules(hidden_states)
+        return (input_ids, image_features, targets, attn_mask, loss_mask)
+    
+
+class Adapter_AIM(Adapter_Template):
+    
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.p0 = nn.Sequential(
+            nn.LayerNorm(config.vision_config.hidden_size*4),
+            nn.Linear(config.vision_config.hidden_size*4, config.intermediate_size),
+            nn.GELU(),
+            nn.Linear(config.intermediate_size, config.intermediate_size),
+            nn.GELU(),
+        )
+        self.proj = nn.Linear(config.intermediate_size, config.vision_config.proj_output_dim)
+
+    def freeze(self):
+        self.freeze_module(self.p0)
+        self.freeze_module(self.proj)
+
+    def pixel_shuffle(self, x, scale_factor=0.5):
+        n, w, h, c = x.size()
+        # N, W, H, C --> N, W, H * scale, C // scale
+        x = x.reshape(n, w, int(h * scale_factor), int(c / scale_factor))
+        # N, W, H * scale, C // scale --> N, H * scale, W, C // scale
+        x = x.permute(0, 2, 1, 3).contiguous()
+        # N, H * scale, W, C // scale --> N, H * scale, W * scale, C // (scale ** 2)
+        x = x.view(n, int(h * scale_factor), int(w * scale_factor),
+                   int(c / (scale_factor * scale_factor)))
+        return x
+
+    def forward_adapter_modules(self, hidden_states):
+        h = w = int(hidden_states.shape[1] ** 0.5)
+        hidden_states = hidden_states.reshape(hidden_states.shape[0], h, w, -1)
+        hidden_states = self.pixel_shuffle(hidden_states, scale_factor=0.5)
+        hidden_states = hidden_states.reshape(hidden_states.shape[0], -1, hidden_states.shape[-1])
+
+        hidden_states = self.proj(self.p0(hidden_states))
+
+        return hidden_states

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

config.json

@@ -0,0 +1,114 @@
+{
+  "_name_or_path": "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/lishuo/repo/fine_tuning_package/model/",
+  "architectures": [
+    "FlashVLDynamic"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_FlashVLDynamic.FlashVLDynamicConfig",
+    "AutoModel": "modeling_FlashVLDynamic.FlashVLDynamic"
+  },
+  "intermediate_size": 7168,
+  "image_token_num": 256,
+  "image_split": 4,
+  "llm_config":{
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "eos_token_id": 151645,
+    "hidden_act": "silu",
+    "hidden_size": 1536,
+    "initializer_range": 0.02,
+    "intermediate_size": 8960,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 21,
+    "model_type": "qwen2",
+    "num_attention_heads": 12,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 2,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000.0,
+    "sliding_window": null,
+    "tie_word_embeddings": true,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.45.0.dev0",
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 151936
+  },
+  "vision_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_size": 1536,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 448,
+    "intermediate_size": 4096,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "aimv2",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 12,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 24,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 14,
+    "prefix": null,
+    "problem_type": null,
+    "proj_output_dim": 1536,
+    "projection_dropout": 0.0,
+    "pruned_heads": {},
+    "qkv_bias": false,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-05,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_bias": false
+  }
+}

configuration.json

@@ -0,0 +1 @@
+{}

configuration_FlashVLDynamic.py

@@ -0,0 +1,26 @@
+import copy
+import transformers
+from transformers import PretrainedConfig, Qwen2Config
+from .configuration_aimv2 import AIMv2Config
+
+class FlashVLDynamicConfig(PretrainedConfig):
+    model_type = 'FlashVLDynamicConfig'
+    is_composition = True
+
+    def __init__(
+            self,
+            vision_config,
+            llm_config,
+            **kwargs
+        ):
+        super().__init__(**kwargs)
+        self.vision_config = AIMv2Config(**vision_config)
+        self.llm_config = Qwen2Config(**llm_config)
+
+    def to_dict(self):
+        
+        output = copy.deepcopy(self.__dict__)
+        output['vision_config'] = self.vision_config.to_dict()
+        output['llm_config'] = self.llm_config.to_dict()
+
+        return output

configuration_aimv2.py

@@ -0,0 +1,62 @@
+from typing import Any
+
+from transformers.configuration_utils import PretrainedConfig
+
+__all__ = ["AIMv2Config"]
+
+
+class AIMv2Config(PretrainedConfig):
+    """This is the configuration class to store the configuration of an [`AIMv2Model`].
+
+    Instantiating a configuration with the defaults will yield a similar configuration
+    to that of the [apple/aimv2-large-patch14-224](https://huggingface.co/apple/aimv2-large-patch14-224).
+
+    Args:
+        hidden_size: Dimension of the hidden representations.
+        intermediate_size: Dimension of the SwiGLU representations.
+        num_hidden_layers: Number of hidden layers in the Transformer.
+        num_attention_heads: Number of attention heads for each attention layer
+            in the Transformer.
+        num_channels: Number of input channels.
+        image_size: Image size.
+        patch_size: Patch size.
+        rms_norm_eps: Epsilon value used for the RMS normalization layer.
+        attention_dropout: Dropout ratio for attention probabilities.
+        projection_dropout: Dropout ratio for the projection layer after the attention.
+        qkv_bias: Whether to add a bias to the queries, keys and values.
+        use_bias: Whether to add a bias in the feed-forward and projection layers.
+        kwargs: Keyword arguments for the [`PretrainedConfig`].
+    """
+
+    model_type: str = "aimv2"
+
+    def __init__(
+        self,
+        hidden_size: int = 1024,
+        intermediate_size: int = 2816,
+        num_hidden_layers: int = 24,
+        num_attention_heads: int = 8,
+        num_channels: int = 3,
+        image_size: int = 224,
+        patch_size: int = 14,
+        rms_norm_eps: float = 1e-5,
+        attention_dropout: float = 0.0,
+        projection_dropout: float = 0.0,
+        qkv_bias: bool = False,
+        use_bias: bool = False,
+        **kwargs: Any,
+    ):
+        super().__init__(**kwargs)
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_channels = num_channels
+        self.patch_size = patch_size
+        self.image_size = image_size
+        self.attention_dropout = attention_dropout
+        self.rms_norm_eps = rms_norm_eps
+
+        self.projection_dropout = projection_dropout
+        self.qkv_bias = qkv_bias
+        self.use_bias = use_bias

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "max_new_tokens": 2048,
+  "pad_token_id": 151643,
+  "transformers_version": "4.45.0.dev0"
+}

mm_constants.py

@@ -0,0 +1,7 @@
+# Model Constants
+IGNORE_INDEX = -100
+IMAGE_TOKEN_INDEX = -200
+IMAGE_PAD_TOKEN_INDEX = -201
+
+DEFAULT_SLICE_START_TOKEN = "[PLACEHOLDER_0]"
+DEFAULT_SLICE_END_TOKEN = "[PLACEHOLDER_1]"

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f9d37145fe62c36bb43d02b3e1fba703465d4241f3c9545ded3c96291873893
+size 4918323328

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec806b72857f9a0db631fe56bddcbb7b6493d9cd616e25428af426533a4aa240
+size 212893224

model.safetensors.index.json

@@ -0,0 +1,527 @@
+{
+  "metadata": {
+    "total_size": 5131156480
+  },
+  "weight_map": {
+    "adp.p0.0.bias": "model-00001-of-00002.safetensors",
+    "adp.p0.0.weight": "model-00001-of-00002.safetensors",
+    "adp.p0.1.bias": "model-00002-of-00002.safetensors",
+    "adp.p0.1.weight": "model-00002-of-00002.safetensors",
+    "adp.p0.3.bias": "model-00002-of-00002.safetensors",
+    "adp.p0.3.weight": "model-00002-of-00002.safetensors",
+    "adp.proj.bias": "model-00002-of-00002.safetensors",
+    "adp.proj.weight": "model-00002-of-00002.safetensors",
+    "llm.lm_head.weight": "model-00001-of-00002.safetensors",
+    "llm.model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "llm.model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "llm.model.norm.weight": "model-00001-of-00002.safetensors",
+    "vit.preprocessor.patchifier.norm.weight": "model-00001-of-00002.safetensors",
+    "vit.preprocessor.patchifier.proj.bias": "model-00001-of-00002.safetensors",
+    "vit.preprocessor.patchifier.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.preprocessor.pos_embed": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.0.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.1.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.10.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.11.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.12.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.13.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.14.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.15.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.16.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.17.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.18.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.19.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.2.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.20.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.21.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.22.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.23.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.3.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.4.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.5.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.6.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.7.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.8.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.attn.proj.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.attn.qkv.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.mlp.fc1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.mlp.fc2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.mlp.fc3.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.norm_1.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.blocks.9.norm_2.weight": "model-00001-of-00002.safetensors",
+    "vit.trunk.post_trunk_norm.weight": "model-00001-of-00002.safetensors"
+  }
+}

modeling_FlashVLDynamic.py

@@ -0,0 +1,392 @@
+import os
+import math
+import copy
+
+import torch
+import torch.nn.functional as F
+from torch.nn import CrossEntropyLoss
+
+from PIL import Image
+from functools import partial
+from typing import List, Optional, Tuple, Union, Dict
+from dataclasses import dataclass
+
+import transformers
+from transformers.modeling_outputs import ModelOutput
+from transformers.modeling_utils import PreTrainedModel
+from transformers import AutoModelForCausalLM
+
+from .processing_FlashVL import tokenizer_image_token_qwen
+from .adapters import Adapter_AIM
+from .mm_constants import IGNORE_INDEX, IMAGE_TOKEN_INDEX, DEFAULT_SLICE_START_TOKEN, DEFAULT_SLICE_END_TOKEN 
+from .utils_data import split_image_ur
+from .configuration_FlashVLDynamic import FlashVLDynamicConfig
+from .modeling_aimv2 import AIMv2Model
+
+@dataclass
+class FlashVLDynamicOutputWithPast(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+
+
+class FlashVLDynamic(PreTrainedModel):
+    config_class = FlashVLDynamicConfig
+    
+    def __init__(self, config):
+        super().__init__(config)
+        self.llm = AutoModelForCausalLM.from_config(config.llm_config, trust_remote_code=True)
+        self.vit = AIMv2Model(config.vision_config)
+        self.adp = Adapter_AIM(config)
+        
+        self.image_token_num = config.image_token_num
+        self.image_size = config.vision_config.image_size
+        self.image_split = config.image_split
+    
+    def merge_text_image_tokens(self, inputs, add_start_end=False):
+        input_ids, image_features, targets, attn_mask, loss_mask = inputs
+        micro_batch_size, tokens_len = input_ids.shape
+        device = input_ids.device
+        
+        img_rows, img_cols = torch.where(input_ids == IMAGE_TOKEN_INDEX)
+        image_idxs = {i: [] for i in range(micro_batch_size)}
+        for row, col in zip(img_rows.tolist(), img_cols.tolist()):
+            image_idxs[row].append(col)
+        for row in range(micro_batch_size):
+            image_idxs[row] = sorted(image_idxs[row])
+
+        split_sizes = []
+        for row in range(micro_batch_size):
+            image_num = len(image_idxs[row])
+            if image_num == 0:
+                split_sizes.append(tokens_len)
+                continue
+
+            if image_idxs[row][0] != 0:
+                split_sizes.append(image_idxs[row][0])
+
+            for idx in range(image_num - 1):
+                split_sizes.append(self.image_token_num)
+                if image_idxs[row][idx + 1] > image_idxs[row][idx] + self.image_token_num:
+                    split_sizes.append(image_idxs[row][idx + 1] - (image_idxs[row][idx] + self.image_token_num))
+
+            if image_idxs[row][image_num - 1] + self.image_token_num >= tokens_len:
+                split_sizes.append(tokens_len - image_idxs[row][image_num - 1])
+            else:
+                split_sizes.append(self.image_token_num)
+                split_sizes.append(tokens_len - (image_idxs[row][image_num - 1] + self.image_token_num))
+
+        input_ids_noim = torch.where(input_ids < 0, 151643, input_ids)
+        input_ids_noim = input_ids_noim.view(-1)
+        input_embeds = self.llm.model.embed_tokens(input_ids_noim)
+        input_embeds_split = torch.split(input_embeds, split_sizes, dim=0)
+
+        vl_embeds_list = []
+        cur_language_idx = 0
+        cur_image_idx = 0
+        for row in range(micro_batch_size):
+            image_num = len(image_idxs[row])
+            if image_num == 0:
+                vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                cur_language_idx += 1
+                vl_embeds_list.append(image_features[cur_image_idx][0:0])
+                cur_image_idx += 1
+                continue
+
+            if image_idxs[row][0] != 0:
+                vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                cur_language_idx += 1
+
+            for idx in range(image_num - 1):
+                vl_embeds_list.append(image_features[cur_image_idx])
+                cur_language_idx += 1
+                cur_image_idx += 1
+
+                if image_idxs[row][idx + 1] > image_idxs[row][idx] + self.image_token_num:
+                    vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                    cur_language_idx += 1
+
+            if image_idxs[row][image_num - 1] + self.image_token_num >= tokens_len:
+                vl_embeds_list.append(image_features[cur_image_idx][0 : tokens_len - image_idxs[row][image_num - 1]])
+                cur_language_idx += 1
+                cur_image_idx += 1
+            else:
+                vl_embeds_list.append(image_features[cur_image_idx])
+                cur_language_idx += 1
+                cur_image_idx += 1
+                vl_embeds_list.append(input_embeds_split[cur_language_idx])
+                cur_language_idx += 1
+
+        vl_embeds = torch.cat(vl_embeds_list)
+        vl_embeds = vl_embeds.view(micro_batch_size, tokens_len, vl_embeds.shape[-1])
+        return (input_ids, vl_embeds, targets, attn_mask, loss_mask)
+    
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        pixel_values: torch.FloatTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        local_pos_batch: Optional[torch.LongTensor] = None,
+        image_idx_batch: Optional[torch.Tensor] = None,
+        loss_mask_batch: Optional[torch.Tensor] = None,
+        use_cache: Optional[bool] = None,
+    ):
+        inputs = [input_ids, pixel_values, labels, attention_mask, loss_mask_batch]
+        
+        if isinstance(inputs[1], list):
+            pixel_values = [p.bfloat16() for p in inputs[1]]
+        else:
+            pixel_values = inputs[1].bfloat16()
+        img_token = self.vit.forward(pixel_values)
+        
+        if hasattr(img_token, 'last_hidden_state'):
+            img_token = img_token.last_hidden_state
+        
+        inputs = self.adp(inputs[:1]+[img_token]+inputs[2:])
+
+        inputs = self.merge_text_image_tokens(inputs)
+        tokens, hidden_states, targets, attn_mask, loss_mask = inputs
+
+        outputs = self.llm.forward(
+            inputs_embeds=hidden_states,
+            attention_mask=attn_mask,
+            use_cache=use_cache)
+        
+        lm_logits = outputs.logits
+        
+        loss = None
+        if targets is not None:
+            labels = targets.to(lm_logits.device)
+            shift_logits = lm_logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+
+            loss_fct = CrossEntropyLoss(reduction='none')
+            loss = loss_fct(
+                shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1)
+            )
+
+            batch_size = labels.size(0)
+            loss_mask = loss_mask[:, 1:].to(loss.dtype)
+            loss = (loss.view(batch_size, -1) * loss_mask).sum() / loss_mask.sum()
+
+        return FlashVLDynamicOutputWithPast(
+            loss=loss,
+            logits=lm_logits,
+        )
+    
+    def get_input_embeddings(self):
+        return self.llm.get_input_embeddings()
+    
+    def split_image_minicpm(self, image):
+
+        splits, grid_shapes = split_image_ur(image, self.image_split, self.image_size, force_min_size=True)
+
+        prefix = ''
+        flatten_splits = [splits[0]] # global image
+        prefix += '<image>\n'
+        if len(splits) > 1:
+            prefix += DEFAULT_SLICE_START_TOKEN # slice starts
+            for i in range(1, len(splits)):
+                prefix += '<image>'
+                prefix += '\n'
+                flatten_splits += [splits[i]]
+            prefix += DEFAULT_SLICE_END_TOKEN # slice ends
+        
+        return flatten_splits, prefix
+    
+    def to_llava_format(self, data):
+        img_pil = data['img']
+        messages = data['messages']
+        text_only = data['text_only']
+        is_video=False
+        if 'is_video' in data:
+            is_video=data['is_video']
+        messages.append({'role': 'assistant', 'content': ''})
+        conversations = []
+        for i,m in enumerate(messages):
+            if m['role'] == 'user':
+                value = str(m['content']).replace('<image>', '')
+                
+                if i == 0 and not text_only:
+                    assert not isinstance(img_pil, list)
+                    img_pil, prefix = self.split_image_minicpm(img_pil)
+                    value = prefix + value
+                
+                conversations.append({'from': 'human', 'value': value})
+            elif m['role'] == 'assistant':
+                conversations.append({'from': 'gpt', 'value': str(m['content']).replace('<image>', '')})
+            else:
+                raise ValueError(f"Wrong role in conversation. {m['role']}")
+        return {'image': img_pil,
+                'text_only': text_only,
+                'is_video':is_video,
+                'conversations': conversations}
+    
+    def generate(
+        self,
+        input_ids=None,
+        pixel_values=None,
+        attention_mask=None,
+        streamer=None, 
+        **kwargs
+        ):   
+        image = kwargs.get('image')
+        img_token = self.vit.forward(image.bfloat16())
+        if hasattr(img_token, 'last_hidden_state'):
+            img_token = img_token.last_hidden_state
+        inputs = self.adp((
+                    input_ids.to(self.device),
+                    img_token,
+                    None, None, None))
+        inputs = self.merge_text_image_tokens(inputs)
+        tokens, hidden_states, targets, attn_mask, loss_mask = inputs
+
+        keys_to_pop = ['loss_mask', 'paddings','targets','attn_mask','image']
+        kwargs = {k: v for k, v in kwargs.items() if k not in keys_to_pop}
+        outputs = self.llm.generate(
+            inputs_embeds=hidden_states.bfloat16(),
+            max_new_tokens=2048,
+            do_sample=False,
+            **kwargs
+        )
+        
+        return outputs
+    
+    def chat(self, pil_image, messages, answer_prompt=None, do_sample=True, max_new_tokens=256):
+            
+        data={}
+        data['img'] = pil_image
+        data['text_only'] = (pil_image is None)
+        data['messages'] = messages
+
+        sources = self.to_llava_format(data)
+        sources = [sources]
+        has_image = not sources[0]['text_only']
+
+        if has_image:
+            img_list = sources[0]['image']
+            if not isinstance(img_list, list):
+                img_list = [img_list]
+            image = torch.stack([torch.from_numpy(self.im_trans(i)['pixel_values'][0]) for i in img_list], dim=0)
+        
+        sources = copy.deepcopy([e["conversations"] for e in sources])
+        
+        data_dict = self.preprocess_qwen(
+                sources,
+                self.tokenizer,
+                has_image=has_image,
+                )
+        
+        input_ids_data = data_dict["input_ids"][0]
+        data_dict["input_ids"] = [ input_ids_data, ]
+        
+        if not has_image:
+            image = torch.zeros(1, 3, self.image_size, self.image_size)
+        data_dict = dict(tokens=data_dict["input_ids"][0],)
+
+        img_token = self.vit.forward(image.cuda().bfloat16())
+        
+        if hasattr(img_token, 'last_hidden_state'):
+            img_token = img_token.last_hidden_state
+
+        inputs = self.adp((
+                    data_dict['tokens'].unsqueeze(0).to(self.device),
+                    img_token, 
+                    None, None, None))
+        
+        inputs = self.merge_text_image_tokens(inputs)
+        tokens, hidden_states, targets, attn_mask, loss_mask = inputs
+
+        outputs = self.llm.generate(
+            inputs_embeds=hidden_states.bfloat16(),
+            return_dict_in_generate=False,
+            max_new_tokens=max_new_tokens,
+            do_sample=do_sample,
+            pad_token_id=False,
+        )
+        decoded = self.tokenizer.decode(outputs[0])
+
+        stop_words_ids = [self.llm.generation_config.bos_token_id,
+                          self.llm.generation_config.eos_token_id,
+                          self.tokenizer.convert_tokens_to_ids('<|im_start|>')]
+        stop_words = [self.tokenizer.decode(w) for w in stop_words_ids]
+
+        for stop_word in stop_words:
+            decoded = decoded.replace(stop_word, "").strip()
+
+        return decoded
+    
+    def preprocess_qwen(
+        self,
+        sources, 
+        tokenizer: transformers.PreTrainedTokenizer, 
+        has_image: bool = False, 
+        max_len=2048, 
+        system_message: str = "You are a helpful assistant.",) -> Dict:
+
+        roles = {"human": "user", "gpt": "assistant"}
+        tokenizer = copy.deepcopy(tokenizer)
+        
+        tokenizer.add_tokens(["<image>"], special_tokens=True)
+        image_token_index = tokenizer.convert_tokens_to_ids("<image>")
+        im_start, im_end = tokenizer.additional_special_tokens_ids[:2]
+        # unmask_tokens = ["<|im_start|>", "<|im_start|>", "\n"]
+        unmask_tokens_idx =  [198, im_start, im_end]
+        nl_tokens = tokenizer("\n").input_ids
+        
+        chat_template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
+        tokenizer.chat_template = chat_template
+        
+        input_ids, targets = [], []
+        for i, source in enumerate(sources):
+            if roles[source[0]["from"]] != roles["human"]:
+                source = source[1:]
+            input_id, target = [], []
+            
+            input_id += tokenizer.apply_chat_template([{"role" : "system", "content" : system_message}])
+            target += [IGNORE_INDEX] * len(input_id)
+            i=0
+            for conv in source:
+                try:
+                    role = conv["role"]
+                    content = conv["content"]
+                except:
+                    role = conv["from"]
+                    content = conv["value"]
+                role =  roles.get(role, role)
+                
+                if i==len(source)-1:
+                    conv = [{"role" : role, "content" : content}]
+                    encode_id = tokenizer.apply_chat_template(conv,add_generation_prompt=True)
+                else:     
+                    conv = [{"role" : role, "content" : content}]
+                    encode_id = tokenizer.apply_chat_template(conv)
+                i=i+1
+                if image_token_index in encode_id:
+                    encode_id = tokenizer_image_token_qwen(encode_id, tokenizer, image_token_index,image_token_num=self.image_token_num)
+                    
+                input_id += encode_id 
+                if role in ["user", "system"]:
+                    target += [IGNORE_INDEX] * len(encode_id)
+                else:
+                    target += encode_id
+
+
+            assert len(input_id) == len(target), f"{len(input_id)} != {len(target)}"
+            for idx, encode_id in enumerate(input_id):
+                if encode_id in unmask_tokens_idx:
+                    target[idx] = encode_id
+                if encode_id == image_token_index:
+                    input_id[idx] = IMAGE_TOKEN_INDEX
+            input_ids.append(input_id)
+            targets.append(target)
+        input_ids = torch.tensor(input_ids, dtype=torch.long)
+        targets = torch.tensor(targets, dtype=torch.long)
+        return dict(
+            input_ids=input_ids,
+            labels=targets,
+        )

modeling_aimv2.py

@@ -0,0 +1,192 @@
+from typing import Optional, Tuple, Union
+
+import torch
+from .configuration_aimv2 import AIMv2Config
+from torch import nn
+from torch.nn import functional as F
+from transformers.modeling_outputs import BaseModelOutputWithNoAttention
+from transformers.modeling_utils import PreTrainedModel
+
+__all__ = ["AIMv2Model"]
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(dim))
+        self.eps = eps
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        output = self._norm(x.float()).type_as(x)
+        return output * self.weight
+
+    def extra_repr(self) -> str:
+        return f"{tuple(self.weight.shape)}, eps={self.eps}"
+
+    def _norm(self, x: torch.Tensor) -> torch.Tensor:
+        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+
+
+class AIMv2SwiGLUFFN(nn.Module):
+    def __init__(self, config: AIMv2Config):
+        super().__init__()
+        hidden_features = config.intermediate_size
+        in_features = config.hidden_size
+        bias = config.use_bias
+
+        self.fc1 = nn.Linear(in_features, hidden_features, bias=bias)
+        self.fc2 = nn.Linear(hidden_features, in_features, bias=bias)
+        self.fc3 = nn.Linear(in_features, hidden_features, bias=bias)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = F.silu(self.fc1(x)) * self.fc3(x)
+        x = self.fc2(x)
+        return x
+
+
+class AIMv2PatchEmbed(nn.Module):
+    def __init__(self, config: AIMv2Config):
+        super().__init__()
+        self.proj = nn.Conv2d(
+            config.num_channels,
+            config.hidden_size,
+            kernel_size=(config.patch_size, config.patch_size),
+            stride=(config.patch_size, config.patch_size),
+        )
+        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = self.proj(x).flatten(2).transpose(1, 2)
+        x = self.norm(x)
+        return x
+
+
+class AIMv2ViTPreprocessor(nn.Module):
+    def __init__(self, config: AIMv2Config):
+        super().__init__()
+        num_patches = (config.image_size // config.patch_size) ** 2
+
+        self.patchifier = AIMv2PatchEmbed(config)
+        self.pos_embed = nn.Parameter(torch.zeros((1, num_patches, config.hidden_size)))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        tokens = self.patchifier(x)
+        _, N, _ = tokens.shape
+        pos_embed = self.pos_embed.to(tokens.device)
+        tokens = tokens + pos_embed[:, :N]
+        return tokens
+
+
+class AIMv2Attention(nn.Module):
+    def __init__(self, config: AIMv2Config):
+        super().__init__()
+        dim = config.hidden_size
+
+        self.num_heads = config.num_attention_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=config.qkv_bias)
+        self.attn_drop = nn.Dropout(config.attention_dropout)
+        self.proj = nn.Linear(dim, dim, bias=config.use_bias)
+        self.proj_drop = nn.Dropout(config.projection_dropout)
+
+    def forward(
+        self, x: torch.Tensor, mask: Optional[torch.Tensor] = None
+    ) -> torch.Tensor:
+        B, N, C = x.shape
+        qkv = (
+            self.qkv(x)
+            .reshape(B, N, 3, self.num_heads, C // self.num_heads)
+            .permute(2, 0, 3, 1, 4)
+        )
+        q, k, v = qkv.unbind(0)
+
+        x = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)
+        x = x.transpose(1, 2).contiguous().reshape(B, N, C)
+        x = self.proj(x)
+        x = self.proj_drop(x)
+        return x
+
+
+class AIMv2Block(nn.Module):
+    def __init__(self, config: AIMv2Config):
+        super().__init__()
+        self.attn = AIMv2Attention(config)
+        self.norm_1 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.mlp = AIMv2SwiGLUFFN(config)
+        self.norm_2 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self, x: torch.Tensor, mask: Optional[torch.Tensor] = None
+    ) -> torch.Tensor:
+        x = x + self.attn(self.norm_1(x), mask)
+        x = x + self.mlp(self.norm_2(x))
+        return x
+
+
+class AIMv2Transformer(nn.Module):
+    def __init__(self, config: AIMv2Config):
+        super().__init__()
+        self.blocks = nn.ModuleList(
+            [AIMv2Block(config) for _ in range(config.num_hidden_layers)]
+        )
+        self.post_trunk_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        tokens: torch.Tensor,
+        mask: Optional[torch.Tensor] = None,
+        output_hidden_states: bool = False,
+    ) -> Tuple[torch.Tensor, Optional[Tuple[torch.Tensor, ...]]]:
+        hidden_states = () if output_hidden_states else None
+        for block in self.blocks:
+            tokens = block(tokens, mask)
+            if output_hidden_states:
+                hidden_states += (tokens,)
+        tokens = self.post_trunk_norm(tokens)
+        return tokens, hidden_states
+
+
+class AIMv2PretrainedModel(PreTrainedModel):
+    config_class = AIMv2Config
+    base_model_prefix = "aimv2"
+    main_input_name = "pixel_values"
+    _no_split_modules = ["AIMv2ViTPreprocessor", "AIMv2Block"]
+    _supports_sdpa = True
+
+
+class AIMv2Model(AIMv2PretrainedModel):
+    def __init__(self, config: AIMv2Config):
+        super().__init__(config)
+        self.preprocessor = AIMv2ViTPreprocessor(config)
+        self.trunk = AIMv2Transformer(config)
+
+    def forward(
+        self,
+        pixel_values: torch.Tensor,
+        mask: Optional[torch.Tensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[
+        Tuple[torch.Tensor],
+        Tuple[torch.Tensor, Tuple[torch.Tensor, ...]],
+        BaseModelOutputWithNoAttention,
+    ]:
+        if output_hidden_states is None:
+            output_hidden_states = self.config.output_hidden_states
+        if return_dict is None:
+            return_dict = self.config.use_return_dict
+
+        x = self.preprocessor(pixel_values)
+        x, hidden_states = self.trunk(
+            x, mask, output_hidden_states=output_hidden_states
+        )
+
+        if not return_dict:
+            res = (x,)
+            res += (hidden_states,) if output_hidden_states else ()
+            return res
+
+        return BaseModelOutputWithNoAttention(
+            last_hidden_state=x,
+            hidden_states=hidden_states,
+        )
+

preprocessor_config.json

@@ -0,0 +1,27 @@
+{
+  "crop_size": {
+    "height": 448,
+    "width": 448
+  },
+  "do_center_crop": true,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "CLIPImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 448
+  }
+}

processing_FlashVL.py

@@ -0,0 +1,19 @@
+from .mm_constants import IMAGE_TOKEN_INDEX, IMAGE_PAD_TOKEN_INDEX
+
+def tokenizer_image_token_qwen(prompt, tokenizer, image_token_index, image_token_num=256):
+    prompt_chunks, tmp = [], []
+    for n in prompt:
+        if n == image_token_index:
+            prompt_chunks.append(tmp)
+            tmp = []
+        else:
+            tmp.append(n)
+    if tmp: prompt_chunks.append(tmp)
+
+    input_ids = []
+    for i, chunk in enumerate(prompt_chunks):
+        if i > 0:
+            input_ids.extend([IMAGE_TOKEN_INDEX] + [IMAGE_PAD_TOKEN_INDEX] * (image_token_num - 1))
+        input_ids.extend(chunk)
+
+    return input_ids

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

utils_data.py

@@ -0,0 +1,185 @@
+import torch
+import os
+from PIL import Image
+import numpy as np
+import math
+import torchvision.transforms.functional as F
+from torchvision.transforms import InterpolationMode
+
+
+def split_image_ur(img, max_slice_num, image_size, force_min_size=False):
+    if force_min_size:
+        img = resize_by_patch_size_ur(img, min_size= image_size, max_size= image_size * max_slice_num, patch_size=14)
+    slice_config = {
+        "max_slice_nums": max_slice_num,
+        "scale_resolution": image_size,
+        "patch_size": 14
+    }
+    source_image, sub_images, _ = do_slice_by_minicpmv_strategy_ur(
+                    img, max_slice_nums=slice_config["max_slice_nums"], scale_resolution=slice_config["scale_resolution"], patch_size=slice_config["patch_size"])
+    splits = []
+    splits.append(source_image)
+    for i in range(len(sub_images)):
+        for j in range(len(sub_images[0])):
+            splits.append(sub_images[i][j])
+    sliced_images, sliced_shapes = [], []
+    for slice_image in splits:
+        sliced_images.append(slice_image)
+        sliced_shapes.append(np.array((slice_image.size[0] // slice_config["patch_size"], slice_image.size[1] // slice_config["patch_size"])))
+    
+    return sliced_images, sliced_shapes
+
+# Strategy: MiniCPM-V
+def do_slice_by_minicpmv_strategy_ur(image, max_slice_nums=9, scale_resolution=1120, patch_size=14, never_split=False):
+
+    original_size = image.size
+    original_width, original_height = original_size
+    log_ratio = math.log(original_width / original_height)
+    ratio = original_width * original_height / (scale_resolution * scale_resolution)
+    multiple = min(math.ceil(ratio), max_slice_nums)
+
+    source_image = None
+    best_grid = None
+    patches = []
+
+    if multiple <= 1 or never_split:
+        # dont need to slice, upsample
+        # best_size = find_best_resize(
+        #     original_size, scale_resolution, patch_size, allow_upscale=True
+        # )
+        best_size = (scale_resolution, scale_resolution)
+        source_image = image.resize(best_size, Image.BICUBIC)
+    else:
+        candidate_split_grids_nums = []
+        for i in [multiple - 1, multiple, multiple + 1]:
+            if i == 1 or i > max_slice_nums:
+                continue
+            candidate_split_grids_nums.append(i)
+
+        # source image, down-sampling and ensure divided by patch_size
+        # best_resize = find_best_resize(original_size, scale_resolution, patch_size)
+        # source_image = image.copy().resize(best_resize, Image.BICUBIC)
+        source_image = image.copy().resize((scale_resolution,scale_resolution), Image.BICUBIC)
+        candidate_grids = []
+
+        # find best grid
+        for split_grids_nums in candidate_split_grids_nums:
+            m = 1
+            while m <= split_grids_nums:
+                if split_grids_nums % m == 0:
+                    candidate_grids.append([m, split_grids_nums // m])
+                m += 1
+        # print("candidate_grids: ", candidate_grids)
+
+        best_grid = [1, 1]
+        min_error = float("inf")
+        for grid in candidate_grids:
+            error = abs(log_ratio - math.log(grid[0] / grid[1]))
+            if error < min_error:
+                best_grid = grid
+                min_error = error
+
+        refine_size = get_refine_size(
+            original_size, best_grid, scale_resolution, patch_size, allow_upscale=True
+        )
+
+        refine_image = image.resize(refine_size, Image.BICUBIC)
+        patches = split_to_patches(refine_image, best_grid)
+
+    return source_image, patches, best_grid
+
+
+def ensure_divide(length, patch_size):
+    return max(round(length / patch_size) * patch_size, patch_size)
+
+
+def find_best_resize(original_size, scale_resolution, patch_size, allow_upscale=False):
+    width, height = original_size
+    if (width * height > scale_resolution * scale_resolution) or allow_upscale:
+        r = width / height
+        height = int(scale_resolution / math.sqrt(r))
+        width = int(height * r)
+    best_width = ensure_divide(width, patch_size)
+    best_height = ensure_divide(height, patch_size)
+
+    # print(best_width, best_height, scale_resolution)
+    while best_width * best_height  > scale_resolution ** 2:
+        # print(best_width)
+        best_width -= patch_size
+    
+    return (best_width, best_height)
+
+
+def get_refine_size(original_size, grid, scale_resolution, patch_size, allow_upscale=False):
+    width, height = original_size
+    grid_x, grid_y = grid
+
+    # refine_width = ensure_divide(width, grid_x)
+    # refine_height = ensure_divide(height, grid_y)
+
+    # grid_width = refine_width / grid_x
+    # grid_height = refine_height / grid_y
+
+    # best_grid_size = find_best_resize(
+    #     (grid_width, grid_height),
+    #     scale_resolution,
+    #     patch_size,
+    #     allow_upscale=allow_upscale,
+    # )
+
+    refine_size = (scale_resolution * grid_x, scale_resolution * grid_y)
+
+    return refine_size
+
+
+def split_to_patches(image, grid):
+    patches = []
+    width, height = image.size
+    grid_x = int(width / grid[0])
+    grid_y = int(height / grid[1])
+
+    for i in range(0, height, grid_y):
+        images = []
+        for j in range(0, width, grid_x):
+            box = (j, i, j + grid_x, i + grid_y)
+            patch = image.crop(box)
+            images.append(patch)
+        patches.append(images)
+
+    return patches
+
+def resize_by_patch_size_ur(img, min_size=1152, max_size=2240, patch_size=14):
+    interpolation=InterpolationMode.BICUBIC
+    # min_size=756, max_size=756 * 4, patch_size=14
+    if isinstance(img, torch.Tensor):
+        height, width = img.shape[:2]
+    else:
+        width, height = img.size
+        
+    # Check if the shorter side is less than min_size
+    if min(height, width) < min_size:
+        # print('less than min_size')
+        scale_factor = min_size / min(height, width)
+        new_height = max(min_size, round(height * scale_factor))
+        new_width = max(min_size, round(width * scale_factor))
+        # print(self.max_size)
+
+        # Check if the longer side after resizing is greater than max_size
+        if max(new_height, new_width) > max_size:
+            scale_factor = max_size / max(new_height, new_width)
+            new_height = min(max_size, round(new_height * scale_factor))
+            new_width = min(max_size, round(new_width * scale_factor))
+    else:
+        scale_factor = min(max_size / max(height, width), 1)
+        new_height = round(height * scale_factor)
+        new_width = round(width * scale_factor)
+
+    # # Make sure the new height and width are divisible by patch_size
+    # new_height = (new_height // patch_size) * patch_size
+    # new_width = (new_width // patch_size) * patch_size
+
+    # Resize the image
+    # img = F.resize(img, (new_height, new_width), interpolation)
+    img = img.resize((new_width, new_height), Image.BICUBIC)
+
+    return img