README.md

# ImageRAG


<font size=4><div align='center'> [[Github](https://github.com/om-ai-lab/ImageRAG)] [[📄 Paper](https://ieeexplore.ieee.org/document/11039502)] [[📝 Blog (in Mandarin)](https://mp.weixin.qq.com/s/BcFejPAcxh4Rx_rh1JvRJA)]</div></font> 
## ✨ Highlight

Ultrahigh resolution (UHR) remote sensing imagery (RSI) (e.g. 10,000 X 10,000 pixels) poses a significant challenge for current RS vision-language models (RSVLMs). If one chooses to resize the UHR image to the standard in
put image size, the extensive spatial and contextual information that UHR images contain will be neglected. Otherwise, the original size of these images often exceeds the token limits of standard RSVLMs, making it difficult to pro
cess the entire image and capture long-range dependencies to answer the query based on the abundant visual context. 

* Three crucial aspects for MLLMs to effectively handle UHR RSI are:
  
    * Managing small targets, ensuring that the model can accurately be aware and analyze fine details 
  within images
  
    * Processing the UHR image in a way that integrates with MLLMs without significantly increasing 
  the number of image tokens, which would lead to high computational costs
  
    * Achieving these goals while minimizing the need for additional training or specialized 
  annotation.


*  We contribute the ImageRAG framework, which offers several key advantages as follows:

    * It retrieves and emphasizes relevant visual context from the UHR image based on the text query, allowing the MLLM to focus on important details, even tiny ones.
      
    * It integrates various external knowledge sources (store in vector database) to guide the model, enhancing the understanding of the query and UHR RSI
      
    * ImageRAG requires only a small amount of training, making it a practical solution for efficiently handling UHR RSI.



## 🚀 Update
🔥🔥🔥 Last Updated on 2025.07.10 🔥🔥🔥

* **TODO**: Validate the codebase using uploaded data in isolate enviromemt

* **2025.07.10**: Upload checkpoint, cache and dataset.

* **2025.06.25**: Upload codebase and scripts.
  
* **2025.05.24**: ImageRAG is accepted by IEEE Geoscience and Remote Sensing Magazine 🎉
  * IEEE Early Access (we prefer this version): https://ieeexplore.ieee.org/document/11039502
    
  * Arxiv: https://arxiv.org/abs/2411.07688


## 📖 Setup Codebase and Data

* Clone this repo:
   ```bash
   git clone https://github.com/om-ai-lab/ImageRAG.git
   ```
   
* Download data, caches and checkpoints for ImageRAG from huggingface: 
   * https://huggingface.co/omlab/ImageRAG
      
   * Using the [hf mirror](https://hf-mirror.com/) if you encounter connection problems:
       ```bash
       ./hfd.sh omlab/ImageRAG --local-dir ImageRAG_hf
       ```
   * Merge two repos:
      ```bash
      mv ImageRAG_hf/cache ImageRAG_hf/checkpoint ImageRAG_hf/data  ImageRAG/
      ```
   * Unzip all zip files:
      * cache/patch/mmerealworld.zip
      * cache/vector_database/crsd_vector_database.zip
      * cache/vector_database/lrsd_vector_database.zip

   * The ImageRAG directory structure should look like this:
        ```bash
            /training/zilun/ImageRAG

            ├── codebase                        
                ├── inference
                ├── patchify
                ├── main_inference_mmerealworld_imagerag_preextract.py
                ......                                                     
            ├── config                        
                ├── config_mmerealworld-baseline-zoom4kvqa10k2epoch_server.yaml 
                ├── config_mmerealworld-detectiongt-zoom4kvqa10k2epoch_server.yaml 
                ......                                                      
            ├── data                        
                ├── dataset
                    ├── MME-RealWorld
                        ├── remote_sensing
                            ├── remote_sensing
                                ├── 03553_Toronto.png 
                                ......
                ├── crsd_clip_3M.pkl
                ......
            ├── cache                        
                ├── patch
                    ├── mmerealworld
                        ├── vit
                        ├── cc
                        ├── grid
                ├── vector_database 
                    ├── crsd_vector_database
                    ├── lrsd_vector_database
            ├── checkpoint                        
                ├── InternVL2_5-8B_lora32_vqa10k_zoom4k_2epoch_merged
                ......     
            ├── script                        
                ├── clip_cc.sh
                ......
        ```

## 📖 Setup Env

```bash
conda create -n imagerag python=3.10
conda activate imagerag
cd /training/zilun/ImageRAG
export PYTHONPATH=$PYTHONPATH:/training/zilun/ImageRAG
# Install torch, torchvision and flash attention accroding to your cuda version
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
pip install ninja
MAX_JOBS=16 pip install flash-attn --no-build-isolation
```

```bash
pip install requirement.txt
```

```bash
python
>>> import nltk
>>> nltk.download('stopwords')

python -m spacy download en_core_web_sm
```

## 📖 Setup SGLang (Docker)
* Host Qwen2.5-32B-Instruct using SGLang for text parsing module

```bash
# Pull sglang docker (we use mirror just for speeding up the download process)
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/lmsysorg/sglang:latest
docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/lmsysorg/sglang:latest  docker.io/lmsysorg/sglang:latest

# docker load -i sglang.tar

bash script/sglang_start.sh
```


## 📖 Feature Extraction (Optional, use Ray to parallelize the process)
* Necessary if you need to run ImageRAG in cutomized data
  
```bash
ray start --head --port=6379
# extract patch features (e.g. MMERealworld-RS)
python codebase/ray_feat_extract_patch.py --ray_mode auto --num_runner 8
# extract image & text features for vector database (Section V-D, external data)
python codebase/ray_feat_extract_vectorstore.py --ray_mode auto --num_runner 8
# ray stop (optional)
```

## 📖 Inference

* See imagerag_result directory for result examples. 

### Run Baseline Inference (No ImageRAG, No GT, No Inference while detecting)
```bash
# inference
CUDA_VISIBLE_DEVICES=0 python codebase/main_inference_mmerealworld_imagerag_preextract.py --cfg_path config/config_mmerealworld-baseline-zoom4kvqa10k2epoch_server.yaml

# eval inference result
python codebase/inference/MME-RealWorld-RS/eval_your_results.py --results_file data/eval/mmerealworld_zoom4kvqa10k2epoch_baseline.jsonl
```

### Run Regular VQA Task Inference in Parallel

```bash
# {clip, georsclip, remoteclip, mcipclip} x {vit, cc, grid} x {rerank, mean, cluster} x {0, ... ,7}
bash script/georsclip_grid.sh rerank 0
```

### Run Inferring VQA Task Inference (No ImageRAG, No Inference while detecting, BBoxes are needed)
```bash
# inference
CUDA_VISIBLE_DEVICES=0 python codebase/main_inference_mmerealworld_imagerag_preextract.py --cfg_path config/config_mmerealworld-detectiongt-zoom4kvqa10k2epoch_server.yaml

# eval inference result
python codebase/inference/MME-RealWorld-RS/eval_your_results.py --results_file data/eval/mmerealworld_zoom4kvqa10k2epoch_baseline.jsonl
```

## 👨‍🏫 Contact
[email protected]

## 🖊️ Citation
```bash
@ARTICLE{11039502,
  author={Zhang, Zilun and Shen, Haozhan and Zhao, Tiancheng and Guan, Zian and Chen, Bin and Wang, Yuhao and Jia, Xu and Cai, Yuxiang and Shang, Yongheng and Yin, Jianwei},
  journal={IEEE Geoscience and Remote Sensing Magazine}, 
  title={Enhancing Ultrahigh Resolution Remote Sensing Imagery Analysis With ImageRAG: A new framework}, 
  year={2025},
  volume={},
  number={},
  pages={2-27},
  keywords={Image resolution;Visualization;Benchmark testing;Training;Image color analysis;Standards;Remote sensing;Analytical models;Accuracy;Vehicle dynamics},
  doi={10.1109/MGRS.2025.3574742}}

```