Update README.md

README.md

@@ -7,50 +7,95 @@ sdk: static
 pinned: false
 ---
 
-# MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents
 
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/66337cb5bd8ef15a47e72ce0/hhAX190AZageb5Bqqr_lp.png)
+# MMDocRAG Overview
+
+MMDocRAG is built for (i) multimodal document retrieval and (ii) retrieval-augmented multimodal generation:
+
+- **MMDocIR** (📖<a href="https://arxiv.org/abs/2501.08828">Paper</a>  🏠<a href="https://mmdocrag.github.io/MMDocIR/">Homepage</a>  👉<a href="https://github.com/mmdocrag/MMDocIR">Github</a>): Benchmarking Multi-Modal Retrieval for Long Documents:
+  - encompass two distinct tasks: **page-level** and **layout-level** retrieval. 
+  - [MMDocIR_Evaluation_Dataset](https://huggingface.co/datasets/MMDocIR/MMDocIR_Evaluation_Dataset): 1,685 expert-annotated questions for evaluation.
+  - [MMDocIR_Train_Dataset](https://huggingface.co/datasets/MMDocIR/MMDocIR_Train_Dataset): 173,843 bootstrapped questions for training.
+  - [Retriever Checkpoints](https://huggingface.co/MMDocIR/MMDocIR_Retrievers): 6 text and 4 vision retrievers.
+- **MMDocRAG** (📖<a href="https://arxiv.org/abs/2505.16470">Paper</a>  🏠<a href="https://mmdocrag.github.io/MMDocRAG/">Homepage</a>  👉<a href="https://github.com/mmdocrag/MMDocRAG">Github</a>): Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering
+  - design for **multimodal quotes selection** and **multimodal integration**
+  - [MMDocRAG](https://huggingface.co/datasets/MMDocIR/MMDocRAG): 4,055 expert-annotated questions and multimodal answer.
+  - [MMDocRAG Training](https://huggingface.co/datasets/MMDocIR/MMDocRAG/blob/main/train.jsonl): 4,110 training samples derived from dev set.
+  - [Retriever Checkpoints](https://huggingface.co/MMDocIR/MMDocIR_Retrievers): 6 text and 4 vision retrievers.
+
+
+
+
+
+<p align="center">
+  <h2 align="center">MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents</h2>
+    <p align="center">
+    <strong>Kuicai Dong*</strong></a>
+    ·
+    <strong>Yujing Chang*</strong>
+    ·
+    <strong>Derrick Xin Deik Goh*</strong>
+    ·
+    <strong>Dexun Li</strong>
+    ·
+    <a href="https://scholar.google.com/citations?user=fUtHww0AAAAJ&hl=en"><strong>Ruiming Tang</strong></a>
+    ·
+    <a href="https://stephenliu0423.github.io/"><strong>Yong Liu</strong></a>
+  </p>
+<p align="center">
+    📖<a href="https://arxiv.org/abs/2501.08828">Paper</a> |
+    🏠<a href="https://mmdocrag.github.io/MMDocIR/">Homepage</a> |
+    🤗<a href="https://huggingface.co/MMDocIR">Huggingface</a> |
+	👉<a href="https://github.com/mmdocrag/MMDocIR">Github</a>
+</p>
+<p align="left">
+  <p>
+Multimodal document retrieval aims to identify and retrieve various forms of multimodal content, such as figures, tables, charts, and layout information from extensive documents. Despite its increasing popularity, there is a notable lack of a comprehensive and robust benchmark to effectively evaluate the performance of systems in such tasks. To address this gap, this work introduces a new benchmark, named MMDocIR, that encompasses two distinct tasks: page-level and layout-level retrieval. The former evaluates the performance of identifying the most relevant pages within a long document, while the later assesses the ability of detecting specific layouts, providing a more fine-grained measure than whole-page analysis. A layout
+refers to a variety of elements, including textual paragraphs, equations, figures, tables, or charts. The MMDocIR benchmark comprises a rich dataset featuring 1,685 questions annotated by experts and 173,843 questions with bootstrapped labels, making it a valuable resource in multimodal document retrieval for
+both training and evaluation. Through rigorous experiments, we demonstrate that (i) visual retrievers significantly outperform their text counterparts, (ii) MMDocIR training set effectively enhances the performance of multimodal document retrieval and (iii) text retrievers leveraging VLM-text significantly outperforms retrievers relying on OCR-text.
+  </p>
+  <a href="">
+  <div style="text-align: center;">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/66337cb5bd8ef15a47e72ce0/hhAX190AZageb5Bqqr_lp.png" alt="Logo" width="80%">
+   </div>
+  </a>
+<br>
+
+
+
+
+<p align="center">
+  <h2 align="center">MMDocRAG: Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering</h2>
+    <p align="center">
+    <strong>Kuicai Dong</strong></a>
+    ·
+    <strong>Yujing Chang</strong>
+    ·
+    <strong>Shijie Huang</strong>
+    ·
+    <a href="https://scholar.google.com/citations?user=x-UYeJ4AAAAJ&hl=en"><strong>Yasheng Wang</strong></a>
+    ·
+    <a href="https://scholar.google.com/citations?user=fUtHww0AAAAJ&hl=en"><strong>Ruiming Tang</strong></a>
+    ·
+    <a href="https://stephenliu0423.github.io/"><strong>Yong Liu</strong></a>
+<p align="center">
+    📖<a href="https://arxiv.org/abs/2505.16470">Paper</a> |
+    🏠<a href="https://mmdocrag.github.io/MMDocRAG/">Homepage</a> |
+    🤗<a href="https://huggingface.co/datasets/MMDocIR/MMDocRAG">Huggingface</a> |
+	👉<a href="https://github.com/mmdocrag/MMDocRAG">Github</a>
+</p>
+<p align="left">
+  <p>
+Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents (text, images, tables) and performing cross-modal reasoning. Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches, frequently missing critical visual information. The field also lacks robust benchmarks for assessing multimodal evidence integration and selection. We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains. Our framework introduces innovative metrics for evaluating multimodal quote selection and enables answers that combine text with relevant visual elements. Through large-scale experiments with 60 language/vision models and 14 retrieval systems, we identify persistent challenges in multimodal evidence handling. Key findings reveal proprietary vision-language models show
+moderate advantages over text-only models, while open-source alternatives trail significantly. Notably, fine-tuned LLMs achieve substantial improvements when using detailed image descriptions. MMDocRAG establishes a rigorous testing ground and provides actionable insights for developing more robust multimodal DocVQA systems.
+  </p>
+  <a href="">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/66337cb5bd8ef15a47e72ce0/o-9uRuQyFJNU-bHsE3LSR.png" alt="Logo" width="100%">
+  </a>
+<br>
 
 
-## 1. Abstract
-Multi-modal document retrieval is designed to identify and retrieve various forms of multi-modal content, such as figures, tables, charts, and layout information from extensive documents.
-Despite its significance, there is a notable lack of a robust benchmark to effectively evaluate the performance of systems in multi-modal document retrieval.
-To address this gap, this work introduces a new benchmark, named as **MMDocIR**, encompassing two distinct tasks: **page-level** and **layout-level** retrieval. 
-**The former focuses on localizing the most relevant pages within a long document, while the latter targets the detection of specific layouts, offering a more fine-grained granularity than whole-page analysis.**
-A layout can refer to a variety of elements such as textual paragraphs, equations, figures, tables, or charts. 
-The MMDocIR benchmark comprises a rich dataset featuring expertly annotated labels for 1,685 questions and bootstrapped labels for 173,843 questions, making it a pivotal resource for advancing multi-modal document retrieval for both training and evaluation. 
-Through rigorous experiments, we reveal that 
-(i) visual retrievers significantly outperform their text counterparts; 
-(ii) MMDocIR train set can effectively benefit the training process of multi-modal document retrieval;
-(iii) text retrievers leveraging on VLM-text perform much better than those using OCR-text. 
-These findings underscores the potential advantages of integrating visual elements for multi-modal document retrieval.
-
-
-## 2. Task Setting
-### Page-level Retrieval
-The page-level retrieval task is designed to identify the most relevant pages within a document in response to a user query.
-
-### Layout-level Retrieval
-The layout-level retrieval aims to retrieve most relevant layouts. 
-The layouts are defined as the fine-grained elements such as paragraphs, equations, figures, tables, and charts. 
-This task allows for a more nuanced content retrieval, honing in on specific information that directly answers user queries.
-
-
-
-## 3. Evaluation Set
-### 3.1 Document Analysis
-
-**MMDocIR** evaluation set includes 313 long documents averaging 65.1 pages, categorized into ten main domains: research reports, administration&industry, tutorials&workshops, academic papers, brochures, financial reports, guidebooks, government documents, laws, and news articles. 
-Different domains feature distinct distributions of multi-modal information. For instance, research reports, tutorials, workshops, and brochures predominantly contain images, whereas financial and industry documents are table-rich. In contrast, government and legal documents primarily comprise text. Overall, the modality distribution is: Text (60.4%), Image (18.8%), Table (16.7%), and other modalities (4.1%).
-
-### 3.2 Question and Annotation Analysis
-**MMDocIR** evluation set encompasses 1,658 questions, 2,107 page labels, and 2,638 layout labels. The modalities required to answer these questions distribute across four categories: Text (44.7%), Image (21.7%), Table (37.4%), and Layout/Meta (11.5%). The ``Layout/Meta'' category encompasses questions related to layout information and meta-data statistics. 
-Notably, the dataset poses several challenges: 254 questions necessitate cross-modal understanding, 313 questions demand evidence across multiple pages, and 637 questions require reasoning based on multiple layouts. These complexities highlight the need for advanced multi-modal reasoning and contextual understanding.
-
-
-## 4. Train Set
-
 
 
 
@@ -66,4 +111,15 @@ If you use any datasets from this organization in your research, please cite the
       primaryClass={cs.IR},
       url={https://arxiv.org/abs/2501.08828}, 
 }
+
+@misc{dong2025benchmarkingretrievalaugmentedmultimomalgeneration,
+      title={Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering}, 
+      author={Kuicai Dong and Yujing Chang and Shijie Huang and Yasheng Wang and Ruiming Tang and Yong Liu},
+      year={2025},
+      eprint={2505.16470},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR},
+      url={https://arxiv.org/abs/2505.16470}, 
+}
+
 ```