Update README.md

README.md

@@ -59,3 +59,177 @@ The best way to use this model is via the [OCRFlux toolkit](https://github.com/c
 The toolkit comes with an efficient inference setup via vllm that can handle millions of documents
 at scale.
 
+### Benchmark for single-page parsing
+
+We ship two comprehensive benchmarks to help measure the performance of our OCR system in single-page parsing:
+
+  - [OCRFlux-bench-single](https://huggingface.co/datasets/ChatDOC/OCRFlux-bench-single): Containing 2000 pdf pages (1000 English pages and 1000 Chinese pages) and their ground-truth Markdowns (manually labeled with multi-round check).
+
+  - [OCRFlux-pubtabnet-single](https://huggingface.co/datasets/ChatDOC/OCRFlux-pubtabnet-single): Derived from the public [PubTabNet](https://github.com/ibm-aur-nlp/PubTabNet) benchmark with some format transformation. It contains 9064 HTML table samples, which are split into simple tables and complex tables according to whether they have rowspan and colspan cells.
+
+We emphasize that the released benchmarks are NOT included in our training and evaluation data. The following is the main result:
+
+
+1. In [OCRFlux-bench-single](https://huggingface.co/datasets/ChatDOC/OCRFlux-bench-single), we calculated the Edit Distance Similarity (EDS) between the generated Markdowns and the ground-truth Markdowns as the metric.
+
+    <table>
+      <thead>
+        <tr>
+          <th>Language</th>
+          <th>Model</th>
+          <th>Avg EDS ↑</th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td rowspan="4">English</td>
+          <td>olmOCR-7B-0225-preview</td>
+          <td>0.885</td>
+        </tr>
+        <tr>
+          <td>Nanonets-OCR-s</td>
+          <td>0.870</td>
+        </tr>
+        <tr>
+          <td>MonkeyOCR</td>
+          <td>0.828</td>
+        </tr>
+        <tr>
+          <td><strong><a href="https://huggingface.co/ChatDOC/OCRFlux-3B">OCRFlux-3B</a></strong></td>
+          <td>0.971</td>
+        </tr>
+        <tr>
+          <td rowspan="4">Chinese</td>
+          <td>olmOCR-7B-0225-preview</td>
+          <td>0.859</td>
+        </tr>
+        <tr>
+          <td>Nanonets-OCR-s</td>
+          <td>0.846</td>
+        </tr>
+        <tr>
+          <td>MonkeyOCR</td>
+          <td>0.731</td>
+        </tr>
+        <tr>
+          <td><strong><a href="https://huggingface.co/ChatDOC/OCRFlux-3B">OCRFlux-3B</a></strong></td>
+          <td>0.962</td>
+        </tr>
+        <tr>
+          <td rowspan="4">Total</td>
+          <td>olmOCR-7B-0225-preview</td>
+          <td>0.872</td>
+        </tr>
+        <tr>
+          <td>Nanonets-OCR-s</td>
+          <td>0.858</td>
+        </tr>
+        <tr>
+          <td>MonkeyOCR</td>
+          <td>0.780</td>
+        </tr>
+        <tr>
+          <td><strong><a href="https://huggingface.co/ChatDOC/OCRFlux-3B">OCRFlux-3B</a></strong></td>
+          <td>0.967</td>
+        </tr>
+      </tbody>
+    </table>
+
+2. In [OCRFlux-pubtabnet-single](https://huggingface.co/datasets/ChatDOC/OCRFlux-pubtabnet-single), we calculated the Tree Edit Distance-based Similarity (TEDS) between the generated HTML tables and the ground-truth HTML tables as the metric.
+    <table>
+      <thead>
+        <tr>
+          <th>Type</th>
+          <th>Model</th>
+          <th>Avg TEDS ↑</th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td rowspan="4">Simple</td>
+          <td>olmOCR-7B-0225-preview</td>
+          <td>0.810</td>
+        </tr>
+        <tr>
+          <td>Nanonets-OCR-s</td>
+          <td>0.882</td>
+        </tr>
+        <tr>
+          <td>MonkeyOCR</td>
+          <td>0.880</td>
+        </tr>
+        <tr>
+          <td><strong><a href="https://huggingface.co/ChatDOC/OCRFlux-3B">OCRFlux-3B</a></strong></td>
+          <td>0.912</td>
+        </tr>
+        <tr>
+          <td rowspan="4">Complex</td>
+          <td>olmOCR-7B-0225-preview</td>
+          <td>0.676</td>
+        </tr>
+        <tr>
+          <td>Nanonets-OCR-s</td>
+          <td>0.772</td>
+        </tr>
+        <tr>
+          <td><strong>MonkeyOCR<strong></td>
+          <td>0.826</td>
+        </tr>
+        <tr>
+          <td><a href="https://huggingface.co/ChatDOC/OCRFlux-3B">OCRFlux-3B</a></td>
+          <td>0.807</td>
+        </tr>
+        <tr>
+          <td rowspan="4">Total</td>
+          <td>olmOCR-7B-0225-preview</td>
+          <td>0.744</td>
+        </tr>
+        <tr>
+          <td>Nanonets-OCR-s</td>
+          <td>0.828</td>
+        </tr>
+        <tr>
+          <td>MonkeyOCR</td>
+          <td>0.853</td>
+        </tr>
+        <tr>
+          <td><strong><a href="https://huggingface.co/ChatDOC/OCRFlux-3B">OCRFlux-3B</a></strong></td>
+          <td>0.861</td>
+        </tr>
+      </tbody>
+    </table>
+
+We also conduct some case studies to show the superiority of our model in the [blog](https://ocrflux.pdfparser.io/#/blog) article.
+
+### Benchmark for cross-page table/paragraph merging
+
+PDF documents are typically paginated, which often results in tables or paragraphs being split across consecutive pages. Accurately detecting and merging such cross-page structures is crucial to avoid generating incomplete or fragmented content. 
+
+The detection task can be formulated as follows: given the Markdowns of two consecutive pages—each structured as a list of Markdown elements (e.g., paragraphs and tables)—the goal is to identify the indexes of elements that should be merged across the pages.
+
+Then for the merging task, if the elements to be merged are paragraphs, we can just concate them. However, for two table fragments, their merging is much more challenging. For example, the table spanning multiple pages will repeat the header of the first page on the second page. Another difficult scenario is that the table cell contains long content that spans multiple lines within the cell, with the first few lines appearing on the previous page and the remaining lines continuing on the next page. We also observe some cases where tables with a large number of columns are split vertically and placed on two consecutive pages. More examples of cross-page tables can be found in our [blog](https://ocrflux.pdfparser.io/#/blog) article. To address these issues, we develop the LLM model for cross-page table merging. Specifically, this model takes two split table fragments as input and generates a complete, well-structured table as output.
+
+We ship two comprehensive benchmarks to help measure the performance of our OCR system in cross-page table/paragraph detection and merging tasks respectively:
+
+  - [OCRFlux-bench-cross](https://huggingface.co/datasets/ChatDOC/OCRFlux-bench-cross): Containing 1000 samples (500 English samples and 500 Chinese samples), each sample contains the Markdown element lists of two consecutive pages, along with the indexes of elements that need to be merged (manually labeled through multiple rounds of review). If no tables or paragraphs require merging, the indexes in the annotation data are left empty.
+
+  - [OCRFlux-pubtabnet-cross](https://huggingface.co/datasets/ChatDOC/OCRFlux-pubtabnet-cross): Containing 9064 pairs of split table fragments, along with their corresponding ground-truth merged versions.
+
+The released benchmarks are NOT included in our training and evaluation data neither. The following is the main result:
+
+1. In [OCRFlux-bench-cross](https://huggingface.co/datasets/ChatDOC/OCRFlux-bench-cross), we caculated the Accuracy, Precision, Recall and F1 score as the metric. Notice that the detection results are right only when it accurately judges whether there are elements that need to be merged across the two pages and output the right indexes of them.
+
+    | Language | Precision ↑ | Recall ↑ | F1 ↑  | Accuracy ↑ |
+    |----------|-------------|----------|-------|------------|
+    | English  | 0.992       | 0.964    | 0.978 | 0.978      |
+    | Chinese  | 1.000       | 0.988    | 0.994 | 0.994      |
+    | Total    | 0.996       | 0.976    | 0.986 | 0.986      |
+
+2. In [OCRFlux-pubtabnet-cross](https://huggingface.co/datasets/ChatDOC/OCRFlux-pubtabnet-cross), we calculate the Tree Edit Distance-based Similarity (TEDS) between the generated merged table and the ground-truth merged table as the metric.
+
+    | Table type | Avg TEDS ↑   |
+    |------------|--------------|
+    | Simple     | 0.965        |
+    | Complex    | 0.935        |
+    | Total      | 0.950        |
+