Spaces:
Running
Running
File size: 8,795 Bytes
29f689c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 |
# SMTR
- [SMTR](#smtr)
- [1. Introduction](#1-introduction)
- [2. Environment](#2-environment)
- [Dataset Preparation](#dataset-preparation)
- [3. Model Training / Evaluation](#3-model-training--evaluation)
- [Citation](#citation)
<a name="1"></a>
## 1. Introduction
Paper:
> [Out of Length Text Recognition with Sub-String Matching](https://arxiv.org/abs/2407.12317)
> Yongkun Du, Zhineng Chen\*, Caiyan Jia, Xieping Gao, Yu-Gang Jiang
<a name="model"></a>
Scene Text Recognition (STR) methods have demonstrated robust performance in word-level text recognition. However, in applications the text image is sometimes long due to detected with multiple horizontal words. It triggers the requirement to build long text recognition models from readily available short word-level text datasets, which has been less studied previously. In this paper, we term this the Out of Length (OOL) text recognition. We establish the first Long Text Benchmark (LTB) to facilitate the assessment of different methods in long text recognition. Meanwhile, we propose a novel method called OOL Text Recognition with sub-String Matching (SMTR). SMTR comprises two cross-attention-based modules: one encodes a sub-string containing multiple characters into next and previous queries, and the other employs the queries to attend to the image features, matching the sub-string and simultaneously recognizing its next and previous character. SMTR can recognize text of arbitrary length by iterating the process above. To avoid being trapped in recognizing highly similar sub-strings, we introduce a regularization training to compel SMTR to effectively discover subtle differences between similar sub-strings for precise matching. In addition, we propose an inference augmentation to alleviate confusion caused by identical sub-strings and improve the overall recognition efficiency. Extensive experimental results reveal that SMTR, even when trained exclusively on short text, outperforms existing methods in public short text benchmarks and exhibits a clear advantage on LTB.
The accuracy (%) and model files of SMTR on the public dataset of scene text recognition are as follows:
- Syn: Synth dataset(MJ+ST) from [PARSeq](https://github.com/baudm/parseq)
- U14M: Union14M-L from [Union14M](https://github.com/Mountchicken/Union14M/)
- Test on Long Text Benchmark ([Download LTB](https://drive.google.com/drive/folders/1NChdlw7ustbXtlFBmh_0xnHvRkffb9Ge?usp=sharing)):
| Model | Training Data | LTB | Config&Model&Log |
| :-------: | :-----------: | :--: | :---------------------------------------------------------------------------------------------: |
| SMTR | Syn | 39.6 | [link](https://drive.google.com/drive/folders/11SplakPPOFDMhPixv7ABNgjeTg4jKyfU?usp=sharing) |
| SMTR | U14M | 51.0 | [link](https://drive.google.com/drive/folders/1-K5O0d0q9fhY5fJvU6nn5fFFtSMnbE_-?usp=drive_link) |
| FocalSVTR | U14M | 42.1 | [link](https://drive.google.com/drive/folders/100xF5wFr7xSCVBYM1h_0d_8xv5Qeqobp?usp=sharing) |
- Test on Common Benchmarks from [PARSeq](https://github.com/baudm/parseq):
| Model | Training Data | IC13<br/>857 | SVT | IIIT5k<br/>3000 | IC15<br/>1811 | SVTP | CUTE80 | Avg | Config&Model&Log |
| :-------: | :-----------: | :----------: | :--: | :-------------: | :-----------: | :--: | :----: | :---: | :---------------------: |
| SMTR | Syn | 97.4 | 94.9 | 97.4 | 88.4 | 89.9 | 96.2 | 94.02 | Same as the above table |
| SMTR | U14M | 98.3 | 97.4 | 99.0 | 90.1 | 92.7 | 97.9 | 95.90 | Same as the above table |
| FocalSVTR | U14M | 97.3 | 96.3 | 98.2 | 87.4 | 88.4 | 96.2 | 93.97 | Same as the above table |
- Test on Union14M-L benchmark from [Union14M](https://github.com/Mountchicken/Union14M/).
| Model | Traing Data | Curve | Multi-<br/>Oriented | Artistic | Contextless | Salient | Multi-<br/>word | General | Avg | Config&Model&Log |
| :-------: | :---------: | :---: | :-----------------: | :------: | :---------: | :-----: | :-------------: | :-----: | :---: | :---------------------: |
| SMTR | Syn | 74.2 | 30.6 | 58.5 | 67.6 | 79.6 | 75.1 | 67.9 | 64.79 | Same as the above table |
| SMTR | U14M | 89.1 | 87.7 | 76.8 | 83.9 | 84.6 | 89.3 | 83.7 | 85.00 | Same as the above table |
| FocalSVTR | U14M | 77.7 | 62.4 | 65.7 | 78.6 | 71.6 | 81.3 | 79.2 | 73.80 | Same as the above table |
- Training and test on Chinese dataset, from [Chinese Benckmark](https://github.com/FudanVI/benchmarking-chinese-text-recognition).
| Model | Scene | Web | Document | Handwriting | Avg | Config&Model&Log |
| :-----------: | :---: | :--: | :------: | :---------: | :---: | :---------------------------------------------------------------------------------------------: |
| SMTR w/o Aug | 79.8 | 80.6 | 99.1 | 61.9 | 80.33 | [link](https://drive.google.com/drive/folders/1v8CK5GIu7wunnD5jFh2bLbusjyHeban5?usp=drive_link) |
| SMTR w/ Aug | 83.4 | 83.0 | 99.3 | 65.1 | 82.68 | [link](https://drive.google.com/drive/folders/1SQnwSm0bOBQ0eMKKD08F_4Blkjie_3la?usp=drive_link) |
Download all Configs, Models, and Logs from [Google Drive](https://drive.google.com/drive/folders/1dCuaWwCLP9xIHgy-7NtpeDLOvgk9NoKE?usp=drive_link).
<a name="2"></a>
## 2. Environment
- [PyTorch](http://pytorch.org/) version >= 1.13.0
- Python version >= 3.7
```shell
git clone -b develop https://github.com/Topdu/OpenOCR.git
cd OpenOCR
# A100 Ubuntu 20.04 Cuda 11.8
conda create -n openocr python==3.8
conda activate openocr
conda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt
```
#### Dataset Preparation
- [English dataset download](https://github.com/baudm/parseq)
- [Union14M-L download](https://github.com/Mountchicken/Union14M)
- [Chinese dataset download](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download)
- [LTB download](https://drive.google.com/drive/folders/1NChdlw7ustbXtlFBmh_0xnHvRkffb9Ge?usp=sharing)
The expected filesystem structure is as follows:
```
benchmark_bctr
├── benchmark_bctr_test
│ ├── document_test
│ ├── handwriting_test
│ ├── scene_test
│ └── web_test
└── benchmark_bctr_train
├── document_train
├── handwriting_train
├── scene_train
└── web_train
evaluation
├── CUTE80
├── IC13_857
├── IC15_1811
├── IIIT5k
├── SVT
└── SVTP
OpenOCR
synth
├── MJ
│ ├── test
│ ├── train
│ └── val
└── ST
test # from PARSeq
├── ArT
├── COCOv1.4
├── CUTE80
├── IC13_1015
├── IC13_1095
├── IC13_857
├── IC15_1811
├── IC15_2077
├── IIIT5k
├── SVT
├── SVTP
└── Uber
u14m # lmdb format
├── artistic
├── contextless
├── curve
├── general
├── multi_oriented
├── multi_words
└── salient
ltb # download link: https://drive.google.com/drive/folders/1NChdlw7ustbXtlFBmh_0xnHvRkffb9Ge?usp=sharing
Union14M-LMDB-L # lmdb format
├── train_challenging
├── train_easy
├── train_hard
├── train_medium
└── train_normal
```
<a name="3"></a>
## 3. Model Training / Evaluation
Training:
```shell
# Multi GPU training
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 tools/train_rec.py --c configs/rec/smtr/focalsvtr_smtr.yml
# For RTX 4090
NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 tools/train_rec.py --c configs/rec/smtr/focalsvtr_smtr.yml
```
Evaluation:
```shell
# en
python tools/eval_rec_all_ratio.py --c configs/rec/smtr/focalsvtr_smtr.yml
# long text
python tools/eval_rec_all_long_simple.py --c configs/rec/smtr/focalsvtr_smtr_long.yml
# ch
python tools/eval_rec_all_ch.py --c configs/rec/smtr/focalsvtr_smtr_ch.yml
```
## Citation
```bibtex
@article{Du2024SMTR,
title = {Out of Length Text Recognition with Sub-String Matching},
author = {Yongkun Du, Zhineng Chen, Caiyan Jia, Xieping Gao, Yu-Gang Jiang},
journal = {CoRR},
eprinttype = {arXiv},
primaryClass={cs.CV},
volume = {abs/2407.12317},
year = {2024},
url = {https://arxiv.org/abs/2407.12317}
}
```
|