Update README.md
Browse files
README.md
CHANGED
|
@@ -2347,7 +2347,7 @@ license: apache-2.0
|
|
| 2347 |
- **INF-Retriever-v1** is an LLM-based dense retrieval model developed by [INF TECH](https://www.infly.cn/en).
|
| 2348 |
It is built upon the [gte-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) model and specifically fine-tuned to excel in retrieval tasks, particularly for Chinese and English data.
|
| 2349 |
|
| 2350 |
-
- As of
|
| 2351 |
|
| 2352 |
## Key Features
|
| 2353 |
|
|
@@ -2379,7 +2379,7 @@ document_embeddings = model.encode(documents)
|
|
| 2379 |
|
| 2380 |
scores = (query_embeddings @ document_embeddings.T) * 100
|
| 2381 |
print(scores.tolist())
|
| 2382 |
-
# [[86.8702392578125, 67.
|
| 2383 |
```
|
| 2384 |
|
| 2385 |
### Transformers
|
|
@@ -2440,11 +2440,25 @@ print(scores.tolist())
|
|
| 2440 |
|
| 2441 |
### AIR-Bench
|
| 2442 |
|
| 2443 |
-
**INF-Retriever-v1** has demonstrated superior retrieval capabilities across multiple domains and languages. The results from the Automated Heterogeneous Information Retrieval Benchmark
|
| 2444 |
|
| 2445 |
-
|
| 2446 |
-
|
| 2447 |
-
|
|
| 2448 |
-
|
| 2449 |
-
| [
|
| 2450 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2347 |
- **INF-Retriever-v1** is an LLM-based dense retrieval model developed by [INF TECH](https://www.infly.cn/en).
|
| 2348 |
It is built upon the [gte-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) model and specifically fine-tuned to excel in retrieval tasks, particularly for Chinese and English data.
|
| 2349 |
|
| 2350 |
+
- As of January 23, 2025, **INF-Retriever-v1** ranks both **No.1** on the Automated Heterogeneous Information Retrieval Benchmark of version 24.04 & 24.05([AIR-Bench](https://huggingface.co/spaces/AIR-Bench/leaderboard)), showcasing its cutting-edge performance in heterogeneous information retrieval tasks.
|
| 2351 |
|
| 2352 |
## Key Features
|
| 2353 |
|
|
|
|
| 2379 |
|
| 2380 |
scores = (query_embeddings @ document_embeddings.T) * 100
|
| 2381 |
print(scores.tolist())
|
| 2382 |
+
# [[86.8702392578125, 67.82364654541016], [59.51014709472656, 82.33668518066406]]
|
| 2383 |
```
|
| 2384 |
|
| 2385 |
### Transformers
|
|
|
|
| 2440 |
|
| 2441 |
### AIR-Bench
|
| 2442 |
|
| 2443 |
+
**INF-Retriever-v1** has demonstrated superior retrieval capabilities across multiple domains and languages. The results from the Automated Heterogeneous Information Retrieval Benchmark ([AIR-Bench](https://huggingface.co/spaces/AIR-Bench/leaderboard)) as of January 23, 2025, are as follows:
|
| 2444 |
|
| 2445 |
+
#### AIR-Bench_24.04 (Bilingual, EN & ZH)
|
| 2446 |
+
|
| 2447 |
+
| Model Name | Average⬆️ | wiki_en | wiki_zh | web_en | web_zh | healthcare_en | healthcare_zh | law_en | arxiv_en | news_en | news_zh | finance_en | finance_zh | msmarco_en |
|
| 2448 |
+
|-----------------------------------------------------------------------------------|-----------|-----------|-----------|-----------|----------|---------------|---------------|-----------|-----------|-----------|-----------|------------|------------|------------|
|
| 2449 |
+
| [E5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) | 45.26 | 61.67 | 55.97 | 44.41 | 45.96 | 56.32 | 35.79 | 19.32 | 44.78 | 48.18 | 35.99 | 54.79 | 26.11 | 59.03 |
|
| 2450 |
+
| [BGE-M3](https://huggingface.co/BAAI/bge-m3) | 46.65 | 60.49 | 62.36 | 47.35 | 50.38 | 49.1 | **42.38** | 26.68 | 40.76 | 48.04 | 40.75 | 51.52 | 32.18 | 54.4 |
|
| 2451 |
+
| [BGE-Multilingual-Gemma2](https://huggingface.co/BAAI/bge-multilingual-gemma2) | 46.83 | 63.71 | 67.3 | 50.38 | 53.24 | 47.24 | 42.13 | 22.58 | 23.28 | 50.91 | 44.02 | 49.3 | 31.6 | **63.14** |
|
| 2452 |
+
| [GTE-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) | 48.38 | 63.46 | 66.44 | 51.2 | 51.98 | 54.2 | 38.82 | 22.31 | 40.27 | **54.07** | 43.03 | 58.2 | 26.63 | 58.39 |
|
| 2453 |
+
| **INF-Retriever-v1** | **52.56** | **65.25** | **68.44** | **52.13** | **56.6** | **56.96** | 42.03 | **34.51** | **50.62** | 53.32 | **50.02** | **58.34** | **35.42** | 59.64 |
|
| 2454 |
+
|
| 2455 |
+
#### AIR-Bench_24.05 (Multilingual, 13 languages)
|
| 2456 |
+
Although INF-Retriever-v1 has been fine-tuned exclusively on English and Chinese, it continues to perform exceptionally well across other languages, securing the No. 1 position on this multilingual benchmark.
|
| 2457 |
+
|
| 2458 |
+
| Model Name | Average⬆️ | wiki_en | wiki_zh | wiki_ar | wiki_bn | wiki_de | wiki_es | wiki_fa | wiki_fr | wiki_hi | wiki_id | wiki_ja | wiki_ko | wiki_ru | web_en | web_zh | web_ar | web_bn | web_de | web_es | web_fa | web_fr | web_hi | web_id | web_ja | web_ko | web_ru | healthcare_en | healthcare_zh | healthcare_de | healthcare_es | healthcare_fr | law_en | law_de | law_fr | arxiv_en | science_ru | news_en | news_zh | news_ar | news_bn | news_de | news_es | news_fa | news_fr | news_hi | news_id | news_ja | news_ko | news_ru | finance_en | finance_zh | finance_ar | finance_fr |
|
| 2459 |
+
|--------------------------------------------------------------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|----------|-----------|-----------|----------|--------|-----------|-----------|-----------|---------------|---------------|---------------|---------------|---------------|-----------|-----------|-----------|-----------|------------|-----------|-----------|-----------|-----------|-----------|----------|-----------|----------|-----------|-----------|-----------|-----------|-----------|------------|------------|------------|------------|
|
| 2460 |
+
| [GTE-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) | 50.05 | **73.59** | 67.5 | 59.44 | 58.17 | 63.96 | 67.62 | 57.05 | 70.32 | 60.54 | 61.81 | 62.88 | 59.17 | 62.95 | **58.99** | 51.66 | 55.56 | 51.45 | 48.62 | 54.11 | 49.54 | 55.16 | 53.06 | 55.51 | 57.27 | 57.54 | 55.88 | 54.46 | 38.66 | 53.92 | 53.78 | 30.29 | 22.75 | 13.18 | 13.15 | 41.32 | 45.21 | **52.74** | 43.17 | 37.63 | **61.31** | 44.89 | 45.21 | 30.1 | 49.76 | 30.28 | 46.44 | 44.13 | 47.19 | 46.55 | 59.23 | 34.61 | 43.56 | 39.57 |
|
| 2461 |
+
| [Multilingual-E5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct) | 51.11 | 68.62 | 62.82 | 63.21 | 64.45 | 65.81 | 68.1 | 64.2 | 69.72 | 71.81 | 66.36 | 64.12 | 64.79 | 62.57 | 41.58 | 47.06 | 56.4 | 56.17 | 50.87 | 52.24 | 58.68 | 50.2 | 56.32 | 54.49 | 54.89 | 55.81 | 54.97 | 54.02 | 39.76 | 52.06 | 51.74 | 36.64 | 16.9 | 15.59 | 15.12 | 39.52 | 56.86 | 44.28 | 35.46 | 48.2 | 49.31 | 47.84 | 45.99 | **45.59** | 50.58 | 39.66 | 48.59 | 47.6 | 50.52 | 48.81 | 52.79 | 37.72 | 48.95 | 42.74 |
|
| 2462 |
+
| [BGE-M3](https://huggingface.co/BAAI/bge-m3) | 51.31 | 69.7 | 63.52 | 59.65 | 64.33 | 64.68 | 65.4 | 61.14 | 66.04 | 69.02 | 66.3 | 60.86 | 62.36 | 60.18 | 53.88 | 50.2 | 52.53 | 55.53 | 51.89 | 51.78 | 55.81 | 51.46 | 57.06 | 53.14 | 54.75 | 55.28 | 54.53 | 49.05 | 42.31 | 49 | 53.05 | 39.29 | 26.95 | 20.11 | 20.2 | 41.64 | 55.18 | 47.34 | 41 | 44.93 | 59.03 | 47.87 | 44.7 | 43.81 | 49.52 | 42.12 | 47.45 | 47.09 | 48.14 | 48.31 | 52.92 | 40.23 | 45.76 | 41.44 |
|
| 2463 |
+
| [BGE-Multilingual-Gemma2](https://huggingface.co/BAAI/bge-multilingual-gemma2) | 54.46 | 72.8 | 68.64 | **63.42** | **69.48** | **67.91** | **71.79** | **67.57** | **71.28** | **75.39** | **68.91** | **68.29** | **66.78** | **64.15** | 56.48 | 53.04 | **59.97** | **59.68** | **57.72** | **58.2** | **62.43** | **59.54** | **64.5** | **60** | **60.26** | 59.64 | **60.12** | 47.48 | **42.35** | 55.4 | **63.13** | **45.13** | 22.6 | 15.75 | 14.29 | 24 | 44.13 | 50.29 | 43.42 | 48.41 | 58.77 | **52.05** | **49.9** | 43.4 | **56.8** | **44.89** | 50.65 | **51.51** | 51.64 | 51.48 | 50.08 | 39.23 | 50.25 | **51.1** |
|
| 2464 |
+
| **INF-Retriever-v1** | **54.47** | 73.52 | **69.45** | 63.13 | 61.58 | 66.8 | 69.29 | 63.03 | 69.74 | 69.02 | 68.63 | 63.45 | 64.44 | 62.74 | 57.6 | **56.46** | 58.48 | 53.7 | 55.2 | 57.08 | 53.27 | 57.35 | 55.64 | 58.85 | 59.52 | **60.01** | 58.79 | **57.03** | 41.82 | **55.46** | 57.6 | 43.25 | **34.76** | **21.75** | **21.87** | **51.38** | **59.72** | 52.7 | **49.78** | **49.11** | 43.62 | 51.47 | 49.52 | 40.43 | 54.54 | 38.57 | **51.06** | 51.12 | **53.15** | **51.88** | **59.44** | **44.13** | **50.71** | 44.2 |
|