Commit
•
07a62c4
1
Parent(s):
5f6b23e
Update README.md
Browse files
README.md
CHANGED
|
@@ -139,19 +139,21 @@ tensor([
|
|
| 139 |
|
| 140 |
## retrieval (JMTEB)
|
| 141 |
|
| 142 |
-
[JMTEB](https://github.com/sbintuitions/JMTEB) の評価結果です。
|
|
|
|
|
|
|
| 143 |
なお、japanese-splade-base-v2 は jaqket, mrtydi, jagovfaqs nlp_jornal のドメインを**学習していません**。
|
| 144 |
|
| 145 |
|
| 146 |
| モデル名 | jagovfaqs | jaqket | mrtydi | nlp_journal <br>title_abs | nlp_journal <br>abs_intro | nlp_journal <br>title_intro | Avg <br><512 | Avg <br>ALL |
|
| 147 |
| ----------------------------- | --------- | ------ | ---------- | ------------------------- | ------------------------- | --------------------------- | ------------- | ------------ |
|
| 148 |
-
| japanese-splade-base-v2 | 0.7313 | 0.6986 | **0.5106** | **0.9831** | 0.9067 | 0.8026 | **0.7309** |
|
| 149 |
| GLuCoSE-base-ja-v2 | 0.6979 | 0.6729 | 0.4186 | 0.9511 | 0.9029 | 0.7580 | 0.6851 | 0.7336 |
|
| 150 |
| multilingual-e5-large | 0.7030 | 0.5878 | 0.4363 | 0.9470 | 0.8600 | 0.7248 | 0.6685 | 0.7098 |
|
| 151 |
-
| ruri-large | 0.7668 | 0.6174 | 0.3803 | 0.9658 | 0.8712 | 0.7797 | 0.6826 | 0.7302 |
|
| 152 |
| jinaai/jina-embeddings-v3 | 0.7150 | 0.4648 | 0.4545 | 0.9562 | 0.9843 | 0.9385 | 0.6476 | 0.7522 |
|
| 153 |
-
| sarashina-embedding-v1-1b | 0.7168 | 0.7279 | 0.4195 | 0.9696 | 0.9394 | 0.8833 | 0.7085 | 0.7761 |
|
| 154 |
-
| OpenAI/text-embedding-3-large | 0.7241 | 0.4821 | 0.3488 | 0.9655 | 0.9933 | 0.9547 | 0.6301 | 0.7448 |
|
| 155 |
|
| 156 |
## 学習元データセット
|
| 157 |
|
|
|
|
| 139 |
|
| 140 |
## retrieval (JMTEB)
|
| 141 |
|
| 142 |
+
[JMTEB](https://github.com/sbintuitions/JMTEB) の評価結果です。
|
| 143 |
+
|
| 144 |
+
japanese-splade-base-v2 は [JMTEB をスパースベクトルで評価できるように変更したコード](https://github.com/hotchpotch/JMTEB/tree/add_splade)での評価となっています。
|
| 145 |
なお、japanese-splade-base-v2 は jaqket, mrtydi, jagovfaqs nlp_jornal のドメインを**学習していません**。
|
| 146 |
|
| 147 |
|
| 148 |
| モデル名 | jagovfaqs | jaqket | mrtydi | nlp_journal <br>title_abs | nlp_journal <br>abs_intro | nlp_journal <br>title_intro | Avg <br><512 | Avg <br>ALL |
|
| 149 |
| ----------------------------- | --------- | ------ | ---------- | ------------------------- | ------------------------- | --------------------------- | ------------- | ------------ |
|
| 150 |
+
| japanese-splade-base-v2 | 0.7313 | 0.6986 | **0.5106** | **0.9831** | 0.9067 | 0.8026 | **0.7309** | 0.7722 |
|
| 151 |
| GLuCoSE-base-ja-v2 | 0.6979 | 0.6729 | 0.4186 | 0.9511 | 0.9029 | 0.7580 | 0.6851 | 0.7336 |
|
| 152 |
| multilingual-e5-large | 0.7030 | 0.5878 | 0.4363 | 0.9470 | 0.8600 | 0.7248 | 0.6685 | 0.7098 |
|
| 153 |
+
| ruri-large | **0.7668** | 0.6174 | 0.3803 | 0.9658 | 0.8712 | 0.7797 | 0.6826 | 0.7302 |
|
| 154 |
| jinaai/jina-embeddings-v3 | 0.7150 | 0.4648 | 0.4545 | 0.9562 | 0.9843 | 0.9385 | 0.6476 | 0.7522 |
|
| 155 |
+
| sarashina-embedding-v1-1b | 0.7168 | **0.7279** | 0.4195 | 0.9696 | 0.9394 | 0.8833 | 0.7085 | **0.7761** |
|
| 156 |
+
| OpenAI/text-embedding-3-large | 0.7241 | 0.4821 | 0.3488 | 0.9655 | **0.9933** | **0.9547** | 0.6301 | 0.7448 |
|
| 157 |
|
| 158 |
## 学習元データセット
|
| 159 |
|