Update README.md

8b2d153 verified 8 days ago

10.8 kB

	---
	license: apache-2.0
	language:
	- ko
	- en
	metrics:
	- accuracy
	base_model:
	- BAAI/bge-reranker-v2-m3
	pipeline_tag: text-classification
	library_name: sentence-transformers
	---

	# Reranker (Cross-Encoder)

	Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. You can get a relevance score by inputting query and passage to the reranker. And the score can be mapped to a float value in [0,1] by sigmoid function.

	## Model Details
	- Base model : BAAI/bge-reranker-v2-m3
	- The multilingual model has been optimized for Korean.

	## Usage with Transformers

	```python
	from transformers import AutoTokenizer, AutoModelForSequenceClassification
	import torch

	model = AutoModelForSequenceClassification.from_pretrained('dragonkue/bge-reranker-v2-m3-ko')
	tokenizer = AutoTokenizer.from_pretrained('dragonkue/bge-reranker-v2-m3-ko')

	features = tokenizer([['몇 년도에 지방세외수입법이 시행됐을까?', '실무교육을 통해 ‘지방세외수입법’에 대한 자치단체의 관심을 제고하고 자치단체의 차질 없는 업무 추진을 지원하였다. 이러한 준비과정을 거쳐 2014년 8월 7일부터 ‘지방세외수입법’이 시행되었다.'],
	['몇 년도에 지방세외수입법이 시행됐을까?', '식품의약품안전처는 21일 국내 제약기업 유바이오로직스가 개발 중인 신종 코로나바이러스 감염증(코로나19) 백신 후보물질 ‘유코백-19’의 임상시험 계획을 지난 20일 승인했다고 밝혔다.']], padding=True, truncation=True, return_tensors="pt")

	model.eval()
	with torch.no_grad():
	logits = model(**features).logits
	scores = torch.sigmoid(logits)
	print(scores)
	# [9.9997962e-01 5.0702977e-07]
	```


	## Usage with SentenceTransformers
	First install the Sentence Transformers library:
	```
	pip install -U sentence-transformers
	```

	```python
	from sentence_transformers import CrossEncoder

	model = CrossEncoder('dragonkue/bge-reranker-v2-m3-ko', default_activation_function=torch.nn.Sigmoid())

	scores = model.predict([['몇 년도에 지방세외수입법이 시행됐을까?', '실무교육을 통해 ‘지방세외수입법’에 대한 자치단체의 관심을 제고하고 자치단체의 차질 없는 업무 추진을 지원하였다. 이러한 준비과정을 거쳐 2014년 8월 7일부터 ‘지방세외수입법’이 시행되었다.'],
	['몇 년도에 지방세외수입법이 시행됐을까?', '식품의약품안전처는 21일 국내 제약기업 유바이오로직스가 개발 중인 신종 코로나바이러스 감염증(코로나19) 백신 후보물질 ‘유코백-19’의 임상시험 계획을 지난 20일 승인했다고 밝혔다.']])
	print(scores)
	# [9.9997962e-01 5.0702977e-07]
	```

	## Usage with FlagEmbedding
	First install the FlagEmbedding library:
	```
	pip install -U FlagEmbedding
	```
	```python
	from FlagEmbedding import FlagReranker

	reranker = FlagReranker('dragonkue/bge-reranker-v2-m3-ko')

	scores = reranker.compute_score([['몇 년도에 지방세외수입법이 시행됐을까?', '실무교육을 통해 ‘지방세외수입법’에 대한 자치단체의 관심을 제고하고 자치단체의 차질 없는 업무 추진을 지원하였다. 이러한 준비과정을 거쳐 2014년 8월 7일부터 ‘지방세외수입법’이 시행되었다.'],
	['몇 년도에 지방세외수입법이 시행됐을까?', '식품의약품안전처는 21일 국내 제약기업 유바이오로직스가 개발 중인 신종 코로나바이러스 감염증(코로나19) 백신 후보물질 ‘유코백-19’의 임상시험 계획을 지난 20일 승인했다고 밝혔다.']], normalize=True)
	print(scores)
	# [9.9997962e-01 5.0702977e-07]
	```

	## Fine-tune
	Refer to https://github.com/FlagOpen/FlagEmbedding


	## Evaluation


	### Bi-encoder and Cross-encoder

	Bi-Encoders convert texts into fixed-size vectors and efficiently calculate similarities between them. They are fast and ideal for tasks like semantic search and classification, making them suitable for processing large datasets quickly.

	Cross-Encoders directly compare pairs of texts to compute similarity scores, providing more accurate results. While they are slower due to needing to process each pair, they excel in re-ranking top results and are important in Advanced RAG techniques for enhancing text generation.


	### Korean Embedding Benchmark with AutoRAG
	(https://github.com/Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark)

	This is a Korean embedding benchmark for the financial sector.


	Top-k 1

	Bi-Encoder (Sentence Transformer)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| paraphrase-multilingual-mpnet-base-v2 \| 0.3596 \| 0.3596 \| 0.3596 \|
	\| KoSimCSE-roberta \| 0.4298 \| 0.4298 \| 0.4298 \|
	\| Cohere embed-multilingual-v3.0 \| 0.3596 \| 0.3596 \| 0.3596 \|
	\| openai ada 002 \| 0.4737 \| 0.4737 \| 0.4737 \|
	\| multilingual-e5-large-instruct \| 0.4649 \| 0.4649 \| 0.4649 \|
	\| Upstage Embedding \| 0.6579 \| 0.6579 \| 0.6579 \|
	\| paraphrase-multilingual-MiniLM-L12-v2 \| 0.2982 \| 0.2982 \| 0.2982 \|
	\| openai_embed_3_small \| 0.5439 \| 0.5439 \| 0.5439 \|
	\| ko-sroberta-multitask \| 0.4211 \| 0.4211 \| 0.4211 \|
	\| openai_embed_3_large \| 0.6053 \| 0.6053 \| 0.6053 \|
	\| KU-HIAI-ONTHEIT-large-v1 \| 0.7105 \| 0.7105 \| 0.7105 \|
	\| KU-HIAI-ONTHEIT-large-v1.1 \| 0.7193 \| 0.7193 \| 0.7193 \|
	\| kf-deberta-multitask \| 0.4561 \| 0.4561 \| 0.4561 \|
	\| gte-multilingual-base \| 0.5877 \| 0.5877 \| 0.5877 \|
	\| BGE-m3 \| 0.6578 \| 0.6578 \| 0.6578 \|
	\| bge-m3-korean \| 0.5351 \| 0.5351 \| 0.5351 \|
	\| BGE-m3-ko \| 0.7456 \| 0.7456 \| 0.7456 \|


	Cross-Encoder (Reranker)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| gte-multilingual-reranker-base \| 0.7281 \| 0.7281 \| 0.7281 \|
	\| jina-reranker-v2-base-multilingual \| 0.8070 \| 0.8070 \| 0.8070 \|
	\| bge-reranker-v2-m3 \| 0.8772 \| 0.8772 \| 0.8772 \|
	\| bge-reranker-v2-m3-ko \| 0.9123 \| 0.9123 \| 0.9123 \|


	Top-k 3

	Bi-Encoder (Sentence Transformer)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| paraphrase-multilingual-mpnet-base-v2 \| 0.2368 \| 0.4737 \| 0.1579 \|
	\| KoSimCSE-roberta \| 0.3026 \| 0.6053 \| 0.2018 \|
	\| Cohere embed-multilingual-v3.0 \| 0.2851 \| 0.5702 \| 0.1901 \|
	\| openai ada 002 \| 0.3553 \| 0.7105 \| 0.2368 \|
	\| multilingual-e5-large-instruct \| 0.3333 \| 0.6667 \| 0.2222 \|
	\| Upstage Embedding \| 0.4211 \| 0.8421 \| 0.2807 \|
	\| paraphrase-multilingual-MiniLM-L12-v2 \| 0.2061 \| 0.4123 \| 0.1374 \|
	\| openai_embed_3_small \| 0.3640 \| 0.7281 \| 0.2427 \|
	\| ko-sroberta-multitask \| 0.2939 \| 0.5877 \| 0.1959 \|
	\| openai_embed_3_large \| 0.3947 \| 0.7895 \| 0.2632 \|
	\| KU-HIAI-ONTHEIT-large-v1 \| 0.4386 \| 0.8772 \| 0.2924 \|
	\| KU-HIAI-ONTHEIT-large-v1.1 \| 0.4430 \| 0.8860 \| 0.2953 \|
	\| kf-deberta-multitask \| 0.3158 \| 0.6316 \| 0.2105 \|
	\| gte-multilingual-base \| 0.4035 \| 0.8070 \| 0.2690 \|
	\| BGE-m3 \| 0.4254 \| 0.8508 \| 0.2836 \|
	\| bge-m3-korean \| 0.3684 \| 0.7368 \| 0.2456 \|
	\| BGE-m3-ko \| 0.4517 \| 0.9035 \| 0.3011 \|

	Cross-Encoder (Reranker)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| gte-multilingual-reranker-base \| 0.4605 \| 0.9211 \| 0.3070 \|
	\| jina-reranker-v2-base-multilingual \| 0.4649 \| 0.9298 \| 0.3099 \|
	\| bge-reranker-v2-m3 \| 0.4781 \| 0.9561 \| 0.3187 \|
	\| bge-reranker-v2-m3-ko \| 0.4825 \| 0.9649 \| 0.3216 \|


	Top-k 5

	Bi-Encoder (Sentence Transformer)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| paraphrase-multilingual-mpnet-base-v2 \| 0.1813 \| 0.5439 \| 0.1088 \|
	\| KoSimCSE-roberta \| 0.2164 \| 0.6491 \| 0.1298 \|
	\| Cohere embed-multilingual-v3.0 \| 0.2076 \| 0.6228 \| 0.1246 \|
	\| openai ada 002 \| 0.2602 \| 0.7807 \| 0.1561 \|
	\| multilingual-e5-large-instruct \| 0.2544 \| 0.7632 \| 0.1526 \|
	\| Upstage Embedding \| 0.2982 \| 0.8947 \| 0.1789 \|
	\| paraphrase-multilingual-MiniLM-L12-v2 \| 0.1637 \| 0.4912 \| 0.0982 \|
	\| openai_embed_3_small \| 0.2690 \| 0.8070 \| 0.1614 \|
	\| ko-sroberta-multitask \| 0.2164 \| 0.6491 \| 0.1298 \|
	\| openai_embed_3_large \| 0.2807 \| 0.8421 \| 0.1684 \|
	\| KU-HIAI-ONTHEIT-large-v1 \| 0.3041 \| 0.9123 \| 0.1825 \|
	\| KU-HIAI-ONTHEIT-large-v1.1 \| 0.3099 \| 0.9298 \| 0.1860 \|
	\| kf-deberta-multitask \| 0.2281 \| 0.6842 \| 0.1368 \|
	\| gte-multilingual-base \| 0.2865 \| 0.8596 \| 0.1719 \|
	\| BGE-m3 \| 0.3041 \| 0.9123 \| 0.1825 \|
	\| bge-m3-korean \| 0.2661 \| 0.7982 \| 0.1596 \|
	\| BGE-m3-ko \| 0.3099 \| 0.9298 \| 0.1860 \|

	Cross-Encoder (Reranker)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| gte-multilingual-reranker-base \| 0.3158 \| 0.9474 \| 0.1895 \|
	\| jina-reranker-v2-base-multilingual \| 0.3129 \| 0.9386 \| 0.1877 \|
	\| bge-reranker-v2-m3 \| 0.3216 \| 0.9649 \| 0.1930 \|
	\| bge-reranker-v2-m3-ko \| 0.3216 \| 0.9649 \| 0.1930 \|

	---
	license: apache-2.0
	language:
	- ko
	- en
	metrics:
	- accuracy
	base_model:
	- BAAI/bge-reranker-v2-m3
	pipeline_tag: text-classification
	library_name: sentence-transformers
	---

	# Reranker (Cross-Encoder)

	Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. You can get a relevance score by inputting query and passage to the reranker. And the score can be mapped to a float value in [0,1] by sigmoid function.

	## Model Details
	- Base model : BAAI/bge-reranker-v2-m3
	- The multilingual model has been optimized for Korean.

	## Usage with Transformers

	```python
	from transformers import AutoTokenizer, AutoModelForSequenceClassification
	import torch

	model = AutoModelForSequenceClassification.from_pretrained('dragonkue/bge-reranker-v2-m3-ko')
	tokenizer = AutoTokenizer.from_pretrained('dragonkue/bge-reranker-v2-m3-ko')

	features = tokenizer([['몇 년도에 지방세외수입법이 시행됐을까?', '실무교육을 통해 ‘지방세외수입법’에 대한 자치단체의 관심을 제고하고 자치단체의 차질 없는 업무 추진을 지원하였다. 이러한 준비과정을 거쳐 2014년 8월 7일부터 ‘지방세외수입법’이 시행되었다.'],
	['몇 년도에 지방세외수입법이 시행됐을까?', '식품의약품안전처는 21일 국내 제약기업 유바이오로직스가 개발 중인 신종 코로나바이러스 감염증(코로나19) 백신 후보물질 ‘유코백-19’의 임상시험 계획을 지난 20일 승인했다고 밝혔다.']], padding=True, truncation=True, return_tensors="pt")

	model.eval()
	with torch.no_grad():
	logits = model(**features).logits
	scores = torch.sigmoid(logits)
	print(scores)
	# [9.9997962e-01 5.0702977e-07]
	```


	## Usage with SentenceTransformers
	First install the Sentence Transformers library:
	```
	pip install -U sentence-transformers
	```

	```python
	from sentence_transformers import CrossEncoder

	model = CrossEncoder('dragonkue/bge-reranker-v2-m3-ko', default_activation_function=torch.nn.Sigmoid())

	scores = model.predict([['몇 년도에 지방세외수입법이 시행됐을까?', '실무교육을 통해 ‘지방세외수입법’에 대한 자치단체의 관심을 제고하고 자치단체의 차질 없는 업무 추진을 지원하였다. 이러한 준비과정을 거쳐 2014년 8월 7일부터 ‘지방세외수입법’이 시행되었다.'],
	['몇 년도에 지방세외수입법이 시행됐을까?', '식품의약품안전처는 21일 국내 제약기업 유바이오로직스가 개발 중인 신종 코로나바이러스 감염증(코로나19) 백신 후보물질 ‘유코백-19’의 임상시험 계획을 지난 20일 승인했다고 밝혔다.']])
	print(scores)
	# [9.9997962e-01 5.0702977e-07]
	```

	## Usage with FlagEmbedding
	First install the FlagEmbedding library:
	```
	pip install -U FlagEmbedding
	```
	```python
	from FlagEmbedding import FlagReranker

	reranker = FlagReranker('dragonkue/bge-reranker-v2-m3-ko')

	scores = reranker.compute_score([['몇 년도에 지방세외수입법이 시행됐을까?', '실무교육을 통해 ‘지방세외수입법’에 대한 자치단체의 관심을 제고하고 자치단체의 차질 없는 업무 추진을 지원하였다. 이러한 준비과정을 거쳐 2014년 8월 7일부터 ‘지방세외수입법’이 시행되었다.'],
	['몇 년도에 지방세외수입법이 시행됐을까?', '식품의약품안전처는 21일 국내 제약기업 유바이오로직스가 개발 중인 신종 코로나바이러스 감염증(코로나19) 백신 후보물질 ‘유코백-19’의 임상시험 계획을 지난 20일 승인했다고 밝혔다.']], normalize=True)
	print(scores)
	# [9.9997962e-01 5.0702977e-07]
	```

	## Fine-tune
	Refer to https://github.com/FlagOpen/FlagEmbedding


	## Evaluation


	### Bi-encoder and Cross-encoder

	Bi-Encoders convert texts into fixed-size vectors and efficiently calculate similarities between them. They are fast and ideal for tasks like semantic search and classification, making them suitable for processing large datasets quickly.

	Cross-Encoders directly compare pairs of texts to compute similarity scores, providing more accurate results. While they are slower due to needing to process each pair, they excel in re-ranking top results and are important in Advanced RAG techniques for enhancing text generation.


	### Korean Embedding Benchmark with AutoRAG
	(https://github.com/Marker-Inc-Korea/AutoRAG-example-korean-embedding-benchmark)

	This is a Korean embedding benchmark for the financial sector.


	Top-k 1

	Bi-Encoder (Sentence Transformer)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| paraphrase-multilingual-mpnet-base-v2 \| 0.3596 \| 0.3596 \| 0.3596 \|
	\| KoSimCSE-roberta \| 0.4298 \| 0.4298 \| 0.4298 \|
	\| Cohere embed-multilingual-v3.0 \| 0.3596 \| 0.3596 \| 0.3596 \|
	\| openai ada 002 \| 0.4737 \| 0.4737 \| 0.4737 \|
	\| multilingual-e5-large-instruct \| 0.4649 \| 0.4649 \| 0.4649 \|
	\| Upstage Embedding \| 0.6579 \| 0.6579 \| 0.6579 \|
	\| paraphrase-multilingual-MiniLM-L12-v2 \| 0.2982 \| 0.2982 \| 0.2982 \|
	\| openai_embed_3_small \| 0.5439 \| 0.5439 \| 0.5439 \|
	\| ko-sroberta-multitask \| 0.4211 \| 0.4211 \| 0.4211 \|
	\| openai_embed_3_large \| 0.6053 \| 0.6053 \| 0.6053 \|
	\| KU-HIAI-ONTHEIT-large-v1 \| 0.7105 \| 0.7105 \| 0.7105 \|
	\| KU-HIAI-ONTHEIT-large-v1.1 \| 0.7193 \| 0.7193 \| 0.7193 \|
	\| kf-deberta-multitask \| 0.4561 \| 0.4561 \| 0.4561 \|
	\| gte-multilingual-base \| 0.5877 \| 0.5877 \| 0.5877 \|
	\| BGE-m3 \| 0.6578 \| 0.6578 \| 0.6578 \|
	\| bge-m3-korean \| 0.5351 \| 0.5351 \| 0.5351 \|
	\| BGE-m3-ko \| 0.7456 \| 0.7456 \| 0.7456 \|


	Cross-Encoder (Reranker)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| gte-multilingual-reranker-base \| 0.7281 \| 0.7281 \| 0.7281 \|
	\| jina-reranker-v2-base-multilingual \| 0.8070 \| 0.8070 \| 0.8070 \|
	\| bge-reranker-v2-m3 \| 0.8772 \| 0.8772 \| 0.8772 \|
	\| bge-reranker-v2-m3-ko \| 0.9123 \| 0.9123 \| 0.9123 \|


	Top-k 3

	Bi-Encoder (Sentence Transformer)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| paraphrase-multilingual-mpnet-base-v2 \| 0.2368 \| 0.4737 \| 0.1579 \|
	\| KoSimCSE-roberta \| 0.3026 \| 0.6053 \| 0.2018 \|
	\| Cohere embed-multilingual-v3.0 \| 0.2851 \| 0.5702 \| 0.1901 \|
	\| openai ada 002 \| 0.3553 \| 0.7105 \| 0.2368 \|
	\| multilingual-e5-large-instruct \| 0.3333 \| 0.6667 \| 0.2222 \|
	\| Upstage Embedding \| 0.4211 \| 0.8421 \| 0.2807 \|
	\| paraphrase-multilingual-MiniLM-L12-v2 \| 0.2061 \| 0.4123 \| 0.1374 \|
	\| openai_embed_3_small \| 0.3640 \| 0.7281 \| 0.2427 \|
	\| ko-sroberta-multitask \| 0.2939 \| 0.5877 \| 0.1959 \|
	\| openai_embed_3_large \| 0.3947 \| 0.7895 \| 0.2632 \|
	\| KU-HIAI-ONTHEIT-large-v1 \| 0.4386 \| 0.8772 \| 0.2924 \|
	\| KU-HIAI-ONTHEIT-large-v1.1 \| 0.4430 \| 0.8860 \| 0.2953 \|
	\| kf-deberta-multitask \| 0.3158 \| 0.6316 \| 0.2105 \|
	\| gte-multilingual-base \| 0.4035 \| 0.8070 \| 0.2690 \|
	\| BGE-m3 \| 0.4254 \| 0.8508 \| 0.2836 \|
	\| bge-m3-korean \| 0.3684 \| 0.7368 \| 0.2456 \|
	\| BGE-m3-ko \| 0.4517 \| 0.9035 \| 0.3011 \|

	Cross-Encoder (Reranker)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| gte-multilingual-reranker-base \| 0.4605 \| 0.9211 \| 0.3070 \|
	\| jina-reranker-v2-base-multilingual \| 0.4649 \| 0.9298 \| 0.3099 \|
	\| bge-reranker-v2-m3 \| 0.4781 \| 0.9561 \| 0.3187 \|
	\| bge-reranker-v2-m3-ko \| 0.4825 \| 0.9649 \| 0.3216 \|


	Top-k 5

	Bi-Encoder (Sentence Transformer)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| paraphrase-multilingual-mpnet-base-v2 \| 0.1813 \| 0.5439 \| 0.1088 \|
	\| KoSimCSE-roberta \| 0.2164 \| 0.6491 \| 0.1298 \|
	\| Cohere embed-multilingual-v3.0 \| 0.2076 \| 0.6228 \| 0.1246 \|
	\| openai ada 002 \| 0.2602 \| 0.7807 \| 0.1561 \|
	\| multilingual-e5-large-instruct \| 0.2544 \| 0.7632 \| 0.1526 \|
	\| Upstage Embedding \| 0.2982 \| 0.8947 \| 0.1789 \|
	\| paraphrase-multilingual-MiniLM-L12-v2 \| 0.1637 \| 0.4912 \| 0.0982 \|
	\| openai_embed_3_small \| 0.2690 \| 0.8070 \| 0.1614 \|
	\| ko-sroberta-multitask \| 0.2164 \| 0.6491 \| 0.1298 \|
	\| openai_embed_3_large \| 0.2807 \| 0.8421 \| 0.1684 \|
	\| KU-HIAI-ONTHEIT-large-v1 \| 0.3041 \| 0.9123 \| 0.1825 \|
	\| KU-HIAI-ONTHEIT-large-v1.1 \| 0.3099 \| 0.9298 \| 0.1860 \|
	\| kf-deberta-multitask \| 0.2281 \| 0.6842 \| 0.1368 \|
	\| gte-multilingual-base \| 0.2865 \| 0.8596 \| 0.1719 \|
	\| BGE-m3 \| 0.3041 \| 0.9123 \| 0.1825 \|
	\| bge-m3-korean \| 0.2661 \| 0.7982 \| 0.1596 \|
	\| BGE-m3-ko \| 0.3099 \| 0.9298 \| 0.1860 \|

	Cross-Encoder (Reranker)

	\| Model name \| F1 \| Recall \| Precision \|
	\|---------------------------------------\|------------\|------------\|------------\|
	\| gte-multilingual-reranker-base \| 0.3158 \| 0.9474 \| 0.1895 \|
	\| jina-reranker-v2-base-multilingual \| 0.3129 \| 0.9386 \| 0.1877 \|
	\| bge-reranker-v2-m3 \| 0.3216 \| 0.9649 \| 0.1930 \|
	\| bge-reranker-v2-m3-ko \| 0.3216 \| 0.9649 \| 0.1930 \|