Update README.md

README.md

@@ -5,137 +5,171 @@ tags:
 - sentence-transformers
 - sentence-similarity
 - feature-extraction
+license: mit
+datasets:
+- airesearch/WangchanX-Legal-ThaiCCL-RAG
+- VISAI-AI/nitibench
+language:
+- th
+- en
+base_model:
+- BAAI/bge-m3
 ---
 
-# SentenceTransformer
+# Auto-Finetuned BGE-M3 CCL
 
-This is a [sentence-transformers](https://www.SBERT.net) model trained. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+This is a finetuned [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) model on [`airesearch/WangchanX-Legal-ThaiCCL-RAG`](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG) queries.
+However, we didn't use the same positives as shown in the dataset card, the positives was collected without any human supervision.
 
-## Model Details
+## Finetuning Details
 
-### Model Description
-- **Model Type:** Sentence Transformer
-<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
-- **Maximum Sequence Length:** 8192 tokens
-- **Output Dimensionality:** 1024 tokens
-- **Similarity Function:** Cosine Similarity
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
+Apart from the original [`airesearch/WangchanX-Legal-ThaiCCL-RAG`](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG) which requires human to rerank and remove irrelevant documents, the model was finetuned on a completely automated environment.
+Specifically, given the query in the WangchanX-Legal-ThaiCCL-RAG dataset and a set of law sections to be retrieved, we follow the following procedure:
+1. Use [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) to retrieve N positive law sections based on thresholding score of 0.8
+2. Among those N documents, we use [`BAAI/bge-reranker-v2-m3`](https://huggingface.co/BAAI/bge-reranker-v2-m3) to rerank documents and filtered any document that reranker scores less than 0.8 - achieving final positive law sections
+3. Using positives from (2), we finetuned BGE-M3 model
 
-### Model Sources
+## Model Performance
+| **Dataset**      | **Top-K** | **HR@k** | **Multi HR@k** | **Recall@k** | **MRR@k** | **Multi MRR@k** |
+|:----------------:|:---------:|:-------:|:-------------:|:-----------:|:--------:|:---------------:|
+| **NitiBench-CCL**    | 1         | 0.731   | –             | 0.731       | 0.731    | –               |
+| **NitiBench-CCL**    | 5         | 0.900   | –             | 0.900       | 0.800    | –               |
+| **NitiBench-CCL**    | 10        | 0.934   | –             | 0.934       | 0.804    | –               |
+| **NitiBench-Tax**| 1         | 0.520   | 0.160         | 0.281       | 0.520    | 0.281           |
+| **NitiBench-Tax**| 5         | 0.700   | 0.200         | 0.382       | 0.587    | 0.329           |
+| **NitiBench-Tax**| 10        | 0.780   | 0.260         | 0.483       | 0.600    | 0.345           |
 
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
-- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
 
-### Full Model Architecture
+## Usage
 
+Install: 
+```
+git clone https://github.com/FlagOpen/FlagEmbedding.git
+cd FlagEmbedding
+pip install -e .
+```
+or: 
 ```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
-  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-  (2): Normalize()
-)
+pip install -U FlagEmbedding
 ```
 
-## Usage
 
-### Direct Usage (Sentence Transformers)
 
-First install the Sentence Transformers library:
+### Generate Embedding for text
 
-```bash
-pip install -U sentence-transformers
+- Dense Embedding
+```python
+from FlagEmbedding import BGEM3FlagModel
+
+model = BGEM3FlagModel('VISAI-AI/nitibench-ccl-auto-finetuned-bge-m3',  
+                       use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
+
+sentences_1 = ["What is BGE M3?", "Defination of BM25"]
+sentences_2 = ["BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", 
+               "BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document"]
+
+embeddings_1 = model.encode(sentences_1, 
+                            batch_size=12, 
+                            max_length=8192, # If you don't need such a long length, you can set a smaller value to speed up the encoding process.
+                            )['dense_vecs']
+embeddings_2 = model.encode(sentences_2)['dense_vecs']
+similarity = embeddings_1 @ embeddings_2.T
+print(similarity)
+# [[0.6265, 0.3477], [0.3499, 0.678 ]]
 ```
+You also can use sentence-transformers and huggingface transformers to generate dense embeddings.
+Refer to [baai_general_embedding](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/baai_general_embedding#usage) for details.
 
-Then you can load this model and run inference.
+
+- Sparse Embedding (Lexical Weight)
 ```python
-from sentence_transformers import SentenceTransformer
-
-# Download from the 🤗 Hub
-model = SentenceTransformer("sentence_transformers_model_id")
-# Run inference
-sentences = [
-    'The weather is lovely today.',
-    "It's so sunny outside!",
-    'He drove to the stadium.',
-]
-embeddings = model.encode(sentences)
-print(embeddings.shape)
-# [3, 1024]
-
-# Get the similarity scores for the embeddings
-similarities = model.similarity(embeddings, embeddings)
-print(similarities.shape)
-# [3, 3]
-```
+from FlagEmbedding import BGEM3FlagModel
 
-<!--
-### Direct Usage (Transformers)
+model = BGEM3FlagModel('VISAI-AI/nitibench-ccl-auto-finetuned-bge-m3',  use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
 
-<details><summary>Click to see the direct usage in Transformers</summary>
+sentences_1 = ["What is BGE M3?", "Defination of BM25"]
+sentences_2 = ["BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", 
+               "BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document"]
 
-</details>
--->
+output_1 = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=False)
+output_2 = model.encode(sentences_2, return_dense=True, return_sparse=True, return_colbert_vecs=False)
 
-<!--
-### Downstream Usage (Sentence Transformers)
+# you can see the weight for each token:
+print(model.convert_id_to_token(output_1['lexical_weights']))
+# [{'What': 0.08356, 'is': 0.0814, 'B': 0.1296, 'GE': 0.252, 'M': 0.1702, '3': 0.2695, '?': 0.04092}, 
+#  {'De': 0.05005, 'fin': 0.1368, 'ation': 0.04498, 'of': 0.0633, 'BM': 0.2515, '25': 0.3335}]
 
-You can finetune this model on your own dataset.
 
-<details><summary>Click to expand</summary>
+# compute the scores via lexical mathcing
+lexical_scores = model.compute_lexical_matching_score(output_1['lexical_weights'][0], output_2['lexical_weights'][0])
+print(lexical_scores)
+# 0.19554901123046875
 
-</details>
--->
+print(model.compute_lexical_matching_score(output_1['lexical_weights'][0], output_1['lexical_weights'][1]))
+# 0.0
+```
 
-<!--
-### Out-of-Scope Use
+- Multi-Vector (ColBERT)
+```python
+from FlagEmbedding import BGEM3FlagModel
 
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
+model = BGEM3FlagModel('VISAI-AI/nitibench-ccl-auto-finetuned-bge-m3',  use_fp16=True) 
 
-<!--
-## Bias, Risks and Limitations
+sentences_1 = ["What is BGE M3?", "Defination of BM25"]
+sentences_2 = ["BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", 
+               "BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document"]
 
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
+output_1 = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=True)
+output_2 = model.encode(sentences_2, return_dense=True, return_sparse=True, return_colbert_vecs=True)
 
-<!--
-### Recommendations
+print(model.colbert_score(output_1['colbert_vecs'][0], output_2['colbert_vecs'][0]))
+print(model.colbert_score(output_1['colbert_vecs'][0], output_2['colbert_vecs'][1]))
+# 0.7797
+# 0.4620
+```
 
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
 
-## Training Details
+### Compute score for text pairs
+Input a list of text pairs, you can get the scores computed by different methods.
+```python
+from FlagEmbedding import BGEM3FlagModel
 
-### Framework Versions
-- Python: 3.11.9
-- Sentence Transformers: 3.1.1
-- Transformers: 4.45.1
-- PyTorch: 2.4.1+cu121
-- Accelerate: 0.34.2
-- Datasets: 3.0.1
-- Tokenizers: 0.20.0
+model = BGEM3FlagModel('VISAI-AI/nitibench-ccl-auto-finetuned-bge-m3',  use_fp16=True) 
 
-## Citation
+sentences_1 = ["What is BGE M3?", "Defination of BM25"]
+sentences_2 = ["BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", 
+               "BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document"]
 
-### BibTeX
+sentence_pairs = [[i,j] for i in sentences_1 for j in sentences_2]
 
-<!--
-## Glossary
+print(model.compute_score(sentence_pairs, 
+                          max_passage_length=128, # a smaller max length leads to a lower latency
+                          weights_for_different_modes=[0.4, 0.2, 0.4])) # weights_for_different_modes(w) is used to do weighted sum: w[0]*dense_score + w[1]*sparse_score + w[2]*colbert_score
 
-*Clearly define terms in order to be accessible across audiences.*
--->
+# {
+#   'colbert': [0.7796499729156494, 0.4621465802192688, 0.4523794651031494, 0.7898575067520142], 
+#   'sparse': [0.195556640625, 0.00879669189453125, 0.0, 0.1802978515625], 
+#   'dense': [0.6259765625, 0.347412109375, 0.349853515625, 0.67822265625], 
+#   'sparse+dense': [0.482503205537796, 0.23454029858112335, 0.2332356721162796, 0.5122477412223816], 
+#   'colbert+sparse+dense': [0.6013619303703308, 0.3255828022956848, 0.32089319825172424, 0.6232916116714478]
+# }
+```
 
-<!--
-## Model Card Authors
 
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
+## Acknowledgement
+Thanks to Pirat Pothavorn for evaluating the model performance on NitiBench, Supavish Punchun for finetuning the model and preparing positives for finetuning. Additionally, we thank you all authors of this open-sourced project.
 
-<!--
-## Model Card Contact
+## Citation
 
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->
+### BibTeX
+```
+@misc{akarajaradwong2025nitibenchcomprehensivestudiesllm,
+      title={NitiBench: A Comprehensive Studies of LLM Frameworks Capabilities for Thai Legal Question Answering}, 
+      author={Pawitsapak Akarajaradwong and Pirat Pothavorn and Chompakorn Chaksangchaichot and Panuthep Tasawong and Thitiwat Nopparatbundit and Sarana Nutanong},
+      year={2025},
+      eprint={2502.10868},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2502.10868}, 
+}
+```