Update README.md

README.md

@@ -28,9 +28,94 @@ language:
 <b>The crispy rerank family from <a href="https://mixedbread.com"><b>Mixedbread</b></a>.</b>
 </p>
 
-<p align="center">
-<sup> 🍞 Looking for a simple end-to-end retrieval solution? Meet Omni, our multimodal and multilingual model. <a href="https://mixedbread.com"><b>Get in touch for access.</a> </sup>
-</p>
+
+# Rewritten Mixedbread Reranker as classifier
+
+This repo is the Mixedbread reranker rewritten as Classifier that is as of March 2025 the most powerful reranker, e.g. for RAG.
+
+# FP8 Deployment on H100
+
+```yaml
+build_commands: []
+environment_variables: {}
+external_package_dirs: []
+model_metadata:
+  example_model_input:
+    input: 'ERROR: This redirects to the embedding endpoint. Use the /sync API to
+      reach /sync/predict'
+model_name: BEI-mixedbread-ai-mxbai-rerank-base-v2-reranker-fp8-truss-example
+python_version: py39
+requirements: []
+resources:
+  accelerator: L4
+  cpu: '1'
+  memory: 10Gi
+  use_gpu: true
+secrets: {}
+system_packages: []
+trt_llm:
+  build:
+    base_model: encoder
+    checkpoint_repository:
+      repo: michaelfeil/mxbai-rerank-base-v2-seq
+      revision: main
+      source: HF
+    max_num_tokens: 32768
+    max_seq_len: 1000001
+    num_builder_gpus: 4
+    quantization_type: fp8
+```
+
+To push the deployment on Baseten.co
+```bash
+truss push --publish
+```
+More info:
+https://github.com/basetenlabs/truss-examples/tree/main/11-embeddings-reranker-classification-tensorrt/BEI-mixedbread-ai-mxbai-rerank-base-v2-reranker-fp8
+
+## Usage as classifier
+For usage with Baseten.co or with github.com/michaelfeil/infinity, you should use the classification API. 
+You need to manually create the following prompt template that is very specfic to this model. 
+This template follows the reference implementation at https://github.com/mixedbread-ai/mxbai-rerank/tree/main.
+
+```python
+def create_mxbai_v2_reranker_prompt_template(query: str, document: str, instruction: str = "") -> str:
+    """
+    Create a carefully formatted chat template string (without tokenizer) for ranking relevance.
+
+    Parameters:
+        query (str): The search query.
+        document (str): The document text to evaluate.
+        instruction (str): Special instructions (e.g., "You are an expert for Mockingbirds.")
+
+    Returns:
+        str: The formatted chat template.
+    """
+    instruction = f"instruction: {instruction}\n" if instruction else ""    
+    # fixed system prompt, keep as is.
+    system_prompt = "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."
+    assert not "\n" in system_prompt
+    assert not "\n" in instruction[:-1]
+    assert isinstance(query, str)
+    assert isinstance(document, str)
+    templated = (
+        # keep spacing, newlines as is.
+        # template for mixedbread reranker v2
+        # https://huggingface.co/michaelfeil/mxbai-rerank-base-v2-seq/
+       f"<|endoftext|><|im_start|>system\n{system_prompt}\n"
+        "<|im_end|>\n"
+        "<|im_start|>user\n"
+        f"{instruction}"
+        f"query: {query} \n"
+        f"document: {document} \n"
+        "You are a search relevance expert who evaluates how well documents match search queries. "
+        "For each query-document pair, carefully analyze the semantic relationship between them, then provide your binary relevance judgment (0 for not relevant, 1 for relevant).\n"
+        "Relevance:<|im_end|>\n"
+        "<|im_start|>assistant\n"
+    )
+    return templated
+```
+
 
 # 🍞 mxbai-rerank-large-v2