Update README.md

README.md

@@ -13,7 +13,7 @@ tags:
 
 
 ## Model Details
-We release ChatQA-1.5, which excels at RAG-based conversational question answering (QA). ChatQA-1.5 is built using the training recipe from [ChatQA (1.0)](https://arxiv.org/abs/2401.10225), and it is built on top of Llama-3 foundation model. Additionally, we incorporate more conversational QA data to enhance its tabular and arithmatic calculation capability. ChatQA-1.5 has two variants: ChatQA-1.5-8B and ChatQA-1.5-70B.
+We introduce ChatQA-1.5, which excels at conversational question answering (QA) and retrieval-augumented generation (RAG). ChatQA-1.5 is built using the training recipe from [ChatQA (1.0)](https://arxiv.org/abs/2401.10225), and it is built on top of Llama-3 foundation model. Additionally, we incorporate more conversational QA data to enhance its tabular and arithmatic calculation capability. ChatQA-1.5 has two variants: ChatQA-1.5-8B and ChatQA-1.5-70B. Both models were originally trained using [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), we converted the checkpoints to Hugging Face format.
 
 
 ## Benchmark Results
@@ -54,6 +54,9 @@ Assistant:
 
 
 ## How to use
+
+### take the whole document as context 
+This can be applied to the scenario where the whole document can be fitted into the model, so that there is no need to run retrieval over the document.
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 import torch
@@ -67,7 +70,7 @@ messages = [
     {"role": "user", "content": "what is the percentage change of the net income from Q4 FY23 to Q4 FY24?"}
 ]
 
-context = """NVIDIA (NASDAQ: NVDA) today reported revenue for the fourth quarter ended January 28, 2024, of $22.1 billion, up 22% from the previous quarter and up 265% from a year ago.\nFor the quarter, GAAP earnings per diluted share was $4.93, up 33% from the previous quarter and up 765% from a year ago. Non-GAAP earnings per diluted share was $5.16, up 28% from the previous quarter and up 486% from a year ago.\nQ4 Fiscal 2024 Summary\nGAAP\n| $ in millions, except earnings per share | Q4 FY24 | Q3 FY24 | Q4 FY23 | Q/Q | Y/Y |\n| Revenue | $22,103 | $18,120 | $6,051 | Up 22% | Up 265% |\n| Gross margin | 76.0% | 74.0% | 63.3% | Up 2.0 pts | Up 12.7 pts |\n| Operating expenses | $3,176 | $2,983 | $2,576 | Up 6% | Up 23% |\n| Operating income | $13,615 | $10,417 | $1,257 | Up 31% | Up 983% |\n| Net income | $12,285 | $9,243 | $1,414 | Up 33% | Up 769% |\n| Diluted earnings per share | $4.93 | $3.71 | $0.57 | Up 33% | Up 765% |"""
+document = """NVIDIA (NASDAQ: NVDA) today reported revenue for the fourth quarter ended January 28, 2024, of $22.1 billion, up 22% from the previous quarter and up 265% from a year ago.\nFor the quarter, GAAP earnings per diluted share was $4.93, up 33% from the previous quarter and up 765% from a year ago. Non-GAAP earnings per diluted share was $5.16, up 28% from the previous quarter and up 486% from a year ago.\nQ4 Fiscal 2024 Summary\nGAAP\n| $ in millions, except earnings per share | Q4 FY24 | Q3 FY24 | Q4 FY23 | Q/Q | Y/Y |\n| Revenue | $22,103 | $18,120 | $6,051 | Up 22% | Up 265% |\n| Gross margin | 76.0% | 74.0% | 63.3% | Up 2.0 pts | Up 12.7 pts |\n| Operating expenses | $3,176 | $2,983 | $2,576 | Up 6% | Up 23% |\n| Operating income | $13,615 | $10,417 | $1,257 | Up 31% | Up 983% |\n| Net income | $12,285 | $9,243 | $1,414 | Up 33% | Up 769% |\n| Diluted earnings per share | $4.93 | $3.71 | $0.57 | Up 33% | Up 765% |"""
 
 def get_formatted_input(messages, context):
     system = "System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context."
@@ -79,18 +82,12 @@ def get_formatted_input(messages, context):
             item['content'] = instruction + " " + item['content']
             break
 
-    conversation = ""
-    for item in messages:
-        if item["role"] == "user":
-            conversation += "User: " + item["content"] + "\n\n"
-        else:
-            conversation += "Assistant: " + item["content"] + "\n\n"
-    conversation += "Assistant:"
-
+    conversation = '\n\n'.join(["User: " + item["content"] if item["role"] == "user" else "Assistant: " + item["content"] for item in messages]) + "\n\nAssistant:"
     formatted_input = system + "\n\n" + context + "\n\n" + conversation
+    
     return formatted_input
 
-formatted_input = get_formatted_input(messages, context)
+formatted_input = get_formatted_input(messages, document)
 tokenized_prompt = tokenizer(tokenizer.bos_token + formatted_input, return_tensors="pt").to(model.device)
 
 terminators = [
@@ -104,6 +101,63 @@ response = outputs[0][tokenized_prompt.input_ids.shape[-1]:]
 print(tokenizer.decode(response, skip_special_tokens=True))
 ```
 
+### run retrieval to get top-n chunks as context
+This can be applied to the scenario when the document is very long, so that it is necessary to run retrieval. Here, we use our [Dragon-multiturn](https://huggingface.co/nvidia/dragon-multiturn-query-encoder) retriever which can handle conversatinoal query. In addition, we provide a few [documents]() for users to play with.
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModel
+import torch
+import json
+
+## load ChatQA-1.5 tokenizer and model
+model_id = "nvidia/ChatQA-1.5-70B"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
+
+## load retriever tokenizer and model
+retriever_tokenizer = AutoTokenizer.from_pretrained('nvidia/dragon-multiturn-query-encoder')
+query_encoder = AutoModel.from_pretrained('nvidia/dragon-multiturn-query-encoder')
+context_encoder = AutoModel.from_pretrained('nvidia/dragon-multiturn-context-encoder')
+
+## prepare documents, we take landrover car manual document that we provide as an example
+chunk_list = json.load(open("docs.json"))['landrover']
+
+messages = [
+    {"role": "user", "content": "how to connect the bluetooth in the car?"}
+]
+
+### running retrieval
+## convert query into a format as follows:
+## user: {user}\nagent: {agent}\nuser: {user}
+formatted_query_for_retriever = '\n'.join([turn['role'] + ": " + turn['content'] for turn in messages]).strip()
+
+query_input = retriever_tokenizer(formatted_query_for_retriever, return_tensors='pt')
+ctx_input = retriever_tokenizer(chunk_list, padding=True, truncation=True, max_length=512, return_tensors='pt')
+query_emb = query_encoder(**query_input).last_hidden_state[:, 0, :]
+ctx_emb = context_encoder(**ctx_input).last_hidden_state[:, 0, :]
+
+## Compute similarity scores using dot product and rank the similarity
+similarities = query_emb.matmul(ctx_emb.transpose(0, 1)) # (1, num_ctx)
+ranked_results = torch.argsort(similarities, dim=-1, descending=True) # (1, num_ctx)
+
+## get top-n chunks (n=5)
+retrieved_chunks = [chunk_list[idx] for idx in ranked_results.tolist()[0][:5]]
+context = "\n\n".join(retrieved_chunks)
+
+### running text generation
+formatted_input = get_formatted_input(messages, context)
+tokenized_prompt = tokenizer(tokenizer.bos_token + formatted_input, return_tensors="pt").to(model.device)
+
+terminators = [
+    tokenizer.eos_token_id,
+    tokenizer.convert_tokens_to_ids("<|eot_id|>")
+]
+outputs = model.generate(input_ids=tokenized_prompt.input_ids, attention_mask=tokenized_prompt.attention_mask, max_new_tokens=128, eos_token_id=terminators)
+
+response = outputs[0][tokenized_prompt.input_ids.shape[-1]:]
+print(tokenizer.decode(response, skip_special_tokens=True))
+```
+
 ## Correspondence to
 Zihan Liu ([email protected]), Wei Ping ([email protected])