Update README.md

README.md

@@ -1,3 +1,79 @@
 ---
 license: apache-2.0
+base_model: tiiuae/falcon-7b
+language:
+  - en
+tags:
+  - falcon-7b
+  - falcon
+  - onnxruntime
+  - onnx
+  - llm
 ---
+
+# falcon-7b for ONNX Runtime
+
+## Introduction
+
+This repository hosts the optimized version of **falcon-7b** to accelerate inference with ONNX Runtime CUDA execution provider.
+
+See the [usage instructions](#usage-example) for how to inference this model with the ONNX files hosted in this repository.
+
+## Model Description
+
+- **Developed by:** TIIUAE
+- **Model type:** Pretrained generative text model
+- **License:** Apache 2.0 License
+- **Model Description:** This is a conversion of the [falcon-7b](https://huggingface.co/tiiuae/falcon-7b) for [ONNX Runtime](https://github.com/microsoft/onnxruntime) inference with CUDA execution provider.
+
+
+## Performance Comparison
+
+#### Latency for token generation
+
+Below is average latency of generating a token using a prompt of varying size using NVIDIA A100-SXM4-80GB GPU:
+
+| Prompt Length      | Batch Size | PyTorch 2.1 torch.compile    | ONNX Runtime CUDA |
+|-------------|------------|----------------|-------------------|
+| 16      | 1          | N/A            | N/A           |
+| 256      | 1          | N/A            | N/A       |
+| 1024     | 1          | N/A        | N/A           |
+| 2048     | 1          | N/A       | N/A         |
+| 16      | 4          | N/A            | N/A           |
+| 256      | 4          | N/A            | N/A          |
+| 1024     | 4          | N/A        | N/A           |
+| 2048     | 4          | N/A       | N/A          |
+
+## Usage Example
+
+1. Clone onnxruntime repository.
+```shell
+git clone https://github.com/microsoft/onnxruntime
+cd onnxruntime
+```
+
+2. Install required dependencies
+```shell
+python3 -m pip install -r onnxruntime/python/tools/transformers/models/llama/requirements-cuda.txt
+```
+
+5. Inference using custom model API, or use Hugging Face's ORTModelForCausalLM
+```python
+from optimum.onnxruntime import ORTModelForCausalLM
+from onnxruntime import InferenceSession
+from transformers import AutoConfig, AutoTokenizer
+
+sess = InferenceSession("falcon-7b.onnx", providers = ["CUDAExecutionProvider"])
+config = AutoConfig.from_pretrained("tiiuae/falcon-7b")
+
+model = ORTFalconForCausalLM(sess, config, use_cache = True, use_io_binding = True)
+
+tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b")
+
+inputs = tokenizer("Instruct: What is a fermi paradox?\nOutput:", return_tensors="pt")
+
+outputs = model.generate(**inputs)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+