|
|
--- |
|
|
model-index: |
|
|
- name: CodeLlama-7b |
|
|
results: |
|
|
- task: |
|
|
type: code-generation |
|
|
dataset: |
|
|
name: Humaneval |
|
|
type: humaneval |
|
|
metrics: |
|
|
- type: pass@1 (BASELINE) |
|
|
value: 0.3048780487804878 |
|
|
- type: pass@1 (BASIC) |
|
|
value: 0.3170731707317073 |
|
|
--- |
|
|
This is a d-Matrix functional reference of the CODELLAMA-7B model. |
|
|
The reference provides the following functional *configurations*: |
|
|
Configuration | Explanation |
|
|
:-- | :-- |
|
|
**`BASELINE`** | a reference functionally equivalent to the original model |
|
|
**`BASIC`** | all linear algebraic operands quantized to `MXINT8-64` |
|
|
|
|
|
|
|
|
### Usage |
|
|
|
|
|
Install d-Matrix [Dmx_Compressor](https://github.com/d-matrix-ai/dmx-compressor) first. |
|
|
```sh |
|
|
pip install dmx_compressor |
|
|
``` |
|
|
|
|
|
The following is an example model and its evaluation. |
|
|
|
|
|
```sh |
|
|
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git |
|
|
cd bigcode-evaluation-harness |
|
|
pip install -e . |
|
|
``` |
|
|
|
|
|
```python |
|
|
# code_eval.py included in HuggingFace repo |
|
|
from code_eval import evaluate_model |
|
|
|
|
|
eval_results = evaluate_model("d-matrix/CodeLlama-7b", dmx_config="BASELINE") # For BASELINE configuration |
|
|
``` |
