metadata
model-index:
- name: CodeLlama-7b
results:
- task:
type: code-generation
dataset:
name: Humaneval
type: humaneval
metrics:
- type: pass@1 (BASELINE)
value: 0.3048780487804878
- type: pass@1 (BASIC)
value: 0.3170731707317073
This is a d-Matrix functional reference of the CODELLAMA-7B model. The reference provides the following functional configurations:
| Configuration | Explanation |
|---|---|
BASELINE |
a reference functionally equivalent to the original model |
BASIC |
all linear algebraic operands quantized to MXINT8-64, and all other operations transformed to approximated kernel simulations |
Usage
Install d-Matrix Dmx_Compressor first.
pip install dmx_compressor
The following is an example model and its evaluation.
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
# code_eval.py included in HuggingFace repo
from code_eval import evaluate_model
eval_results = evaluate_model("d-matrix/CodeLlama-7b", dmx_config="BASELINE") # For BASELINE configuration