Exl2 version of Undi95/OpenDolphinMaid-4x7b

branch

main : 8bpw h8
b8h8 : 8bpw h8

Using ThePile 0007.parquet as dataset

Quantization settings : python convert.py -i models/flemmingmiguel_TurdusDareBeagle-7B -o TurdusDareBeagle-7B-temp -cf TurdusDareBeagle-7B-8bpw-h8-exl2 -c 0007.parquet -l 8192 -b 8 -hb 8 -ml 8192

below this line is original readme

TurdusDareBeagle-7B

TurdusDareBeagle-7B is a merge of the following models using LazyMergekit:

As an experiment to find the best base merge to further fine-tuning, expect a lot of experiments named using parts of the component models until a clear winner emerges in the benchmarks

In this case .

🧩 Configuration

slices:
    - sources:
      - model: udkai/Turdus
        layer_range: [0, 32]
      - model: flemmingmiguel/DareBeagle-7B
        layer_range: [0, 32]
merge_method: slerp
base_model: flemmingmiguel/DareBeagle-7B
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.45 # fallback for rest of tensors
dtype: float16

πŸ’» Usage

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "flemmingmiguel/TurdusDareBeagle-7B"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
Downloads last month
15
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.