.ipynb_checkpoints/README-checkpoint.md

metadata

license: apache-2.0
tags:
  - merge
  - mergekit
  - lazymergekit
  - BeaverAI/mistral-doryV2-12b
  - intervitens/mini-magnum-12b-v1.1
  - grimjim/mistralai-Mistral-Nemo-Base-2407

NemoMix-12B-DellaV1b

NemoMix-12B-DellaV1b is a merge of the following models using mergekit:

• BeaverAI/mistral-doryV2-12b
• intervitens/mini-magnum-12b-v1.1
• grimjim/mistralai-Mistral-Nemo-Base-2407

🧩 Configuration

models:
  - model: BeaverAI/mistral-doryV2-12b
    parameters:
      weight: 0.30
      density: 0.42
  - model: intervitens/mini-magnum-12b-v1.1
    parameters:
      weight: 0.35
      density: 0.66
  - model: grimjim/mistralai-Mistral-Nemo-Base-2407
    parameters:
      weight: 0.35
      density: 0.78
merge_method: della_linear
base_model: grimjim/mistralai-Mistral-Nemo-Base-2407
parameters:
  int8_mask: true
  normalize: true
  epsilon: 0.1  
  lambda: 1.0   
  density: 0.7
dtype: bfloat16