Update README.md

README.md

@@ -12,3 +12,31 @@ Full weight fine tuned on two epochs of [SlimOrca](https://huggingface.co/datase
 The base model for this came from a variation on Undi's [Mistral 11B recipe](https://huggingface.co/Undi95/Mistral-11B-v0.1). The `o_proj` and `down_proj` tensors were set to zero in the added layers, making the output exactly identical to Mistral 7B before training.
 
 Benchmarks look good locally but still evaluating actual usefulness.
+
+
+### Reproducing
+
+This [mergekit](https://github.com/cg123/mergekit) config was used to produce the base model:
+```yml
+slices:
+  - sources:
+      - model: mistralai/Mistral-7B-v0.1
+        layer_range: [0, 24]
+  - sources: # add middle layers with residuals scaled to zero
+      - model: mistralai/Mistral-7B-v0.1
+        layer_range: [8, 24]
+        parameters:
+          scale:
+            - filter: o_proj
+              value: 0.0
+            - filter: down_proj
+              value: 0.0
+            - value: 1.0
+  - sources:
+      - model: mistralai/Mistral-7B-v0.1
+        layer_range: [24, 32]
+merge_method: passthrough
+dtype: bfloat16
+```
+
+The axolotl config for fine tuning is available [here](https://huggingface.co/chargoddard/mistral-11b-slimorca/blob/main/axolotl_config.yaml).