Update README.md

README.md

@@ -9,60 +9,104 @@ license: apache-2.0
 
 # phi-2-DLEC
 
-The DLEC (Distributive Layer Expansion Curve) methodology offers a novel approach to improving neural network models by focusing on the strategic duplication of certain effective layers. 
-Developed with the aim of enhancing model performance, DLEC carefully identifies and amplifies the impact of key layers within the model's architecture. 
+The DLEC (Distributive Layer Expansion Curve) methodology offers a novel approach to improving neural network models by focusing on the strategic duplication of certain effective layers. Developed with the aim of enhancing model performance, DLEC carefully identifies and amplifies the impact of key layers within the model's architecture.
 
-Below is a overview of the method and its implementation, particularly in how it integrates with the Hugging Face Transformers library and utilizes PyTorch and BitsAndBytes for efficient operation.
+## Code overview:
 
-Overview
-Setting Up: First, the script ensures all necessary components are in place, from libraries to the model and dataset.
+Setting Up: 
 
-Database for Activations: A SQLite database is established to track layer activations, providing a clear view into how individual neurons react and which layers are most influential — these are our 'beneficial layers.'
+First, the script ensures all necessary components are in place, from libraries to the model and dataset.
 
-Analyzing and Identifying: By analyzing activation data, the script pinpoints which layers are most valuable to the model's performance.
+Database for Activations: 
 
-Configuring DLEC: A configuration is then created, guiding how the model should incorporate duplicates of these beneficial layers to boost effectiveness without unnecessarily increasing complexity.
+A SQLite database is established to track layer activations, providing a clear view into how 
+individual neurons react and which layers are most influential — these are our 'beneficial layers.'
 
-Reconfiguring and Running the Model: Finally, the model is adjusted according to DLEC's insights, focusing enhancement on the identified layers.
+Analyzing and Identifying: 
 
-Key Features:
-Selective Layer Duplication: DLEC doesn't just add more layers; it doubles down on the ones that really matter. This methodical selection ensures we're making the most of the model's capabilities without wasteful expansion.
+By analyzing activation data, the script pinpoints which layers are most valuable to the model's performance.
 
-Smart Resource Management: By honing in on specific areas for improvement, DLEC aims to make better use of computational and memory resources, promoting more efficient learning without adding undue complexity to the model.
+Configuring DLEC: 
+
+A configuration is then created, guiding how the model should incorporate duplicates of these beneficial layers to boost effectiveness without unnecessarily increasing complexity.
+
+# Key Features: 
+Selective Layer Duplication: 
+
+DLEC doesn't just add more layers; it doubles down on the ones that really matter. This methodical selection ensures we're making the most of the model's capabilities without wasteful expansion.
+
+Smart Resource Management: 
+
+By honing in on specific areas for improvement, DLEC aims to make better use of computational and memory resources, promoting more efficient learning without adding undue complexity to the model.
 
 This approach is about making informed, strategic enhancements to model architecture, prioritizing efficiency and effectiveness in utilizing neural network capabilities.
 
-```ymal
--Possible_Beneficial_layers: #Layers of significance 
-  - 0
-  - 3
-  - 4
-  - 7
-  - 8
-  - 11
-  - 12
-  - 15
-  - 16
-  - 19
-  - 20
-  - 23
-  - 24
-  - 27
-  - 28
-  - 31
-  - 32
+# Information Loss:
+It is common to observe a loss of intelligence when merging models, especially with Passthrough merging, which typically results in a loss of around 3 points per billion parameters duplicated, assuming the merge is done correctly. If the merge is suboptimal, the loss can be much larger, ranging from 3-8 points or more per billion parameters duplicated. However, with DLEC, I was able to increase Phi-2 from 2.78b to 3.25b with a minimal loss of around 0.44 points on average.
+
+DLEC Expanded Model:
+[TheSkullery/phi-2-DLEC](https://huggingface.co/TheSkullery/phi-2-DLEC)
+2.78 -> 3.25, a ~17% increase in size
+```
+Metric -> Value
+Avg. 46.72
+AGIEval 29.64
+GPT4All 69.48
+TruthfulQA 50.29
 ```
-Currently, I am still limited to Mergekit, for this method, which does not support single layer duping, this may have a impact on performance.
 
-# This Method is still in development and I do not expect "Game Changing" or will I oversell this method, it is purely done for fun. Please let me know how the model works for you.
+Original Model:
+[abacaj/phi-2-super](https://huggingface.co/abacaj/phi-2-super))
+```
+Metric -> Value
+Avg. 47.16
+AGIEval 31.95
+GPT4All 70.81
+TruthfulQA 48.39
+```
+
+Loss or Increase:
+Avg. -0.44
+AGIEval -2.31
+GPT4All -1.33
+TruthfulQA +1.90
+
+
+Example of loss:
+[Steelskull/Etheria-55b-v0.1](https://huggingface.co/Steelskull/Etheria-55b-v0.1)
+```
+Metric -> Value
+Avg. 64.69
+AI2 Reasoning Challenge 65.10
+HellaSwag 81.93
+MMLU 73.66
+TruthfulQA 56.16
+Winogrande 76.09
+GSM8k 35.18
+```
+
+[Yi-34B-200K-DARE-megamerge-v8](https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-megamerge-v8)
+```
+Metric -> Value
+Avg. 72.56
+AI2 Reasoning Challenge 67.75
+HellaSwag 86.06
+MMLU 77.03
+TruthfulQA 56.31
+Winogrande 82.79
+GSM8k 65.43
+```
 
-## ⚙️ Evals
-[My Leaderboard:](https://huggingface.co/spaces/Steelskull/YALL-Leaderboard)
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/pS7KFYDheWmFEaGybxr3K.png)
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/CF9_p8AWMFraCnfiMa_no.png)
+Merge Loss (Yi-34B-200K-DARE-megamerge-v8 compared to Etheria-55b-v0.1):
+Avg. -7.87
+AI2 Reasoning Challenge -2.65
+HellaSwag -4.13
+MMLU -3.37
+TruthfulQA +0.15
+Winogrande -6.70
+GSM8k -30.25
 
-As you know there is a usual loss of intelligence with model mergers, especially with Passthrough merging, on the par of 3ish points per billion duped, IF you get the right merge, if not your looking at a much larger loss (anywhere from 3-8 points per billion duped).
-Using DLEC, I was able to increase Phi-2 from 2.78b -> 3.25b with less than or around a single point of loss.
+In the example comparing Etheria-55b-v0.1 and Yi-34B-200K-DARE-megamerge-v8, there is a significant decrease in performance across all metrics, with the average score decreasing by 7.87 points. The most notable is in the GSM8k benchmark, where Yi-34B-200K-DARE-megamerge-v8 outperforms Etheria-55b-v0.1 by 30.25 points.
 
 This method is still in active development, and I am currently tweaking the algorithm to improve the layer selection process, 
 I am also working on a single layer duping script as merge kit does not currently support this and I am merging layers that are unneeded and its degrading performance.