Update README.md
Browse files
README.md
CHANGED
@@ -93,6 +93,44 @@ Empowered by an Innovative Growth Network for dynamic adaptation:
|
|
93 |
- **Initial Capacity:** 250000
|
94 |
- **Input Size:** 2048
|
95 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
96 |
## Seamless Integration with Hugging Face 🤗
|
97 |
|
98 |
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
93 |
- **Initial Capacity:** 250000
|
94 |
- **Input Size:** 2048
|
95 |
|
96 |
+
---
|
97 |
+
|
98 |
+
## Hyperparameters Overview
|
99 |
+
|
100 |
+
Here's a concise overview of the key hyperparameters used for training your model:
|
101 |
+
|
102 |
+
**Training Parameters**
|
103 |
+
- `max_neurons`: 250,000
|
104 |
+
- `epochs`: 50
|
105 |
+
- `clip`: 5
|
106 |
+
- `patience`: 7
|
107 |
+
- `adaptation_rate`: 0.05
|
108 |
+
- `sequence_length`: 200
|
109 |
+
- `max_sequence_length`: 200
|
110 |
+
- `weight_decay`: 0.005
|
111 |
+
- `num_embeddings`: 25,000
|
112 |
+
- `embedding_dim`: 768
|
113 |
+
- `hidden_dim`: 2048
|
114 |
+
- `learning_rate`: 1e-5
|
115 |
+
- `some_intermediate_size`: 3072
|
116 |
+
|
117 |
+
**Additional Parameters**
|
118 |
+
- `input_dimension`: 768
|
119 |
+
- `initial_neuron_count`: 5000
|
120 |
+
- `some_adaptation_rate`: 0.05
|
121 |
+
- `complexity_metric`: None
|
122 |
+
|
123 |
+
**New Parameters**
|
124 |
+
- `growth_improvement_threshold`: 0.01
|
125 |
+
- `pruning_improvement_threshold`: 0.005
|
126 |
+
- `stability_threshold`: 0.02
|
127 |
+
- `max_complexity`: 10,000
|
128 |
+
- `low_stability_threshold`: 0.01
|
129 |
+
- `min_epochs_before_growth`: 5
|
130 |
+
- `desired_improvement_rate`: 0.02
|
131 |
+
|
132 |
+
---
|
133 |
+
|
134 |
## Seamless Integration with Hugging Face 🤗
|
135 |
|
136 |
from transformers import AutoTokenizer, AutoModelForCausalLM
|