File size: 21,032 Bytes
9ef6e9e d2a7b67 f97ab2a d2a7b67 f97ab2a d2a7b67 f97ab2a d2a7b67 f97ab2a 9ef6e9e d2a7b67 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
---
tags:
- generated_from_trainer
datasets:
- toy_graph
metrics:
- accuracy
base_model: toy/model
model-index:
- name: output_toy
results:
- task:
type: text-generation
name: Causal Language Modeling
dataset:
name: toy_graph
type: toy_graph
metrics:
- type: accuracy
value: 0.4525254617525837
name: Accuracy
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# output_toy
This model is a fine-tuned version of [toy/model](https://huggingface.co/toy/model) on the toy_graph dataset.
It achieves the following results on the evaluation set:
- Loss: 1.2691
- Accuracy: 0.4525
- Transition Accuracy: 0.5634
- First Transition Accuracy: 0.88
- Multicode K: 1
- Dead Code Fraction/layer0: 0.9969
- Mse/layer0: 220380.4595
- Input Norm/layer0: 333.7717
- Output Norm/layer0: 12.9360
- Dead Code Fraction/layer1: 0.9535
- Mse/layer1: 132.7843
- Input Norm/layer1: 6.5450
- Output Norm/layer1: 13.1449
- Dead Code Fraction/layer2: 0.9349
- Mse/layer2: 365.9396
- Input Norm/layer2: 6.1370
- Output Norm/layer2: 18.3248
- Dead Code Fraction/layer3: 0.9819
- Mse/layer3: 415.9804
- Input Norm/layer3: 7.4097
- Output Norm/layer3: 18.4665
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 1024
- eval_batch_size: 512
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- training_steps: 20000
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy | Transition Accuracy | First Transition Accuracy | Multicode K | Dead Code Fraction/layer0 | Mse/layer0 | Input Norm/layer0 | Output Norm/layer0 | Dead Code Fraction/layer1 | Mse/layer1 | Input Norm/layer1 | Output Norm/layer1 | Dead Code Fraction/layer2 | Mse/layer2 | Input Norm/layer2 | Output Norm/layer2 | Dead Code Fraction/layer3 | Mse/layer3 | Input Norm/layer3 | Output Norm/layer3 |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:-------------------:|:-------------------------:|:-----------:|:-------------------------:|:----------:|:-----------------:|:------------------:|:-------------------------:|:----------:|:-----------------:|:------------------:|:-------------------------:|:----------:|:-----------------:|:------------------:|:-------------------------:|:----------:|:-----------------:|:------------------:|
| 2.2465 | 0.03 | 500 | 1.8386 | 0.3565 | 0.3555 | 0.31 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.5981 | 0.05 | 1000 | 1.4652 | 0.4204 | 0.5015 | 0.58 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.3928 | 0.07 | 1500 | 1.3541 | 0.4378 | 0.555 | 0.79 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.3405 | 0.1 | 2000 | 1.3264 | 0.4427 | 0.5756 | 0.82 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.3189 | 0.12 | 2500 | 1.3187 | 0.4446 | 0.5576 | 0.86 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.308 | 0.15 | 3000 | 1.3064 | 0.4468 | 0.5573 | 0.82 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.3009 | 0.17 | 3500 | 1.2963 | 0.4493 | 0.5763 | 0.87 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2965 | 0.2 | 4000 | 1.2922 | 0.4494 | 0.5677 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2919 | 0.23 | 4500 | 1.2880 | 0.4499 | 0.5821 | 0.91 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2889 | 0.25 | 5000 | 1.2856 | 0.4501 | 0.56 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2855 | 0.28 | 5500 | 1.2816 | 0.4503 | 0.6016 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2828 | 0.3 | 6000 | 1.2844 | 0.4502 | 0.5734 | 0.87 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2805 | 0.33 | 6500 | 1.2777 | 0.4516 | 0.6084 | 0.95 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2793 | 0.35 | 7000 | 1.2796 | 0.4511 | 0.5681 | 0.93 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2785 | 0.38 | 7500 | 1.2748 | 0.4519 | 0.5919 | 0.95 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2764 | 0.4 | 8000 | 1.2767 | 0.4518 | 0.5760 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2763 | 0.42 | 8500 | 1.2801 | 0.4507 | 0.5827 | 0.94 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2755 | 0.45 | 9000 | 1.2755 | 0.4516 | 0.5765 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2746 | 0.47 | 9500 | 1.2736 | 0.4523 | 0.5865 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2734 | 0.5 | 10000 | 1.2740 | 0.4519 | 0.5779 | 0.91 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2732 | 0.53 | 10500 | 1.2744 | 0.4516 | 0.5879 | 0.89 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2723 | 0.55 | 11000 | 1.2690 | 0.4525 | 0.5811 | 0.89 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2712 | 0.57 | 11500 | 1.2705 | 0.4526 | 0.5779 | 0.93 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2716 | 0.6 | 12000 | 1.2701 | 0.4527 | 0.5760 | 0.89 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2708 | 0.62 | 12500 | 1.2716 | 0.4522 | 0.5485 | 0.95 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2705 | 0.65 | 13000 | 1.2676 | 0.4529 | 0.5734 | 0.93 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2696 | 0.68 | 13500 | 1.2717 | 0.4519 | 0.5994 | 0.91 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2687 | 0.7 | 14000 | 1.2687 | 0.4524 | 0.5756 | 0.9 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2685 | 0.72 | 14500 | 1.2709 | 0.4521 | 0.6127 | 0.89 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2685 | 0.75 | 15000 | 1.2706 | 0.4519 | 0.5873 | 0.91 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2675 | 0.78 | 15500 | 1.2691 | 0.4527 | 0.6365 | 0.96 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2677 | 0.8 | 16000 | 1.2686 | 0.4526 | 0.5589 | 0.93 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2676 | 0.82 | 16500 | 1.2639 | 0.4529 | 0.5940 | 0.89 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2662 | 0.85 | 17000 | 1.2655 | 0.4530 | 0.5955 | 0.94 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2666 | 0.88 | 17500 | 1.2636 | 0.4526 | 0.6013 | 0.96 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2664 | 0.9 | 18000 | 1.2681 | 0.4526 | 0.6034 | 0.96 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.266 | 0.93 | 18500 | 1.2624 | 0.4527 | 0.5839 | 0.88 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2653 | 0.95 | 19000 | 1.2688 | 0.4519 | 0.5837 | 0.92 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2654 | 0.97 | 19500 | 1.2619 | 0.4534 | 0.5973 | 0.92 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1.2649 | 1.0 | 20000 | 1.2647 | 0.4525 | 0.59 | 0.93 | 1 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 |
### Framework versions
- Transformers 4.28.1
- Pytorch 2.0.1+cu117
- Datasets 2.12.0
- Tokenizers 0.13.3
|