| base_model: "JackFram/llama-68m" | |
| language: | |
| - en | |
| license: apache-2.0 | |
| library_name: transformers | |
| datasets: | |
| - allenai/c4 | |
| ## Model Description | |
| This model is based of <a href="https://huggingface.co/JackFram/llama-68m">llama-68m</a> and was initialized from this model with tied embeddings and then further pretrained on the uncleaned c4 dataset. This model is obviously a beta based on the name and has not been trained on the full training set which is the first 1 million samples of the uncleaned c4 dataset as mentioned above. | |