base_model: "JackFram/llama-68m" language:
- en license: apache-2.0 library_name: transformers datasets:
- allenai/c4
Model Description
This model is based of llama-68m and was initialized from this model with tied embeddings and then further pretrained on the uncleaned c4 dataset. This model is obviously a beta based on the name and has not been trained on the full training set which is the first 1 million samples of the uncleaned c4 dataset as mentioned above.