Edit model card

GPT-Neo Romanian 780M

This model is a GPT-Neo transformer decoder model designed using EleutherAI's replication of the GPT-3 architecture.

It was trained on a thoroughly cleaned corpus of Romanian text of about 40GB composed of Oscar, Opus, Wikipedia, literature and various other bits and pieces of text, joined together and deduplicated. It was trained for about a month, totaling 1.5M steps on a v3-32 TPU machine.

Authors:

  • Dumitrescu Stefan
  • Mihai Ilie

Evaluation

Evaluation to be added soon, also on https://github.com/dumitrescustefan/Romanian-Transformers

Acknowledgements

Thanks TPU Research Cloud for the TPUv3 machine needed to train this model!

Downloads last month
80
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Space using dumitrescustefan/gpt-neo-romanian-780m 1