versae's picture
Adding checkpointing, wandb, and new mlm script
d988382
|
raw
history blame
1.84 kB
metadata
language: 'no'
license: CC-BY 4.0
tags:
  - spanish
  - roberta
pipeline_tag: fill-mask
widget:
  - text: Lo hizo en un abrir y cerar de <mask>.

BERTIN

BERTIN is a series of BERT-based models for Spanish. This one is a RoBERTa-large model trained from scratch on the Spanish portion of mC4 using Flax, including training scripts.

This is part of the Flax/Jax Community Week, organised by HuggingFace and TPU usage sponsored by Google.

Spanish mC4

The Spanish portion of mC4 containes about 416 million records and 235 billion words.

$ zcat c4/multilingual/c4-es*.tfrecord*.json.gz | wc -l
416057992
$ zcat c4/multilingual/c4-es*.tfrecord-*.json.gz | jq -r '.text | split(" ") | length' | paste -s -d+ - | bc
235303687795

Team members

  • Javier de la Rosa (versae)
  • Manu Romero (mrm8488)
  • María Grandury (mariagrandury)
  • Ari Polakov (aripo99)
  • Pablogps
  • daveni
  • Sri Lakshmi

Useful links