metadata
license: bsd-3-clause
language:
- zh
- en
- id
- ja
- es
TUBELEX FastText Word Embeddings
FastText Word Embeddings trained on the TUBELEX YouTube subtitle corpora. We use the 300-dimensional fastText CBOW model with position weights, 10 negative samples, 10 epochs, character 5-grams (other paramters: default) (Grave et al., 2018).
We provide both '*.bin' files (for fastText) and '*.vec' files that follow the common Word2vec format, and can be used for instance with the gensim package.
What is TUBELEX?
TUBELEX is a YouTube subtitle corpus currently available for Chinese, English, Indonesian, Japanese, and Spanish.
- TODO: paper link
- KenLM n-gram models
- word frequencies and code
Usage
To download and use the fastText models in Python, first install dependencies:
pip install huggingface_hub
pip install fasttext
You can then use e.g. the English (en) model in the following way:
import fasttext
from huggingface_hub import hf_hub_download
model_file = hf_hub_download(repo_id='naist-nlp/tubelex-kenlm', filename='tubelex-en.bin')
model = fasttext.load_model(model_file)
print(model['koala'])