armvectores/wikipedia_arlis_tokens_fasttextskipgram_300_5

414M tokens

1. 73M hy wikipedia
2. 341M arlis database

74951 unique words

3-5 ngrams

5 window length

300 embedding dim

skipgram

minimum number of words 150

100 epochs, 0.05 start lr

26 hours on 20 xeon gold cores

How to use

1. Install fastText

pip install fasttext-wheel

2. Import fastText in python

import fasttext
from huggingface_hub import hf_hub_download

model_path = hf_hub_download(local_dir=".",
                             repo_id="armvectores/wikipedia_arlis_tokens_fasttextskipgram_300_5",
                             filename="model.bin")
model = fasttext.load_model(model_path)

3. Examples of usage

word = 'զենքեր'
print(model.get_nearest_neighbors(word))
print(model.get_sentence_vector(word))