|
--- |
|
license: mit |
|
language: |
|
- fr |
|
- zh |
|
- fa |
|
- ky |
|
- ru |
|
- lt |
|
- uz |
|
- en |
|
- pt |
|
- bg |
|
- th |
|
- pl |
|
- ur |
|
- sw |
|
- tr |
|
- es |
|
- ar |
|
- it |
|
- hi |
|
- de |
|
- el |
|
- nl |
|
- vi |
|
- ja |
|
pipeline_tag: text-classification |
|
tags: |
|
- pytorch |
|
- mt0 |
|
--- |
|
# language identification mt0 |
|
|
|
This model is a fine-tuned version of encoder from [bigscience/mt0-small](https://huggingface.co/bigscience/mt0-small) on the [Language Identification](https://huggingface.co/datasets/papluca/language-identification#additional-information) dataset as well as some private data. |
|
|
|
## Limitations |
|
|
|
Currently, it supports the following 20 languages: |
|
|
|
arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), kyrgyz (ky), uzbek (uz), persian (fa), lithuanian (lt), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh) |
|
|
|
## Inference |
|
|
|
First you will need to have this library installed |
|
|
|
```python |
|
pip install bert-for-sequence classification |
|
``` |
|
|
|
|
|
```python |
|
from bert_clf import EncoderCLF |
|
import torch |
|
|
|
model = EncoderCLF("whitefoxredhell/language_identification") |
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
model = model.to(device) |
|
model = model.eval() |
|
|
|
text = "London is the capital of Great Britain" |
|
|
|
model.predict(text) |
|
# 'en' |
|
|
|
model.predict_proba(text) |
|
# { |
|
# 'fr': 3.022890814463608e-05, |
|
# 'zh': 2.328997834410984e-05, |
|
# 'fa': 5.344639430404641e-05, |
|
# 'ky': 3.5296812711749226e-05, |
|
# 'ru': 2.3277720174519345e-05, |
|
# 'lt': 0.00021786204888485372, |
|
# 'uz': 3.461417873040773e-05, |
|
# 'en': 0.999232292175293, |
|
# 'pt': 1.2590448022820055e-05, |
|
# 'bg': 1.5775613064761274e-05, |
|
# 'th': 9.429674719285686e-06, |
|
# 'pl': 2.4624938305350952e-05, |
|
# 'ur': 3.982995986007154e-05, |
|
# 'sw': 4.8921840061666444e-05, |
|
# 'tr': 2.6844283638638444e-05, |
|
# 'es': 2.325668538105674e-05, |
|
# 'ar': 2.4103366740746424e-05, |
|
# 'it': 1.8611381165101193e-05, |
|
# 'hi': 1.4575023669749498e-05, |
|
# 'de': 2.210299498983659e-05, |
|
# 'el': 1.3880739061278291e-05, |
|
# 'nl': 2.767637124634348e-05, |
|
# 'vi': 1.3878144272894133e-05, |
|
# 'ja': 1.3629408385895658e-05 |
|
# } |
|
``` |
