Hugging Face's logo

whitefoxredhell
/
language_identification

Text Classification
Transformers
PyTorch
mt5
text2text-generation
mt0
Model card Files Community
language_identification
File size: 2,202 Bytes 
4471600
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9e07931
 
 
4471600
1f8179c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b6e2a51
1f8179c
 
 
 
 
e5d293a
1f8179c
 
e5d293a
 
 
1f8179c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
---
license: mit
language:
- fr
- zh
- fa
- ky
- ru
- lt
- uz
- en
- pt
- bg
- th
- pl
- ur
- sw
- tr
- es
- ar
- it
- hi
- de
- el
- nl
- vi
- ja
pipeline_tag: text-classification
tags:
- pytorch
- mt0
---
# language identification mt0

This model is a fine-tuned version of encoder from [bigscience/mt0-small](https://huggingface.co/bigscience/mt0-small) on the [Language Identification](https://huggingface.co/datasets/papluca/language-identification#additional-information) dataset as well as some private data.

## Limitations

Currently, it supports the following 20 languages:

arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), kyrgyz (ky), uzbek (uz), persian (fa), lithuanian (lt), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh)

## Inference

First you will need to have this library installed

```python
pip install bert-for-sequence classification
```


```python
from bert_clf import EncoderCLF
import torch

model = EncoderCLF("whitefoxredhell/language_identification")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model = model.eval()

text = "London is the capital of Great Britain"

model.predict(text)
# 'en'

model.predict_proba(text)
# {
#   'fr': 3.022890814463608e-05,
#   'zh': 2.328997834410984e-05,
#   'fa': 5.344639430404641e-05,
#   'ky': 3.5296812711749226e-05,
#   'ru': 2.3277720174519345e-05,
#   'lt': 0.00021786204888485372,
#   'uz': 3.461417873040773e-05,
#   'en': 0.999232292175293,
#   'pt': 1.2590448022820055e-05,
#   'bg': 1.5775613064761274e-05,
#   'th': 9.429674719285686e-06,
#   'pl': 2.4624938305350952e-05,
#   'ur': 3.982995986007154e-05,
#   'sw': 4.8921840061666444e-05,
#   'tr': 2.6844283638638444e-05,
#   'es': 2.325668538105674e-05,
#   'ar': 2.4103366740746424e-05,
#   'it': 1.8611381165101193e-05,
#   'hi': 1.4575023669749498e-05,
#   'de': 2.210299498983659e-05,
#   'el': 1.3880739061278291e-05,
#   'nl': 2.767637124634348e-05,
#   'vi': 1.3878144272894133e-05,
#   'ja': 1.3629408385895658e-05
# }
```