Hugging Face's logo

tahrirchi
/
tahrirchi-bert-base

Fill-Mask
Transformers
PyTorch
Safetensors
Uzbek
bert
Inference Endpoints
Model card Files Community
1
tahrirchi-bert-base
File size: 6,354 Bytes 
fea59e3
 
 
 
 
 
 
6012d68
 
4e4d68a
 
 
6012d68
 
 
fea59e3
 
6012d68
fea59e3
 
 
 
 
 
 
 
 
 
4e4d68a
fea59e3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4e4d68a
fea59e3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6012d68
fea59e3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2d1a421
fea59e3
 
 
 
 
 
 
 
96aebe9
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
 
---
license: apache-2.0
language:
- uz
library_name: transformers
pipeline_tag: fill-mask
datasets:
- tahrirchi/uz-crawl
- tahrirchi/uz-books
tags:
- bert
widget:
- text: >-
    Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning <mask>,
    mutafakkiri va davlat arbobi bo‘lgan.
---

# TahrirchiBERT base model

The TahrirchiBERT-base is an encoder-only Transformer text model with 110 million parameters. 
It is pretrained model on Uzbek language (latin script) using a masked language modeling (MLM) objective. This model is case-sensitive: it does make a difference between uzbek and Uzbek.

For full details of this model please read our paper (coming soon!) and [release blog post](https://tahrirchi.uz/grammatika-tekshiruvi).

## Model variations

This model is part of the family of **TahrirchiBERT models** trained with different number of parameters that will continuously expanded in the future.

| Model | Number of parameters | Language | Script
|------------------------|--------------------------------|-------|-------|
| [`tahrirchi-bert-small`](https://huggingface.co/tahrirchi/tahrirchi-bert-small) | 67M   | Uzbek | Latin
| [`tahrirchi-bert-base`](https://huggingface.co/tahrirchi/tahrirchi-bert-base) | 110M   | Uzbek | Latin

## Intended uses & limitations

This model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. 

### How to use

You can use this model directly with a pipeline for masked language modeling:

```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='tahrirchi/tahrirchi-bert-base')
>>> unmasker("Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning <mask>, mutafakkiri va davlat arbobi bo‘lgan.")

[{'score': 0.4616584777832031,
  'token': 10879,
  'token_str': ' shoiri',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning shoiri, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.19899587333202362,
  'token': 10013,
  'token_str': ' olimi',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning olimi, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.055418431758880615,
  'token': 12224,
  'token_str': ' asoschisi',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning asoschisi, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.037673842161893845,
  'token': 24597,
  'token_str': ' faylasufi',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning faylasufi, mutafakkiri va davlat arbobi bo‘lgan.'},
 {'score': 0.029616089537739754,
  'token': 9543,
  'token_str': ' farzandi',
  'sequence': 'Alisher Navoiy – ulug‘ o‘zbek va boshqa turkiy xalqlarning farzandi, mutafakkiri va davlat arbobi bo‘lgan.'}]


>>> unmasker("Egiluvchan boʻgʻinlari va <mask>, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.")

[{'score': 0.1740381121635437,
  'token': 12571,
  'token_str': ' oyoqlari',
  'sequence': 'Egiluvchan bo‘g‘inlari va oyoqlari, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.05455964431166649,
  'token': 2073,
  'token_str': ' uzun',
  'sequence': 'Egiluvchan bo‘g‘inlari va uzun, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.050441522151231766,
  'token': 19725,
  'token_str': ' barmoqlari',
  'sequence': 'Egiluvchan bo‘g‘inlari va barmoqlari, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.04490342736244202,
  'token': 10424,
  'token_str': ' tanasi',
  'sequence': 'Egiluvchan bo‘g‘inlari va tanasi, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'},
 {'score': 0.03777358680963516,
  'token': 27116,
  'token_str': ' bukilgan',
  'sequence': 'Egiluvchan bo‘g‘inlari va bukilgan, yarim bukilgan tirnoqlari tik qiyaliklar hamda daraxtlarga oson chiqish imkonini beradi.'}]
```

## Training data

TahrirchiBERT is pretrained using a standard Masked Language Modeling (MLM) objective: the model is given a sequence of text with some tokens hidden, and it has to predict these masked tokens. TahrirchiBERT is trained on the [Uzbek Crawl](https://huggingface.co/datasets/tahrirchi/uz-crawl) and all latin portion of [Uzbek Books](https://huggingface.co/datasets/tahrirchi/uz-books), which contains roughly 4000 preprocessd books, 1.2 million curated text documents scraped from the internet and Telegram blogs (equivalent to 5 billion tokens).

## Training procedure

### Preprocessing

The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 30,528 to make fully use of rare words. The inputs of the model take pieces of 512 contiguous tokens that may span over documents. Also, we added number of regular expressions to avoid misrepresentation of different symbols that is used mostly incorrectly in practise.

### Pretraining

The model was trained for one million steps with a batch size of 512. The sequence length was limited to 512 tokens during all pre-training stage. The optimizer used is Adam with a learning rate of 5e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.98\\), a weight decay of 1e-5, learning rate warmup to the full LR for 6% of the training duration with linearly decay to 0.02x the full LR by the end of the training duration.

## Citation

Please cite this model using the following format:

```
@online{Mamasaidov2023TahrirchiBERT,
    author    = {Mukhammadsaid Mamasaidov and Abror Shopulatov},
    title     = {TahrirchiBERT base},
    year      = {2023},
    url       = {https://huggingface.co/tahrirchi/tahrirchi-bert-base},
    note      = {Accessed: 2023-10-27}, % change this date
    urldate   = {2023-10-27} % change this date
}
```

## Gratitude

We are thankfull for these awesome organizations and people for help to make it happen:

 - [MosaicML team](https://mosaicml.com/): for their script for efficiently training BERT models
 - [Ilya Gusev](https://github.com/IlyaGusev/): for advise throughout the process  
 - [David Dale](https://daviddale.ru): for advise throughout the process