README.md

---
library_name: transformers
license: apache-2.0
language:
- hi
pipeline_tag: token-classification
---

## Model Details

### BertWordPieceTokenizer

- tokenizer for hindi language

#### Usage

```py
from transformers import AutoTokenizer

hi_tokenizer = AutoTokenizer.from_pretrained('krinal/BertWordPieceTokenizer-hi')

hi_str = "आज का सूर्य देखो, कितना प्यारा, कितना शीतल है"

# encode text
encoded_str = hi_tokenizer.encode(hi_str)

# decode text
decoded_str = hi_tokenizer.decode(encoded_str)
```

#### Language

- hi

#### Training

- For training see [Train BertWordPieceTokenizer](https://gist.github.com/kjdeveloper8/57d9e16848cd77df778804c9e2214a78)
  
#### Dataset

- trained on BHAAV (hi sentiment analysis dataset)
- dataset source: [Bhaav](https://github.com/midas-research/bhaav)
- Hindi text corpus (20,304 sentences)

#### Citation

```shell
@article{kumar2019bhaav,
  title={BHAAV-A Text Corpus for Emotion Analysis from Hindi Stories},
  author={Kumar, Yaman and Mahata, Debanjan and Aggarwal, Sagar and Chugh, Anmol and Maheshwari, Rajat and Shah, Rajiv Ratn},
  journal={arXiv preprint arXiv:1910.04073},
  year={2019}
}
```