---
pipeline_tag: feature-extraction
---
# Kurmanji Tokenizer

This repository contains the Kurmanji Tokenizer trained on a 50 million token text corpus. The tokenizer was specifically developed to support the Kurmanji dialect of Kurdish, ensuring accurate and efficient tokenization for natural language processing tasks in this language.

## Model Details

- **Model Name**: Kurmanji Tokenizer
- **Language**: Kurmanji Kurdish (kmr)
- **Corpus Size**: 50 million tokens
- **Vocabulary Size**: 52,000 tokens
- **Tokenizer Type**: Byte-Pair Encoding (BPE)

## Training Data

The tokenizer was trained on a corpus of 50 million tokens collected from various sources in Kurmanji Kurdish. The data includes a wide range of text types, ensuring the tokenizer can handle diverse linguistic contexts.

### Sources of the Corpus
- Kurdish Kurmanji website crawling

## Usage

You can easily use this tokenizer with the Hugging Face `transformers` library:

```python
from transformers import PreTrainedTokenizerFast

# Load the tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("asosoft/KurmanjiTokenizer-Whisper")

# Example usage
text = "Navê min Ali ye."
tokens = tokenizer.encode(text)
print(tokens)