---
license: agpl-3.0
datasets:
- HeNLP/HeDC4
- HebArabNlpProject/HebNLI
- pig4431/HeQ_v1
language:
- he
pipeline_tag: text-generation
tags:
- RWKV
- Hebrew
---

# TamiLM Hebrew Nano

A Modern Hebrew specialized LLM based on the RWKVv6 Architecture  
Trained only on Modern Hebrew datasets, with a custom vocabulary optimized for Modern Hebrew  

Trained at [Tel Aviv Makers Hackerspace](https://wiki.telavivmakers.org/) 

### Params

Layers `12`  
Depth  `512`  
Head size `64`  
Train ctx_len `512`  
Train tokens `6,841,411,389 (6 Billion)`  
Vocab size `65536`  

### Train Compute

All compute was performed on a single Nvidia P40 card  
Experiments: `62 hours 52 Minutes (2.6 days)`  
Training run: `208 hours 10 Minutes (8.6 days)`  

### How to run

1. Load model into your favourite RWKV v6 runtime
2. Swap RWKV tokenizer with Huggingface tokenizers
3. Load vocab json from this repo