File size: 858 Bytes
2afd54f 6e42c6f 2afd54f c3ae1c0 6e42c6f 2afd54f 6e42c6f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
---
library_name: tokenizers
tags: [Danish, Mixed Tokenization, CerebrasGPT]
---
```
_______ ___ .___ ___. ______ .______ .______ __ __
| \ / \ | \/ | / __ \ | _ \ | _ \ | | | |
| .--. | / ^ \ | \ / | | | | | | |_) | | |_) | | |__| |
| | | | / /_\ \ | |\/| | | | | | | / | ___/ | __ |
| '--' | / _____ \ | | | | | `--' | | |\ \----.| | | | | |
|_______/ /__/ \__\ |__| |__| \______/ | _| `._____|| _| |__| |__|
```
### DA-MIXED-CEREBRAS-TOKEN
This tokenizer combines morphological and BPE tokenization strategies for use with the CerebrasGPT architecture, aiming to balance linguistic insights and subword efficiency. |