|
|
--- |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- NLP |
|
|
license: mit |
|
|
datasets: |
|
|
- TristanBehrens/jsfakes_garland_2024-100K |
|
|
base_model: None |
|
|
--- |
|
|
|
|
|
# JS Fakes Music xLSTM - An xLSTM model trained on Johann Sebastian Bach Style music |
|
|
|
|
|
Say Hello on [LinkedIn](https://www.linkedin.com/in/dr-tristan-behrens-734967a2/) and [X](https://x.com/DrTBehrens). |
|
|
|
|
|
 |
|
|
|
|
|
This is an xLSTM trained on music. The dataset that has been used is [JS Fakes Garland 100K](https://huggingface.co/datasets/TristanBehrens/jsfakes_garland_2024-100K), which is based on a collection of musical samples extracted from the JS Fake Chorales dataset by Omar Peracha. The samples come in the prototypical Garland notation. |
|
|
|
|
|
The dataset contains 100K samples and comes with a total token count of 80M. |
|
|
|
|
|
The model size is 138.78K trainable parameters. |
|
|
|
|
|
## How to use |
|
|
|
|
|
1. Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/ |
|
|
2. Open and run the notebook `examples/music.ipynb`. |
|
|
3. Enjoy! |
|
|
|
|
|
## Training |
|
|
|
|
|
 |
|
|
|
|
|
Trained with [Helibrunna](https://github.com/AI-Guru/helibrunna) by [Dr. Tristan Behrens](https://de.linkedin.com/dr-tristan-behrens-734967a2). |
|
|
|
|
|
## Configuration |
|
|
|
|
|
``` |
|
|
training: |
|
|
model_name: jsfakes_garland_xlstm |
|
|
batch_size: 16 |
|
|
lr: 0.001 |
|
|
lr_warmup_steps: 312 |
|
|
lr_decay_until_steps: 3125 |
|
|
lr_decay_factor: 0.001 |
|
|
weight_decay: 0.1 |
|
|
amp_precision: bfloat16 |
|
|
weight_precision: float32 |
|
|
enable_mixed_precision: true |
|
|
num_epochs: 1 |
|
|
output_dir: output/jsfakes_garland_xlstm |
|
|
save_every_step: 500 |
|
|
log_every_step: 10 |
|
|
wandb_project: jsfakes_garland_xlstm_2 |
|
|
torch_compile: false |
|
|
model: |
|
|
num_blocks: 4 |
|
|
embedding_dim: 64 |
|
|
mlstm_block: |
|
|
mlstm: |
|
|
num_heads: 4 |
|
|
slstm_block: |
|
|
slstm: |
|
|
num_heads: 4 |
|
|
slstm_at: |
|
|
- 2 |
|
|
context_length: 2048 |
|
|
vocab_size: 115 |
|
|
modelGPT: |
|
|
type: gpt2 |
|
|
num_blocks: 4 |
|
|
embedding_dim: 64 |
|
|
decoder: |
|
|
num_heads: 4 |
|
|
context_length: 2048 |
|
|
dataset: |
|
|
hugging_face_id: TristanBehrens/jsfakes_garland_2024-100K |
|
|
tokenizer: |
|
|
type: whitespace |
|
|
fill_token: '[EOS]' |
|
|
|
|
|
``` |
|
|
|
