Our MERT & BEST-RQ

Our implementation on MERT model. Files modified:

mert_fairseq/models/mert/mert_model.py
mert_fairseq/data/mert_dataset.py
run_training_mulNodes_wotorchdist_womodelparsize.sh

Prepare

The MERT training is implemented with fairseq. You need to clone the fairseq repo inside our repo at ./src/fairseq and MERT implementation codes as a fairseq example projcet.

You can do that by following the steps:

mkdir -c ./src/fairseq
cd ./src
git clone https://github.com/pytorch/fairseq

Docker

mirrors.tencent.com/cloudezhou/mert:v3

Start

1-node training

bash run_training_sglNodes.sh 0 dummy MERT_RVQ-VAE_CQT_330M_multinodes_debug1node

1-node training (BEST-RQ)

bash run_training_sglNodes.sh 0 dummy MERT_RVQ-VAE_CQT_95M_bestrq

4-node training

bash run_training_mulNodes_wotorchdist_womodelparsize.sh 0 dummy MERT_RVQ-VAE_CQT_330M_multinodes
bash run_training_mulNodes_wotorchdist_womodelparsize.sh 1 dummy MERT_RVQ-VAE_CQT_330M_multinodes
bash run_training_mulNodes_wotorchdist_womodelparsize.sh 2 dummy MERT_RVQ-VAE_CQT_330M_multinodes
bash run_training_mulNodes_wotorchdist_womodelparsize.sh 3 dummy MERT_RVQ-VAE_CQT_330M_multinodes

4-node training (BEST-RQ)

bash run_training_mulNodes_wotorchdist_womodelparsize.sh $INDEX dummy MERT_RVQ-VAE_CQT_95M_bestrq_multinodes BEST_RQ $CHIEF_IP

4-node training (MusicFM)

bash run_training_mulNodes_wotorchdist_womodelparsize.sh $INDEX dummy MusicFM_95M_multinodes MUSICFM $CHIEF_IP

4-node training (EAT)

bash run_training_eat.sh $INDEX dummy EAT_pretraining_music_multinodes EAT $CHIEF_IP

You could set the parameters in mert_fairseq/config/pretrain/MERT_RVQ-VAE_CQT_330M.yaml

Our latest checkpoints is loaded at data/fairseq_savedir/ckpt_MERT_RVQ-VAE_CQT/MERT_RVQ-VAE_CQT_330M/checkpoint_last.pt