| # Pretraining | |
| (If you are new to the ideas of `mmpt.processors`, see [README](README.md) first.) | |
| We mostly use [howto100M](https://github.com/antoine77340/howto100m) dataset for pretraining (other datasets are coming). So you are less likely to write a new `MetaProcessor`, `VideoProcessor` or `TextProcessor` but only working on a new `Aligner`, a new model and loss. | |
| ### Data Sharding | |
| Pretraining on Howto100M is heavy on IO since we have millions of videos or captions on the hard disk that cannot be fit into the memory. | |
| It is desirable to have an optimized preprocessing step before the actual dataloading. | |
| We support data sharding to pack multiple videos into a shards of training data for both videos and captions. (see [dataset](DATASET.md) for preprocessing). | |
| These shards will be mapped into memory to reduce the frequency of IO access on millions of files. See (processors starting with `Sharded*`). | |
| This will be the default config for a how2 dataset `projects/task/how2.yaml`. | |
| Great thanks to Dmytro Okhonko for sharing the code from MARGE project. | |
| ### Training | |
| Pretraining on Howto100m is expected on one or multiple nodes, where each node has 8 GPUS with 32 GB mem. | |
| launching a pretraing on MFM+MLM can be done, via: | |
| ```python locallaunch.py projects/mfmmlm/how2.yaml``` | |
| ### Pre-training with a Retrieval Model (VideoCLIP) | |
| This projects now support alternatively run a retrieval model and pre-training. | |
| We implement a basic retrieval model that is built on the hidden states of a video and faiss. | |
| You may need to install faiss via `conda install faiss-cpu -c pytorch`. | |
| Right now, the hidden states of a video is computed as the average of 8 clips of their pooled visual/text hidden states. | |
| See `mmpt/tasks/retritask.py` for more details. | |
| The `.yaml` config for running pre-training with a retrieval model can be found at `projects/retri/videoretri.yaml`. | |