|
--- |
|
title: Video Vision Transformer on medmnist |
|
emoji: 🧑⚕️ |
|
colorFrom: red |
|
colorTo: green |
|
sdk: gradio |
|
app_file: app.py |
|
pinned: false |
|
license: apache-2.0 |
|
library_name: keras |
|
--- |
|
|
|
## Keras Implementation of Video Vision Transformer on medmnist |
|
|
|
This repo contains the model [to this Keras example on Video Vision Transformer](https://keras.io/examples/vision/vivit/). |
|
|
|
## Background Information |
|
This example implements [ViViT: A Video Vision Transformer](https://arxiv.org/abs/2103.15691) by Arnab et al., a pure Transformer-based model for video classification. The authors propose a novel embedding scheme and a number of Transformer variants to model video clips. |
|
|
|
## Datasets |
|
We use the [MedMNIST v2: A Large-Scale Lightweight Benchmark for 2D and 3D Biomedical Image Classification](https://medmnist.com/) dataset. |
|
|
|
## Training Parameters |
|
``` |
|
# DATA |
|
DATASET_NAME = "organmnist3d" |
|
BATCH_SIZE = 32 |
|
AUTO = tf.data.AUTOTUNE |
|
INPUT_SHAPE = (28, 28, 28, 1) |
|
NUM_CLASSES = 11 |
|
|
|
# OPTIMIZER |
|
LEARNING_RATE = 1e-4 |
|
WEIGHT_DECAY = 1e-5 |
|
|
|
# TRAINING |
|
EPOCHS = 80 |
|
|
|
# TUBELET EMBEDDING |
|
PATCH_SIZE = (8, 8, 8) |
|
NUM_PATCHES = (INPUT_SHAPE[0] // PATCH_SIZE[0]) ** 2 |
|
|
|
# ViViT ARCHITECTURE |
|
LAYER_NORM_EPS = 1e-6 |
|
PROJECTION_DIM = 128 |
|
NUM_HEADS = 8 |
|
NUM_LAYERS = 8 |
|
``` |
