---
license: cc-by-nc-4.0
language:
- en
library_name: transformers
---
# Whisper Multitask Analyzer

A transformer encoder-decoder model for automatic audio captioning. As opposed to speech-to-text, captioning describes the content and features of audio clips.

- **Model, codebase & card adapted from:** MU-NLPC/whisper-small-audio-captioning
- **Model type:** Whisper encoder-decoder transformer
- **Language(s) (NLP):** en
- **License:** cc-by-4.0
- **Parent Model:** openai/whisper-small

## Usage

The model expects an audio clip (up to 30s) to the encoder as an input and information about caption style as forced prefix to the decoder.
The forced prefix is an integer which is mapped to various tasks. This mapping is defined in the model config and can be retrieved with a function.

The tag mapping of the current model is:

| Task     | ID | Description                                            |
| -------- | -- | ------------------------------------------------------ |
| tags     | 0  | General descriptions, can include genres and features. |
| genre    | 1  | Estimated musical genres.                              |
| mood     | 2  | Estimated emotional feeling.                           |
| movement | 3  | Estimated audio pace and expression.                   |
| theme    | 4  | Estimated audio usage (not very accurate)              |

```

Minimal example:

```python
# Load model
checkpoint = "DionTimmer/whisper-small-multitask-analyzer"
model = WhisperForAudioCaptioning.from_pretrained(checkpoint)
tokenizer = transformers.WhisperTokenizer.from_pretrained(checkpoint, language="en", task="transcribe")
feature_extractor = transformers.WhisperFeatureExtractor.from_pretrained(checkpoint)

# Load and preprocess audio
input_file = "..."
audio, sampling_rate = librosa.load(input_file, sr=feature_extractor.sampling_rate)
features = feature_extractor(audio, sampling_rate=sampling_rate, return_tensors="pt").input_features

# Mappings by ID
print(model.task_mapping) # {0: 'tags', 1: 'genre', 2: 'mood', 3: 'movement', 4: 'theme'}

# Inverted
print(model.named_task_mapping) # {'tags': 0, 'genre': 1, 'mood': 2, 'movement': 3, 'theme': 4}

# Prepare caption style
style_prefix = f"{model.named_task_mapping['tags']}: "
style_prefix_tokens = tokenizer("", text_target=style_prefix, return_tensors="pt", add_special_tokens=False).labels

# Generate caption
model.eval()
outputs = model.generate(
    inputs=features.to(model.device),
    forced_ac_decoder_ids=style_prefix_tokens,
    max_length=100,
)

print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])
```

Example output: 
*0: advertising, beautiful, beauty, bright, cinematic, commercial, corporate, emotional, epic, film, heroic, hopeful, inspiration, inspirational, inspiring, love, love story, movie, orchestra, orchestral, piano, positive, presentation, romantic, sentimental*

WhisperTokenizer must be initialized with `language="en"` and `task="transcribe"`.

The model class `WhisperForAudioCaptioning` can be found in the git repository or here on the HuggingFace Hub in the model repository. The class overrides default Whisper `generate` method to support forcing decoder prefix.


## Licence

The model weights are published under non-commercial license CC BY-NC 4.0 as the model was finetuned on a dataset for non-commercial use.