|
--- |
|
base_model: HuggingFaceTB/SmolVLM-Instruct |
|
library_name: peft |
|
license: apache-2.0 |
|
datasets: |
|
- HuggingFaceH4/rlaif-v_formatted |
|
language: |
|
- en |
|
pipeline_tag: image-text-to-text |
|
tags: |
|
- trl |
|
- dpo |
|
--- |
|
|
|
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/SmolVLM.png" width="800" height="auto" alt="Image description"> |
|
|
|
# SmolVLM-Instruct-DPO |
|
|
|
SmolVLM is a compact open multimodal model that accepts arbitrary sequences of image and text inputs to produce text outputs. Designed for efficiency, SmolVLM can answer questions about images, describe visual content, create stories grounded on multiple images, or function as a pure language model without visual inputs. Its lightweight architecture makes it suitable for on-device applications while maintaining strong performance on multimodal tasks. |
|
|
|
## Model Summary |
|
|
|
- **Developed by:** Hugging Face 🤗 |
|
- **Model type:** Multi-modal model (image+text) |
|
- **Language(s) (NLP):** English |
|
- **License:** Apache 2.0 |
|
- **Architecture:** Based on [Idefics3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) (see technical summary) |
|
|
|
## Resources |
|
|
|
- **Demo:** [SmolVLM Demo](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM) |
|
- **Blog:** [More Information Needed] |
|
- **Technical Report:** [More Information Needed] |
|
- **Repository:** [More Information Needed] |
|
|
|
## Uses |
|
|
|
SmolVLM can be used for inference on multimodal (image + text) tasks where the input comprises text queries along with one or more images. Text and images can be interleaved arbitrarily, enabling tasks like image captioning, visual question answering, and storytelling based on visual content. The model does not support image generation. |
|
|
|
### Direct Use |
|
|
|
|
|
### Downstream Use [optional] |
|
|
|
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app --> |
|
|
|
[More Information Needed] |
|
|
|
### Out-of-Scope Use |
|
|
|
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. --> |
|
|
|
[More Information Needed] |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
[More Information Needed] |
|
|
|
### Recommendations |
|
|
|
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. More information needed for further recommendations. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
[More Information Needed] |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
[ HuggingFaceH4/rlaif-v_formatted](HTTP://huggingface.co/HuggingFaceH4/rlaif-v_formatted) |
|
|
|
### Training Procedure |
|
|
|
```bash |
|
accelerate launch --config_file examples/accelerate_configs/multi_gpu.yaml examples/scripts/dpo_vlm.py --dataset_name HuggingFaceH4/rlaif-v_formatted --model_name_or_path HuggingFaceTB/SmolVLM-Instruct --per_device_train_batch_size 8 --gradient_accumulation_steps 32 --dataset_num_proc 32 --output_dir dpo_smolvlm_rlaif-v --bf16 --torch_dtype bfloat16 --use_peft --lora_target_modules=all-linear exit |
|
``` |
|
|
|
### Framework versions |
|
|
|
- PEFT 0.13.2 |
