README.md

---
library_name: transformers
license: llama3
datasets:
- remyxai/mantis-spacellava
tags:
- remyx
---


![image/png](https://cdn-uploads.huggingface.co/production/uploads/647777304ae93470ffc28913/2MDiSD0Q3Lfe0JtnkdqxB.png)

# Model Card for SpaceMantis

**SpaceMantis** fine-tunes [Mantis-8B-siglip-llama3](TIGER-Lab/Mantis-8B-siglip-llama3) for enhanced spatial reasoning.


## Model Details

Uses LoRA fine-tune on the [spacellava dataset](https://huggingface.co/datasets/remyxai/vqasynth_spacellava) designed with [VQASynth](https://github.com/remyxai/VQASynth/tree/main) to enhance spatial reasoning as in [SpatialVLM](https://spatial-vlm.github.io/).

### Model Description

This model uses data synthesis techniques and publically available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models.
With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create VQA dataset for spatial reasoning.


- **Developed by:** remyx.ai
- **Model type:** MultiModal Model, Vision Language Model, Llama 3

### Model Sources
- **Dataset:** [SpaceLLaVA](https://huggingface.co/datasets/remyxai/vqasynth_spacellava)
- **Repository:** [VQASynth](https://github.com/remyxai/VQASynth/tree/main)
- **Paper:** [SpatialVLM](https://arxiv.org/abs/2401.12168)



## Citation
```
@article{chen2024spatialvlm,
  title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
  author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
  journal = {arXiv preprint arXiv:2401.12168},
  year = {2024},
  url = {https://arxiv.org/abs/2401.12168},
}

@article{jiang2024mantis,
  title={MANTIS: Interleaved Multi-Image Instruction Tuning},
  author={Jiang, Dongfu and He, Xuan and Zeng, Huaye and Wei, Con and Ku, Max and Liu, Qian and Chen, Wenhu},
  journal={arXiv preprint arXiv:2405.01483},
  year={2024}
}
```