SimCIS

Rethinking Query-based Transformer for Continual Image Segmentation. (CVPR2025)

By Yuchen Zhu*, Cheng Shi*, Dingyou Wang, Jiajin Tang, Zhengxuan Wei, Yu Wu, Guanbin Li and Sibei Yang†

*Equal contribution; †Corresponding Author

Abstract

Class-incremental/Continual image segmentation (CIS) aims to train an image segmenter in stages, where the set of available categories differs at each stage. To leverage the built-in objectness of query-based transformers, which mitigates catastrophic forgetting of mask proposals, current methods often decouple mask generation from the continual learning process. This study, however, identifies two key issues with decoupled frameworks: loss of plasticity and heavy reliance on input data order. To address these, we conduct an in-depth investigation of the built-in objectness and find that highly aggregated image features provide a shortcut for queries to generate masks through simple feature alignment. Based on this, we propose SimCIS, a simple yet powerful baseline for CIS. Its core idea is to directly select image features for query assignment, ensuring "perfect alignment" to preserve objectness, while simultaneously allowing queries to select new classes to promote plasticity. To further combat catastrophic forgetting of categories, we introduce cross-stage consistency in selection and an innovative "visual query"-based replay mechanism. Experiments demonstrate that SimCIS consistently outperforms state-of-the-art methods across various segmentation tasks, settings, splits, and input data orders. All models and codes will be made publicly available at this https URL.

📣 News

[2025.06.17] 🤗 🤗 🤗 We Release the weights on huggingface.
[2025.06.09] 🤗 We fully release SimCIS, including both code and paper!
[2025.03.03] We are preparing the code and camera ready version of our paper!
[2025.02.27] Our paper is accepted by CVPR2025!

📝 To-do list

Release the code and paper.
Release the weights in the next few days.
More detailed instructions.

PLEASE FOLLOW this Github Repo to use the weights!!!

📖 Cite Us

If you find this repository useful in your research, please consider giving a star ⭐ and a citation

@inproceedings{zhu2025rethinking,
  title={Rethinking Query-based Transformer for Continual Image Segmentation},
  author={Zhu, Yuchen and Shi, Cheng and Wang, Dingyou and Tang, Jiajin and Wei, Zhengxuan and Wu, Yu and Li, Guanbin and Yang, Sibei},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={4595--4606},
  year={2025}
}

👍 Acknowledgement and Related Work

This code is mainly based on Mask2Former. We thank them for their excellent work.
Related work for continual image segmentation: Balconpas, ECLIPSE. We appreciate the contributions of these researchers.