RoMA

--- license: mit library_name: transformers pipeline_tag: image-feature-extraction --- <p align="center"> <h2 align="center"><strong>RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing</strong></h2> <div align="center"> <h5> <em>Fengxiang Wang¹, Hongzhen Wang^{2 †}, Yulin Wang², Di Wang³, Mingshuo Chen⁴, Haiyan Zhao²,<br/> Yangang Sun², Shuo Wang², Long Lan¹, Wenjing Yang^{1 †}, Jing Zhang^{3 †} </em> <br><br> ¹ National University of Defense Technology, China, ² Tsinghua University, China, <br/> ³ Wuhan University, China, ⁴ Beijing University of Posts and Telecommunications, China </h5> </div> <h5 align="center"> <a href="https://huggingface.co/initiacms/RoMA"> <img src="https://img.shields.io/badge/🤗-Checkpoints-9C276A.svg"></a> <a href="https://arxiv.org/abs/2503.10392"> <img src="https://img.shields.io/badge/Arxiv-2503.10392-b31b1b.svg?logo=arXiv"></a> </h5> # 📚 Contents - [News](#🔥news) - [Abstract](#📄abstract) - [Overview](#🔍overview) - [Evaluation Results](#✅evaluation-results) - [Scaling Behavior](#📈scaling-behavior) - [Pretraining](#🚀pretraining) - [Checkpoints](#🎯checkpoints) - [Citation](#🔗citation) - [Acknowledgement](#🤝acknowledgement) # 🔥News * **[2025.03.13]** The paper is available on [arXiv](http://arxiv.org/abs/2503.10392). # 📄Abstract Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models. However, the quadratic complexity of self-attention poses a significant barrier to scalability, particularly for large models and high-resolution images. While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited to supervised tasks on small, domain-specific datasets. To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining of Mamba-based RS foundation models using large-scale, diverse, unlabeled data. # 🔍Overview <figure> <img src="assets/image-20250311170540530.png"> <figcaption align = "center"><b>Figure 1: Overview of the RoMA Pretraining Pipeline. </b></figcaption> </figure> The input image is first divided into patches, and high-value patches ar...