|
--- |
|
license: mit |
|
--- |
|
|
|
# AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities (ArXiv 2024) |
|
|
|
[Guillaume Astruc](https://gastruc.github.io/), [Nicolas Gonthier](https://ngonthier.github.io/), [Clement Mallet](https://www.umr-lastig.fr/clement-mallet/), [Loic Landrieu](https://loiclandrieu.com/) |
|
|
|
<p align="center"> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/662b7fba68ed7bbf40bfb0df/Jh9eOnMePFiL84TOzhe86.png" alt="image/png" width="600" height="300"> |
|
</p> |
|
|
|
## Abstract |
|
|
|
We introduce AnySat: a JEPA-based multimodal Earth Observation model that train simultaneously on diverse datasets with different scales, resolutions (spatial, spectral, temporal), and modality combinations. |
|
|
|
For more details and results, please check out our [github](https://github.com/gastruc/AnySat) and [project page](https://gastruc.github.io/projects/omnisat.html). |
|
|
|
<p align="center"> |
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/662b7fba68ed7bbf40bfb0df/2tc0cFdOF2V0_KgptA-qV.png" alt="image/png" width="400" height="200"> |
|
</p> |
|
|
|
### Inference 🔥 |
|
|
|
In order to load our pretrained models, you can run: |
|
|
|
```python |
|
from models.huggingface import AnySat |
|
|
|
## Code to use pretrained weights |
|
model = AnySat(size="base", pretrained=True) #Exists also "small" and "tiny" |
|
``` |
|
|
|
To get features from an observation of a batch of observations, you need to provide to the model a dictionnary where keys are from the list: |
|
| Dataset | Description | Tensor Size | Channels | Resolution | |
|
|---------------|-----------------------------------|-----------------------------------------|-------------------------------------------|------------| |
|
| aerial | Single date tensor |Bx4xHxW | RGB, NiR | 0.2m | |
|
| aerial-flair | Single date tensor |Bx5xHxW | RGB, NiR, Elevation | 0.2m | |
|
| spot | Single date tensor |Bx3xHxW | RGB | 1m | |
|
| naip | Single date tensor |Bx4xHxW | RGB | 1.25m | |
|
| s2 | Time series tensor |BxTx10xHxW | B2, B3, B4, B5, B6, B7, B8, B8a, B11, B12 | 10m | |
|
| s1-asc | Time series tensor |BxTx2xHxW | VV, VH | 10m | |
|
| s1 | Time series tensor |BxTx3xHxW | VV, VH, Ratio | 10m | |
|
| alos | Time series tensor |BxTx3xHxW | HH, HV, Ratio | 30m | |
|
| l7 | Time series tensor |BxTx6xHxW | B1, B2, B3, B4, B5, B7 | 30m | |
|
| l8 | Time series tensor |BxTx11xHxW | B8, B1, B2, B3, B4, B5, B6, B7, B9, B10, B11 | 10m | |
|
| modis | Time series tensor |BxTx7xHxW | B1, B2, B3, B4, B5, B6, B7 | 250m | |
|
|
|
|
|
Time series keys require a "{key}_dates" (for example "s2_dates") tensor of size BxT that value an integer that represent the day of the year. |
|
Then you have to choose at which scale you want te produce features. Scale argument is in meters and represent the size of the desired patch size. |
|
Outputs will be composed of the concatenation of a class token and a flattened feature map where each feature encodes a scale x scale zone. |
|
Scale should divide the spatial cover of all modalities and be a multiple of 10. |
|
Then, you can run: |
|
|
|
```python |
|
features = AnySat(data, scale=scale) #where scale is the size in meters of patches |
|
``` |
|
|
|
And then you can apply those features to the desired downstream task! |
|
|
|
If you want to get a feature map at the density of a specific modality you can specify: |
|
|
|
```python |
|
features = AnySat(data, scale=scale, keep_subpatch=True, modality_keep=modality) #where modality is the name of the desired modality |
|
``` |
|
|
|
Note that the features will be of size 2*D. If you have several modalities of the same desired resolution, you should pick the most informative one (or modify the code to concatenate also the other modalities) |
|
|
|
|
|
Example of use of AnySat: |
|
 |
|
|
|
|
|
To reproduce results, add new modalities, or do more experiments see the full code on [github]('https://github.com/gastruc/AnySat'). |
|
|
|
### Citing 💫 |
|
|
|
```bibtex |
|
|
|
``` |
|
|
