TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich.

TerraMind uses a dual-scale transformer-based encoder-decoder architecture, simultaneously processing pixel-level and token-level data. The model was pre-trained on 500B tokens from 9M spatiotemporally aligned multimodal samples from the TerraMesh dataset.

Modality-specific patch embeddings allow direct processing of raw inputs, while modality-specific FSQ-VAEs are used for image tokenization. For sequence-like modalities such as coordinates, an adapted WordPiece tokenizer is employed. During pre-training, TerraMind leverages masked token reconstruction, learning complex cross-modal correlations to generate high-quality latent representations.

We benchmarked TerraMind against other geospatial foundation models using the PANGAEA benchmark. TerraMind consistently achieved state-of-the-art performance, surpassing existing models in various downstream tasks such as land use segmentation, water body mapping, and vegetation assessments. The evaluation highlights its effectiveness in handling diverse Earth Observation scenarios. We present additional experiments in our paper.

TerraMind is fully integrated into the fine-tuning package TerraTorch. This makes it easy to initialize the pre-trained model or fine-tune it via PyTorch Lightning. The weights are automatically downloaded from Hugging Face.

You can fine-tune TerraMind with a config using TerraTorch:

For testing the fine-tuned TerraMind model, run:

We provide config examples and notebooks with step-by-step explanations at https://github.com/IBM/terramind.

Alternatively, you can build the backbone with the following code and use it in your custom pipeline.

The model supports the following raw inputs which you can specify in modalities: S2L2A, S2L1C, S1GRD, S1RTC, DEM, RGB. If your data does not use all bands of a modality, you can specify a subset with bands={'S2L2A': ['BLUE', 'GREEN', 'RED', 'NIR_NARROW', 'SWIR_1', 'SWIR_2']}. You can pass the inputs as in a dict to the model. If a tensor is directly passed, the model assumes it is the first defined modality. TerraMind can also handle missing input modalities.

The model outputs patch embeddings for each input modality. By default, the patch embeddings are averaged over all modalities to reduce the output size. You can specify another merge_method from 'mean', 'max', 'concat', 'dict', and None.

TerraMind introduces a new Thinking-in-Modalities (TiM) approach, where other modalities are predicted as an intermediate steps. Then, the fine-tuned encoder uses both raw inputs and the generated modalities.

Use TiM models in TerraTorch by adding _tim to the model name:

If you use TiM models, we recommend using the pre-training statistics for standardization.

TerraMind can perform any-to-any generation based on varying combinations of inputs.

Build the full TerraMind model (including de-tokenizer steps) from the FULL_MODEL_REGISTRY:

Like the backbone, pass multiple modalities as a dict or a single modality as a tensor to the model which returns the generated output_modalities as a dict of tensors. Note: These generations are not reconstructions but "mental images" representing how the model imagines the modality. You can control generation details via the number of diffusion steps (timesteps) that you can pass to the constructor or the forward function. By passing standardize=True, the pre-training standardization values are automatically applied to the input and output.

We provide an example notebook for generations at https://github.com/IBM/terramind.

Your feedback is invaluable to us. Please share it with us by starting a discussion in this HF repository or submitting an issue to TerraMind on GitHub.

Already working with TerraMind? Submit your use case to the TerraMind Blue-Sky Challenge, a bi-monthly award spotlighting the boldest, most imaginative ways using TerraMind.

If you use TerraMind in your research, please cite the TerraMind paper.