TerraMind is the first multimodal any-to-any generative foundation model for Earth Observation jointly developed by IBM, ESA, and Forschungszentrum Jülich.
The model is pre-trained using FSQ-VAE tokens as targets. This tokenizer encodes and decodes Sentinel-1 RTC satellite images for the TerraMind model.
The tokenizer uses FSQ with five dimensions and a codebook size of 15'360 tokens.
The decoding process uses diffusion steps for the reconstruction.
The model was pre-trained for up to 100 epochs on nine million S-1 RTC images from the TerraMesh dataset.
The tokenizer is fully integrated into the fine-tuning toolkit
TerraTorch.
You can initialize the pre-trained tokenizer with:
1from terratorch.registry import FULL_MODEL_REGISTRY
2model = FULL_MODEL_REGISTRY.build('terramind_v1_tokenizer_s1rtc', pretrained=True)
Once the model is build, it can be used to encode image and decode tokens.
The number of diffusion steps is defined with timesteps.
Increasing the diffusion steps adds more details to the reconstruction which can also lead to hallucinations.
1# Encode image
2_, _, tokens = model.encode(s1rtc_tensor)
3# Decode tokens
4reconstruction = model.decode_tokens(tokens, verbose=True, timesteps=10)
5# Encode & decode
6reconstruction = model(s1rtc_tensor)
This tokenizer is automatically loaded with TerraMind generation models like
terramind_v1_base_generate, see
here for details.
We provide example code for the tokenizer at
https://github.com/IBM/terramind.
If you have feedback or any questions, please start a discussion in this HF repository or submitting an issue to
TerraMind on GitHub.
If you use TerraMind in your research, please cite our
TerraMind paper.
1@article{jakubik2025terramind,
2 title={TerraMind: Large-Scale Generative Multimodality for Earth Observation},
3 author={Jakubik, Johannes and Yang, Felix and Blumenstiel, Benedikt and Scheurer, Erik and Sedona, Rocco and Maurogiovanni, Stefano and Bosmans, Jente and Dionelis, Nikolaos and Marsocci, Valerio and Kopp, Niklas and others},
4 journal={IEEE/CVF International Conference on Computer Vision (ICCV)},
5 year={2025}
6}
