Encoder-alignment checkpoint used to initialize the default ESFM small training lineage.
A new variable-aware ESFM encoder was distilled from the pretrained Aurora-small encoder on ERA5.
This is primarily a pre-training artifact, not the recommended endpoint for forecasting.
Checkpoint selection: Use this checkpoint to reproduce or continue the documented ESFM initialization pipeline.
For direct deterministic forecasting, use ESFM_s_wm (heterogeneous/missing inputs) or ESFM_s_nm (complete dense inputs).
Model Details
Developed by: The ESFM research team, with the full contributor and author lists linked below.
The paper is currently available as an arXiv preprint.
Uses
Direct Use
Use this checkpoint to reproduce or continue the documented ESFM initialization pipeline.
For direct deterministic forecasting, use ESFM_s_wm (heterogeneous/missing inputs) or ESFM_s_nm (complete dense inputs).
Downstream Use
Initialization of ESFM's ERA5, MODIS, and station-specialized lineages.
Out-of-Scope Use
It should not be presented as an independently evaluated production forecast model or as a drop-in transformers model.
Bias, Risks, and Limitations
This checkpoint inherits biases and coverage limitations from ERA5 and the Aurora teacher.
Knowledge distillation aligns representations but does not establish forecast skill by itself.
All ESFM checkpoints are research artifacts.
Users should validate forecasts for their variables, regions, seasons, lead times, missingness pattern, and decision context.
Do not use the model as the sole basis for safety-critical decisions.
How to Get Started
The checkpoint is not packaged as a Hugging Face Transformers from_pretrained model.
Construct the ESFM architecture with the matching repository config, then load the state dictionary.
The released notebook contains the complete download, model-construction, normalization, and inference workflow.
bash
1git clone https://github.com/swiss-ai/ESFM.git
2cd ESFM
3# Open notebooks/inference_ESFMs_on_ERA5.ipynb
Use the matching configs/config_ESFM_s_enc_KD_nm.yaml and train_encoder_KD.py or select this repository in the released inference notebook for checkpoint inspection.
Training Details
Training Data
ERA5 from 1979 through 2020, using the Aurora encoder as teacher and the randomly initialized ESFM encoder as student.
Dataset preprocessing and the exact variable registry are documented in the ESFM repository and preprint.
Training Procedure
The manuscript reports 40,000 distillation steps with an L1 loss between teacher and student encoder logits, a 1,000-step linear warmup to a learning rate of 5e-4, and cosine decay to 4e-4, using 16 GPUs.
This checkpoint is then used to initialize the masked and unmasked training schemes.
Training objective: Initialization alignment, as specified above
Nominal architecture: ESFM small, approximately 115M parameters
Software environment: PyTorch/Lightning in the released NVIDIA PhysicsNeMo 25.03 container; lightning==2.5.1 is pinned in the Dockerfile
Training regime: Lightning precision="32-true" with FP32 parameters and optimizer state; selected model forward operations use CUDA BF16 autocasting through torch.autocast(dtype=torch.bfloat16).
Evaluation
The manuscript evaluates downstream models initialized by knowledge distillation against random and CMIP6-based initialization.
It reports the distillation-based lineage as the strongest of the compared initializations.
This intermediate checkpoint is not given a standalone result table.
The manuscript uses held-out temporal data and reports task-appropriate metrics: latitude-weighted MAE and Pearson correlation for gridded deterministic forecasts, relative MAE for MODIS comparisons, station metrics for station models, and CRPS for ensembles.
Detailed values are intentionally not copied into this card.
Technical Specifications
ESFM retains Aurora's 3D Swin-UNet backbone and adds variable-specific tokenization, axial attention across variables, perceiver aggregation across variables and pressure levels, learnable NaN tokens for missing patches, resolution-specific tokenizers where configured, and a decoder queried at target pressure levels.
The small configuration uses a 256-dimensional embedding and approximately 115M parameters.
Environmental Impact
Hardware type: NVIDIA GH200 systems with four GPUs per node. This experiment was run on four nodes, totaling 16 GPUs.
Total training time: 200 hours
Compute location: Training used CSCS Alps infrastructure.
Citation
bibtex
1@misc{ozdemir2026esfm,
2 title={Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecasting},
3 author={Firat Ozdemir and Yun Cheng and Salman Mohebi and Fanny Lehmann and Simon Adamov and Zhenyi Zhang and Leonardo Trentini and Dana Grund and Oliver Fuhrer and Torsten Hoefler and Siddhartha Mishra and Sebastian Schemm and Benedikt Soja and Mathieu Salzmann},
4 year={2026},
5 eprint={2605.00850},
6 archivePrefix={arXiv},
7 primaryClass={physics.ao-ph},
8 url={https://arxiv.org/abs/2605.00850}
9}
