You can deploy this model and start earning money today!

AuraWorxAI/
weather-llm-checkpoints

Name: weather-llm-checkpoints
Availability: InStock

Views

No views yet

Weather LLM — training backups & reproducibility

Decoder-only LLaMA-style causal LM (RMSNorm, RoPE, SwiGLU) trained from scratch on U.S. NOAA GHCN-Daily–derived text mixed with English Wikipedia, with planned supervised fine-tuning (SFT) for instruction-style weather analysis. This Hugging Face repository holds checkpoints, tokenizer, processed data, and full project source so runs can be restored or resumed on a new machine.

What is in this repository

Path	Description
`phase_a/checkpoint-20000/`	Phase A pretraining completed at global step 20 000 (wiki-heavy corpus). Full Trainer checkpoint: `model.safetensors`, optimizer, scheduler, RNG state — use to resume Phase B or as a standalone pretrained base.
`phase_b/checkpoint-XXXXX/`	Phase B (weather-heavy) pretraining backup checkpoint(s). Filenames reflect the step when uploaded; re-upload periodically if training continues. Contains full resume state.
`tokenizer/`	SentencePiece + Hugging Face tokenizer trained on the merged weather-heavy training corpus. Fixed for all pretrain and SFT stages.
`data/`	Processed JSONL: `weather.jsonl`, `wikipedia.jsonl`, Phase A / Phase B train & val merges (`train_phase_a.jsonl`, `val_phase_a.jsonl`, `train.jsonl`, `val.jsonl`), plus `manifest.json` where applicable.
`src/`, `scripts/`, `configs/`, `tests/`	Python package, CLI scripts, YAML configs, and unit tests — match the GitHub repo layout.
`pyproject.toml`, `requirements.txt`, `.gitignore`	Install and dependency metadata.

Note: Final SFT weights (artifacts/checkpoints/sft/run1_stage2/ locally) may be published in this repo or a separate model repo when training completes.

Training pipeline (high level)

Data: NOAA download → preprocess_noaa.py → preprocess_wiki.py → merge_corpus.py (Phase A: ~70% wiki / 30% weather; Phase B: ~85% weather / 15% wiki).
Tokenizer: train_tokenizer.py on the Phase B merge corpus → artifacts/tokenizer/.
Phase A pretrain: configs/pretrain_phase_a.yaml + configs/model_stage2.yaml — general English foundation.
Phase B pretrain: configs/pretrain_stage2.yaml — resume from Phase A final checkpoint; weather-domain specialization. Set max_steps to Phase A end + desired Phase B steps (e.g. 20 000 + 25 000 = 45 000) so the global step counter yields the intended number of Phase B steps.
SFT: build_sft_dataset.py → configs/sft_default.yaml → instruction-tuned checkpoint.

Pretraining uses Hugging Face Trainer with load_best_model_at_end on eval_loss where configured; checkpoints include optimizer state for resume.

Download everything locally

bash

pip install huggingface_hub
hf download AuraWorxAI/weather-llm-checkpoints --local-dir ./weather-llm-checkpoints

Resume pretraining from a checkpoint

Point --resume_from_checkpoint at the directory that contains trainer_state.json (and model.safetensors, optimizer.pt, etc.):

bash

torchrun --standalone --nproc_per_node=2 scripts/train_pretrain.py \
  --pretrain_config configs/pretrain_stage2.yaml \
  --model_config configs/model_stage2.yaml \
  --resume_from_checkpoint ./weather-llm-checkpoints/phase_b/checkpoint-24000

Adjust paths and configs to match Phase A vs Phase B and your local layout.

Inference (after SFT checkpoint is available locally)

SFT uses the same instruction format as weather_llm.inference.generate (### Instruction: / ### Response:). Example:

bash

pip install -e .
python -m weather_llm.inference.generate \
  --model_dir artifacts/checkpoints/sft/run1_stage2 \
  --prompt "Compare summer weather patterns in Arizona vs Washington."

Intended use

Research and education on historical U.S. weather language and from-scratch LM pretraining.
Reproducing or extending the training pipeline using the bundled code and data.

Limitations

Not a replacement for official forecasts, aviation weather, or safety-critical decisions.
Model quality depends on the specific checkpoint, tokenizer, and training stage (pretrain vs SFT).
GHCN-derived text describes past observations; the SFT mix includes refusals for live/forecast-style prompts where configured.

Maintainer

AuraWorxAI — see also the project GitHub repository for development history.