QLoRA fine-tune of LiquidAI/LFM2.5-VL-1.6B for autonomous satellite image triage. Classifies 512×512 RGB frames captured at LEO as HIGH (strategic anomaly, downlink immediately), MEDIUM (human infrastructure, store for bulk transfer), or LOW (featureless terrain, discard).
Developed for ORION, an autonomous LEO satellite triage system running on a Raspberry Pi 5 via NASA F-Prime. The Q4_K_M GGUF quantization of this adapter is deployed on-board and runs inference at 51-82 s/frame (mean ~69s across 1,443 frames from 3 end-to-end runs) entirely on CPU.
Uses
Intended use: on-board orbital triage on a satellite OBC. The model receives a 512×512 RGB satellite tile (optionally with GPS coordinates in the prompt) and returns a JSON object with a triage verdict and visual reasoning.
Triage prompt (ChatML format, used identically for training, evaluation, and on-board inference):
<|im_start|>user
<image>
You are an autonomous orbital triage assistant. Analyze this
high-resolution RGB satellite image captured at Longitude: {lon},
Latitude: {lat}.
Strictly use one of these categories based on visual morphology:
- HIGH: Extreme-scale strategic anomalies, dense geometric cargo/vessel
infrastructure, massive cooling towers, sprawling runways, or distinct
geological/artificial chokepoints.
- MEDIUM: Standard human civilization. Ordinary urban grids, low-density
suburban sprawl, regular checkerboard agriculture, or localized
infrastructure.
- LOW: Complete absence of human infrastructure. Featureless deep oceans,
unbroken canopy, barren deserts, or purely natural geological formations.
You MUST output your response as a valid JSON object. To ensure accurate
visual reasoning, you must output the "reason" key FIRST, followed by
the "category" key.<|im_end|>
<|im_start|>assistant
The model responds with {"reason": "...", "category": "HIGH|MEDIUM|LOW"}. Reason-first ordering forces the model to commit to visual evidence before selecting a label. During training, half the samples omit the Longitude/Latitude line (coordinate dropout augmentation).
Out of scope: multispectral analysis, change detection, object detection with bounding boxes, real-time video, or any use case requiring sub-60-second latency without CUDA acceleration.
Dataset
The adapter was trained on the ORION dataset, 360 curated target locations organized by triage priority and visual morphology, fetched as 512×512 RGB tiles from SimSat's Mapbox API.
Hard negatives are included in LOW: coastlines and geological formations that mimic artificial structures (calderas, salt flat fractals, river deltas).
Split (deterministic, random.seed(42)):
Split
Records
Notes
Train
480
240 targets × 2 (coordinate dropout augmentation)
Val
60
Always with coordinates; used for eval_loss + best-checkpoint selection
Test
60
Always with coordinates; held out for ablation and evaluation
Coordinate dropout augmentation: each training target produces two records, one with GPS coordinates in the prompt and one without. This teaches the model to classify from pixels alone when GPS is unavailable or spoofed.
Training Procedure
Base model
LiquidAI/LFM2.5-VL-1.6B loaded in 4-bit NF4 quantization via bitsandbytes.
LoRA configuration
Parameter
Value
Rank (r)
16
Alpha
32
Target modules
q_proj, k_proj, v_proj, o_proj
Dropout
0.05
Bias
none
Task type
CAUSAL_LM
Training arguments
Parameter
Value
Learning rate
2e-4
Epochs
3
Per-device batch size
1
Gradient accumulation steps
16 (effective batch 16)
Optimizer
paged_adamw_8bit
Precision
FP16
Gradient checkpointing
enabled
Best checkpoint selection
eval_loss (lower is better)
Hardware
Component
Spec
GPU
NVIDIA GeForce RTX 4070 Ti, 12 GB VRAM
CUDA
12.2
Driver
535.x
OS
Linux
Training time
Metric
Value
Time per epoch
~830s
Total training time
~2492s
Model Artifacts
Artifact
File
Size
Notes
LoRA adapter (this repo)
orion_lora_weights/
~50 MB
r=16, 4 attention projection modules
Merged FP16 checkpoint
orion_merged/
~3.2 GB
merge_and_unload() output
FP16 GGUF
orion-f16.gguf
~3.2 GB
Intermediate conversion step
Q4_K_M GGUF
orion-q4_k_m.gguf
~730 MB
Deployed to Pi 5 (8 GB RAM)
Vision projector
orion-mmproj-f16.gguf
~814 MB
FP16, deployed alongside Q4 model
Measured on-device: Total ORION process RSS during inference on the Pi 5 is ~1,753 MB (model + vision encoder + KV cache + F-Prime flight software + buffer pool).
The Q4_K_M GGUF + mmproj pair is the deployed artifact. Pre-built files are available on Hugging Face.
Evaluation
Both studies use the same four conditions run against the same 60-sample held-out test set. The ablation (ablation.py) tests the unmodified base model; the evaluation (evaluate.py) tests the fine-tuned adapter. Running both against identical inputs isolates the exact lift from fine-tuning.
Refer to Training Pipeline for more details on how to read this result.
Condition
Input
Purpose
A: Full system
Real image + real GPS coords
Nominal operating condition
B: Vision only
Real image + no coords
GPS-denied or noisy environment
C: Blind LLM
Gaussian noise image + real coords
Coordinates-only baseline (tests GPS reliance)
D: Sensor conflict
Real image + spoofed coords
Adversarial GPS; tests which modality the model trusts
Ablation study: base model (ablation.py)
Condition
Overall accuracy
Notes
A: Vision + GPS coords
58.3%
B: Vision only (no coords)
60.0%
Slightly better: coords can mislead base model
C: Blind LLM (Gaussian noise + coords)
35.0%
Predicts LOW for everything; GPS alone is unreliable
The same 4-condition protocol run against the Q4_K_M GGUF deployed on-device via llama.cpp's HTTP server. This measures accuracy degradation from quantization using the exact same test set.
Condition
Overall accuracy
Notes
A: Vision + GPS coords
55.0%
−3.3 pp from FP16 fine-tuned
B: Vision only (no coords)
63.3%
−1.7 pp from FP16 fine-tuned
C: Blind LLM (Gaussian noise + coords)
28.3%
Predicts HIGH for most noise inputs
D: Sensor conflict
-
Trusts incorrect coords 15.0% of the time (down from 16.7%)
Sensor conflict (Condition D): coordinate-trust failure drops from 20.0% (base) to 16.7% (fine-tuned FP16) to 15.0% (Q4_K_M GGUF). Quantization does not degrade GPS robustness.
Quantization impact on operational conditions (A and B): accuracy loss from Q4_K_M quantization is modest (−3.3 pp and −1.7 pp respectively), confirming that the deployed GGUF retains most of the fine-tuned model's capability. The large drop on Condition C (noise inputs) is not operationally relevant since the model never receives noise images in deployment.
Discussion
Fine-tuning produces measurable improvements on Conditions B, C, and D, but Condition A (the nominal operating condition with both image and GPS) shows no gain on this 360-target dataset. The most likely explanation is the breadth of the HIGH category: mega-ports, mega-airports, energy infrastructure, open-pit mines, and military facilities are all grouped into a single label. The model can learn to output the correct JSON format quickly (training loss drops to 0.18 in ~41 minutes), but 240 training images spread across five visually heterogeneous HIGH sub-types is not enough for the visual encoder to learn a reliable decision boundary.
This is a prototype demonstrating that on-board VLM inference on a Pi 5 is technically viable. The approach will improve significantly with:
Narrower taxonomy: splitting HIGH into mission-specific sub-classes (e.g., ports only, or energy infrastructure only) and training a specialist adapter
Larger corpus: 240 training images is a minimal dataset for a 3-class VLM task; 1,000-5,000 images per class is a more realistic target for robust generalization
Higher-resolution tiles: 512×512 Mapbox tiles lose fine-grained texture that distinguishes, e.g., a cargo terminal from a large parking lot at altitude
Deployment
The adapter is converted to Q4_K_M GGUF via llama-quantize and runs on the Pi 5 via llama.cpp's multimodal (mtmd) API:
Vision encoding (mtmd): ~10-15 s
Token generation (200 max): ~40-55 s
Total per frame: ~51-82 s (CPU only, Cortex-A76, mean ~69 s, 1,443 frames from 3 end-to-end runs)
Trained on Mapbox RGB tiles only; hence, no multispectral, SAR, or thermal data.
512×512 pixel resolution matches the Pi 5 inference pipeline; different resolutions require re-cropping.
Three-class taxonomy (HIGH / MEDIUM / LOW) is fixed at training time. Mission-specific priorities require fine-tuning on a new labeled dataset.
Inference at 51-82 s/frame (mean ~69s across 1,443 frames from 3 end-to-end runs) sets a hard floor on capture interval: the auto-capture timer is 85s to avoid saturating the VLM queue, limiting throughput to ~24 frames per 35-min eclipse. Burst imaging, real-time video, or sub-minute revisit rates are not feasible without faster hardware (GPU/NPU) or a smaller model.
Coordinate dropout improves GPS robustness but does not eliminate coord-biased errors on hard edge cases.
Blank/missing tile hallucination: Mapbox returns blank white tiles at extreme latitudes (|lat| > 75°) where no satellite imagery exists. The model hallucinates strategic significance onto these featureless images (3 out of 8 HIGH classifications across 1,443 frames were blank tiles). These blank tiles are visually distinct from the ocean and ice sheet tiles in the training set. Mitigation: add blank/white tile detection before inference, or include polar blank tiles as explicit LOW training examples.
Natural feature false positives: Coastlines, cloud cover, and geological formations (e.g., river deltas, glacial terrain) can be misclassified as HIGH due to visual similarity to trained HIGH morphologies (e.g., coastlines as "artificial formations," clouds as "volcanic eruptions"). The hard-negative training set mitigates some of this, but edge cases remain.
Training data was generated at 500 km simulated altitude; Pi 5 runs used the SimSat TLE orbit at ~802 km (~0.7 Mapbox zoom levels difference). The model generalized across this mismatch without degradation, but accuracy may differ at significantly different altitudes.
