You can deploy this model and start earning money today!

luksamuk/
LFM2.5-VL-450M-Hand-Tracking

Name: LFM2.5-VL-450M-Hand-Tracking
Availability: InStock

Views

No views yet

LFM2.5-VL-450M Hand Tracking

Fine-tuned LFM2.5-VL-450M-Extract for hand bounding box detection in webcam images. Produces [{"label": "hand", "bbox": [x1, y1, x2, y2]}] in 0-1 normalized coordinates.

Disclaimer: This is a personal learning project — my first VLM fine-tune. It went through ~10 iterations of trial and error, documented honestly below (including what didn't work). It is not a research paper, not a production system, and not affiliated with Liquid AI.

The evaluation data is not scientifically rigorous. Val set results (100% accuracy) reflect controlled conditions on curated images. Empirical webcam testing shows more discrepant results, documented transparently below. This model was built for a specific use case (real-time hand tracking via webcam on consumer hardware) and optimized for that scenario, not for general-purpose hand detection.

Use it as a starting point for your own experiments, not as a drop-in solution.

Model Details

Base model: LiquidAI/LFM2.5-VL-450M-Extract (450M params, LFM2 architecture)
Fine-tuning method: LoRA (r=16, alpha=16)
Trainable parameters: 5,013,504 of 453,732,352 (1.10%)
Architecture: Lfm2VlForConditionalGeneration (lfm2_vl)
Vision encoder: SigLIP2 NaFlex 86M
LM backbone: LFM2.5-350M-Base
Context length: 4096 tokens
Inference speed: ~200ms per frame on RTX 3050 (290 tok/s decode)

Training Details

Hardware

GPU: NVIDIA GeForce RTX 3050 6GB Laptop
CUDA: 12.6, PyTorch: 2.7.0+cu126
Training framework: Unsloth 2026.6.9 + TRL 0.24.0

Hyperparameters

Parameter	Value
LoRA rank	16
LoRA alpha	16
LoRA dropout	0
Epochs	2
Learning rate	2e-4
LR scheduler	linear
Warmup steps	5
Batch size (train)	2
Batch size (eval)	1
Gradient accumulation	4
Effective batch size	8
Optimizer	adamw_8bit
Weight decay	0.001
Precision	BF16
Max sequence length	4096
Max response length	2048
Seed	3407

Training Results

Metric	Value
Total steps	3,540
Training time	~118 min
Final train loss	~0.55
Final eval loss	~0.42
Seed	3407 (CDA), 42 (split)

LoRA Configuration

finetune_vision_layers: False
finetune_language_layers: True
finetune_attention_modules: True
finetune_mlp_modules: True
use_gradient_checkpointing: "unsloth"

Dataset

Composition

Category	Count	Source
Positive (1 hand)	8,578	HaGRID (palm + fist)
Positive (2 hands, oversampled x2)	2,844	HaGRID, duplicated for balance
Negative (HaGRID patches)	2,000	Cropped regions without hands
Negative (COCO val2017, no person)	2,307	Images without any person annotation
Negative (COCO val2017, person no wrists)	594	Images with person but wrists not visible
Total	16,323	30% negatives

Data Sources

HaGRID (HAnd Gesture Recognition Image Dataset)

554,800 FullHD RGB images, 18 gesture classes
I used 2 gestures: palm (open hand) + fist (closed hand) for maximum pose diversity
Bboxes are normalized [x, y, w, h] in 0-1 range, converted to [x1, y1, x2, y2]
Citation: Kapitanov et al., "HaGRID -- HAnd Gesture Recognition Image Dataset", WACV 2024

COCO val2017 (negative samples)

5,000 validation images from COCO 2017
2,307 images without any "person" annotation (guaranteed no hands)
594 images with person annotation but wrists not visible in keypoints (person on screen, hands occluded)
The person-no-wrists subset addresses a critical failure mode: the model associated "person in frame" with "hand must exist", producing false positives whenever a person was on camera without hands raised
Citation: Lin et al., "Microsoft COCO: Common Objects in Context", ECCV 2014

Dataset iteration The negative sampling went through four iterations, each addressing a failure mode observed in testing:

No negatives — model always detected a hand, even on empty scenes
HaGRID patches only — reduced empty-scene false positives, but still detected hands whenever a person was visible
Added COCO no-person — improved diversity, but person-on-screen still triggered false positives
Added COCO person-no-wrists — resolved the "person = hand" bias

Content order augmentation (positional bias)

During testing, I discovered that the Q8_0 quantization was sensitive to the order of items in the user content. With image before text, the model correctly returned [] when no hands were visible. With text before image, the same model produced false positive hand detections.

This is a form of positional bias. To mitigate this, I applied Counterfactual Data Augmentation (CDA): each training example has a 50% chance of having the image/text order swapped. This teaches the model that content order should not affect the response.

References:

AAAI 2024: "Enhance Modality Robustness in Text-Centric Multimodal Alignment"
ACL 2024: "Unveiling Selection Biases: Order and Token Sensitivity in LLMs"
arXiv 2404.01430: "Position-Aware PEFT for Reducing Positional Bias"

Seed: 3407 (reproducible augmentation)

HaGRID patches (negative samples)

2,000 crops from HaGRID training images, regions outside hand bboxes
Various sizes (256-640px), same domain as positives

Data Format

json

{
  "conversations": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "path/to/image.jpg"},
        {"type": "text", "text": "Detect all hands in the image."}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "[{\"label\":\"hand\",\"bbox\":[0.23,0.45,0.67,0.89]}]"}
      ]
    }
  ]
}

Negative examples use "[]" as the assistant response.

Published Files

Model repo: `luksamuk/LFM2.5-VL-450M-Hand-Tracking`

File	Description
`adapter_config.json`	LoRA adapter config
`adapter_model.safetensors`	LoRA weights (5M params, 24MB)
`tokenizer.json` + config	Tokenizer files
`README.md`	This model card

GGUF repo: `luksamuk/LFM2.5-VL-450M-Hand-Tracking-GGUF`

File	Size	Description
`LFM2.5-VL-450M-Hand-Tracking-BF16.gguf`	~900MB	Full precision (for re-quantization)
`LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf`	379MB	Recommended (best quality)
`LFM2.5-VL-450M-Hand-Tracking-Q4_K_M.gguf`	229MB	Edge / lightweight
`mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf`	190MB	Vision encoder (required)

The BF16 file is the merged model (LoRA + base) in full precision. It allows others to create custom quantizations or use the model without quantization. Q8_0 is recommended for PC/laptop. Q4_K_M is recommended for edge devices (Raspberry Pi).

Usage

llama.cpp (GGUF)

bash

llama-server \
  --model LFM2.5-VL-450M-Hand-Tracking-Q8_0.gguf \
  --mmproj mmproj-LFM2.5-VL-450M-Hand-Tracking-F16.gguf \
  --port 8001 \
  --temp 0.1 \
  --jinja

Python (transformers)

python

from unsloth import FastVisionModel
import torch

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="luksamuk/LFM2.5-VL-450M-Hand-Tracking",
    max_seq_length=4096,
    load_in_4bit=False,
)

Example Application

A demo harness (VLM Webcam Vision) is available at github.com/luksamuk/vlm-webcam-vision. It provides real-time webcam hand tracking with LERP smoothing, label sanitization, and camera controls. Supports any OpenAI-compatible endpoint (llama-swap, Ollama, llama-server) via environment variables.

The llama.cpp server also supports multi-model orchestration natively via model and mmproj flags, enabling hand tracking alongside other models without a proxy.

Output Format

[{"label": "hand", "bbox": [0.23, 0.45, 0.67, 0.89]}]

Coordinates are normalized 0-1. Empty array [] when no hands are visible.

Training Plots

Evaluation Results

Category	Q8_0	Q4_K_M
1 hand (recall)	100%	100%
2 hands (recall)	100%	100%
Negative HaGRID (specificity)	100%	100%
Negative COCO no-person	100%	100%
Negative COCO person-no-wrists	100%	100%
Overall accuracy	100%	100%

250 samples (50 per category). Zero false positives, zero false negatives.

Important: Val set accuracy reflects controlled conditions (curated dataset, ideal images). Empirical webcam testing shows more discrepant results — see below.

Empirical Webcam Testing

Structured protocol: ~10s setup + 5s phases (no hands / 1 hand / 2 hands / alternating) + 3s final.

Model	Frames	`[]` (empty)	1 hand	2 hands	Latency	Flicker
450M Q8_0	356	244 (69%)	100 (28%)	12 (3%)	130ms	20
450M Q4_K_M	305	126 (41%)	123 (40%)	56 (18%)	154ms	29
1.6B Q8_0	90	42 (47%)	48 (53%)	0 (0%)	462ms	2
1.6B Q4_K_M	113	53 (47%)	60 (53%)	0 (0%)	405ms	4

Key findings:

2-hand detection is the main weakness across all models in real webcam conditions
450M Q8_0 has the best specificity (69% empty frames) and lowest latency (130ms)
1.6B models show zero 2-hand detection despite 100% val set accuracy — domain gap
Both 1.6B variants were trained with QLoRA (4-bit base) due to VRAM constraints

1.6B experiment (not published)

A 1.6B Extract variant was trained with the same dataset using QLoRA (4-bit base, r=16). Val set showed 100% accuracy, identical to 450M. However, empirical webcam testing revealed zero 2-hand detection and 3x higher latency (400-460ms vs 130-154ms).

The 1.6B is excluded from publication because:

It does not improve 2-hand detection in real conditions (the main weakness)
Latency is too high for real-time webcam use (target: <200ms)
QLoRA may have degraded quality — full LoRA on more capable hardware is left as future work

The domain gap (HaGRID studio images vs webcam real-world) appears to be the primary bottleneck for 2-hand detection, not model size. This affects both 450M and 1.6B equally.

Limitations

Trained on 2 gestures only (palm + fist); may not generalize to all hand poses
2-hand detection in webcam scenarios may flicker between 1 and 2 detections
Optimized for webcam-resolution images (640x480); may need adjustment for higher resolutions
Single-person training data; multi-person scenes not tested

What was tried and discarded

Horizontal flip augmentation: flipped images + bboxes. Introduced label noise (bbox-only flip without image flip in early versions). Corrected version showed no improvement. Discarded.
Distilling with Qwen3-VL (4B): used Qwen3-VL as teacher to generate hand detection labels on COCO images. IoU 0.805 vs HaGRID ground truth, but introduced patterns that degraded person-no-wrists specificity from 100% to 86%. Discarded.
Q6_K and Q5_K_M quants: inconsistent behavior across quantization levels. Q6_K produced broken JSON. Q5_K_M over-detected. Only Q8_0 and Q4_K_M published.

Attribution

Base model: LiquidAI/LFM2.5-VL-450M-Extract -- LFM Open License v1.0
Training dataset: HaGRID by Kapitanov et al. -- CC BY-SA 4.0
Negative samples: COCO by Lin et al. -- CC BY 4.0
Training framework: Unsloth -- Apache 2.0

References

Kapitanov, A., Kvanchiani, K., Nagaev, A., Kraynov, R., Makhliarchuk, A. "HaGRID -- HAnd Gesture Recognition Image Dataset." WACV 2024. arXiv:2206.08219
Nuzhdin, A., et al. "HaGRIDv2: 1M Images for Static and Dynamic Hand Gesture Recognition." 2024. arXiv:2412.01508
Lin, T.-Y., et al. "Microsoft COCO: Common Objects in Context." ECCV 2014. arXiv:1405.0312
Unsloth LFM2.5-VL fine-tuning guide: https://unsloth.ai/docs/models/tutorials/lfm2.5
LFM2 Technical Report: arXiv:2511.23404

luksamuk/LFM2.5-VL-450M-Hand-Tracking