⚠️ Experimental research artifact — NOT competent. This checkpoint comes from a series of experiments on how to build a small multimodal model from scratch. It is not a usable vision-language model. Do not use it in production, for automated decisions, or in any safety-critical context.
Kairos-Proj-80k is the second checkpoint in the Kairos experiment progression: Kairos-Initialized with the projector aligned on 80,000 image-caption pairs from Aquiles-ai/LLaVA-CC3M-Pretrain-595K-Embedded (the embedded version of LLaVA-CC3M-Pretrain-595K).LayerNorm → Linear → GELU → Linear), trained on image-caption pairs with an L2 output cap (projector_output_scale = 0.89, ≈ the mean norm of LFM2.5 text embeddings).train_projector.py):| Setting | Value |
|---|---|
| Dataset | Aquiles-ai/LLaVA-CC3M-Pretrain-595K-Embedded (80k samples) |
| Trainable | Projector only (fp32); vision tower + LLM frozen (bf16) |
| Initialization | Zero-init out_proj (start from the text-only prior) |
| Learning rate | 1e-3, cosine, 100 warmup steps |
| Effective batch size | 64 (16 × 4 grad-accum) |
| Epochs | 1 |
| Output cap | projector_output_scale = 0.89 (hard L2 cap) |
| Supervision | Image-caption pairs; empty-think ( thinking response<CAPTION>) |
1import torch
2from transformers import AutoModelForCausalLM, AutoProcessor
3
4model_id = "Aquiles-ai/Kairos-Proj-80k"
5
6model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.bfloat16)
7processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)transformers >= 5.x (developed with 5.14.1) and trust_remote_code=True.