Views
No views yet
google/gemma-3-4b-it. In Stage 1, the LLM backbone is fully frozen; only the vision encoder and image/video projector are trained. The goal is to align the visual representations with action-recognition vocabulary before full instruction tuning.| Item | Value |
|---|---|
| Base model | google/gemma-3-4b-it |
| Architecture | Gemma3ForConditionalGeneration |
| Training stage | Stage 1 (vision alignment) |
| Trainable components | Vision tower + image projector (embed_vision) |
| Frozen components | LLM backbone (language_model) |
| Property | Value |
|---|---|
| Dataset | Kinetics-400/600/700 (curated subset) |
| Samples | 3,115 video–text pairs |
| Task | Video action recognition |
| Format | Video + "What is the main action in this video?" → short action phrase |
| Parameter | Value |
|---|---|
| Epochs | 1 |
| Total steps | 390 |
| Per-device batch size | 1 |
| Gradient accumulation | 8 (effective batch size = 8) |
| LLM learning rate | 1e-5 |
| Projector learning rate | 2e-5 |
| Vision encoder learning rate | 0.0 (frozen) |
| LR scheduler | Cosine |
| Warmup ratio | 0.03 |
| Optimizer | paged_adamw_8bit |
| Precision | bfloat16 |
| Max sequence length | 4096 |
| Gradient checkpointing | Yes |
| Property | Value |
|---|---|
| Parallelism | DeepSpeed ZeRO Stage 2 |
| Hardware | 1 × GPU |
| Training time | ~2.4 hours (8,589 s) |
| Framework | Transformers 5.5.0 + DeepSpeed |
| Step | Loss |
|---|---|
| 10 | 6.33 |
| 50 | 5.80 |
| 100 | 4.52 |
| 200 | 3.30 |
| 300 | 3.04 |
| 390 (final) | 3.04 |
1from transformers import AutoProcessor, Gemma3ForConditionalGeneration
2import torch
3
4model = Gemma3ForConditionalGeneration.from_pretrained(
5 "bear7011/gemma3-4b-kinetic3K_FT",
6 torch_dtype=torch.bfloat16,
7 device_map="auto",
8)
9processor = AutoProcessor.from_pretrained("bear7011/gemma3-4b-kinetic3K_FT")