Ortho-Vision LoRA — third-angle orthographic views from a 3D block image
A QLoRA adapter for Qwen2.5-VL-3B-Instruct that, given an image of a solid
built from unit cubes, outputs its third-angle orthographic projections
(top / front / right) as three ASCII grids. One narrow, reliable behavior
instilled by fine-tuning — not prompting.
- Base model:
unsloth/Qwen2.5-VL-3B-Instruct-bnb-4bit
- Adapter type: LoRA (r=16, alpha=16), ~41M trainable params (1.08% of the model)
- Training data:
hiyasvyas/orthographic-views-vision — 1,500 train / 150 val, deterministic labels
- Training: 2 epochs, lr 2e-4, batch 2 x grad-accum 4, Unsloth QLoRA on a Colab T4
Behavior Spec (the litmus test)
Given an image of a 3D unit-cube solid, output exactly three ASCII grids
labeled `top:`, `front:`, `right:` — using only `.` and `#`, with no prose
before, between, or after the grids.
A well-prompted base VLM cannot do this reliably; it never even produces the
three-grid format. Fine-tuning fixes the format completely and improves the
spatial content.
Results — base vs. tuned (100 held-out golden images, exact-match)
| metric | BASE | TUNED | delta |
|---|
| `format_ok` | 0.0% | 100.0% | +100.0 |
| `top_ok` | 0.0% | 7.0% | +7.0 |
| `front_ok` | 0.0% | 25.0% | +25.0 |
| `right_ok` | 0.0% | 20.0% | +20.0 |
| `exact` | 0.0% | 3.0% | +3.0 |
Read: the base model scores zero everywhere — it cannot produce the format at
all. The tuned model produces the correct 3-grid format 100% of the time and
gets individual views right a meaningful fraction of the time. Format adherence
(the target behavior) is fully instilled; per-view spatial accuracy is the next
data-iteration target.
How to use
```python
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
from PIL import Image
import torch
base = "unsloth/Qwen2.5-VL-3B-Instruct-bnb-4bit"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "hiyasvyas/ortho-vision-lora")
processor = AutoProcessor.from_pretrained("hiyasvyas/ortho-vision-lora")
prompt = ("Look at this 3D object built from unit cubes. Output its third-angle "
"orthographic views as three ASCII grids labeled 'top:', 'front:', and "
"'right:', using only '.' and '#'. Output nothing except the three grids.")
img = Image.open("your_block_image.png").convert("RGB")
msgs = [{"role": "user", "content": [{"type": "image", "image": img}, {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0], skip_special_tokens=True))
```
Limitations
Per-view spatial accuracy is still low (`exact` = 3%). `top_ok` (7%) lags
`front`/`right`, suggesting the top-down footprint is hardest to read from the
rendered perspective — a data/rendering signal to fix in the next iteration, not
a hyperparameter problem.
Framework versions
