This repo now contains the full OpenVision2 generative model: the vision encoder
(originally released here) plus the caption text decoder it was jointly trained with.
Together they map an image -> a descriptive caption. The encoder files are unchanged;
only the decoder (and this card) were added.
A concat / prefix-LM autoregressive transformer (not a CoCa cross-attention decoder):
ViT patch tokens are linearly projected and prepended as a bidirectional prefix, and
text is generated causally while attending to all image tokens.
24 layers · width 1024 · 16 heads · mlp 4096 · vocab 32000 · pre-LN · gelu(tanh) · LayerNorm eps 1e-6 · no positional embedding on the text stream.
It consumes the encoder's pre-final-norm patch tokens (open_clip output_tokens=True).
Usage
Needs the patched open_clip providing create_vision_encoder_and_transforms
(https://github.com/UCSC-VLAA/OpenVision), plus torch, safetensors, pillow.
See caption_example.py:
python caption_example.py --image your.jpg
Notes
Encoder and decoder are a matched pair exported from the same training checkpoint.
Captions are LLaVA-style dense descriptions (multi-sentence, detailed).
