Views
No views yet
16×16 patch grid it keeps every anchor (I-frame) patch and only the motion-salient predicted (P-frame) patches, while a shared 3D rotary position encoding preserves spatio-temporal structure even after large fractions of the grid are dropped.1024, 16 attention heads, GELU MLP at 4× expansion) processes the variable-length token sequence produced by the codec-driven patchifier, followed by a multi-head attention pooling head. The standalone encoder in this repo consumes pixel_values (and optional patch_positions); codec-driven patch selection is applied upstream in the Mage-VL data pipeline.1pip install "transformers>=5.7" torch torchvision pillow
2# optional, for the fastest GPU attention path:
3# pip install flash-attn --no-build-isolationsdpa (default) → flash_attention_2 → eager, so flash-attn is optional and the model runs on CPU or GPU out of the box.1import torch
2from PIL import Image
3from transformers import AutoModel, AutoImageProcessor
4
5model = AutoModel.from_pretrained(
6 "microsoft/Mage-ViT", trust_remote_code=True
7).to("cuda").eval()
8processor = AutoImageProcessor.from_pretrained(
9 "microsoft/Mage-ViT", trust_remote_code=True
10)
11
12image = Image.open("your_image.jpg")
13pixel_values = processor(images=image, return_tensors="pt")["pixel_values"].to("cuda")
14
15with torch.no_grad():
16 out = model(pixel_values) # pixel_values: [B, 3, H, W]
17
18patch_features = out.last_hidden_state # [B, num_patches, 1024]
19pooled_feature = out.pooler_output # [B, 1024]attn_implementation="flash_attention_2" (or "sdpa" / "eager") to from_pretrained to pick the attention backend.[B, C, T, H, W] and pass a patch_positions tensor of shape [B, T * tokens_per_frame, 3] giving the (t, h, w) grid coordinate of every patch:1import torch
2from PIL import Image
3
4PATCH = 16
5
6def build_patch_positions(num_frames, target_frames, grid_h, grid_w, device="cuda"):
7 # temporal index for each frame, spread across the target timeline
8 t = torch.linspace(0, target_frames - 1, num_frames, device=device).long()
9 t = t.repeat_interleave(grid_h * grid_w) # [T * H * W]
10 h = torch.arange(grid_h, device=device).repeat_interleave(grid_w).repeat(num_frames)
11 w = torch.arange(grid_w, device=device).repeat(grid_h).repeat(num_frames)
12 return torch.stack([t, h, w], dim=-1).unsqueeze(0) # [1, T*H*W, 3]
13
14frames = [Image.open(f"frame_{i}.jpg") for i in range(16)] # your sampled frames
15pv = processor(images=frames, return_tensors="pt")["pixel_values"] # [T, C, H, W]
16video = pv.unsqueeze(0).permute(0, 2, 1, 3, 4).to("cuda") # [1, C, T, H, W]
17
18gh, gw = video.shape[-2] // PATCH, video.shape[-1] // PATCH
19patch_positions = build_patch_positions(num_frames=16, target_frames=64,
20 grid_h=gh, grid_w=gw)
21
22with torch.no_grad():
23 out = model(video, patch_positions=patch_positions)| Field | Shape | Description |
|---|---|---|
last_hidden_state | [B, num_patches, 1024] | per-patch features after the final layer norm |
pooler_output | [B, 1024] | global feature from the multi-head attention pooling head |
| Architecture | Codec-ViT (pre-norm ViT, SigLIP-style MLP) |
| Parameters | ~316M |
| Hidden size / MLP | 1024 / 4096 |
| Layers / heads | 24 / 16 |
| Patch size | 16 |
| Position encoding | shared 3D rotary (4:6:6 split over T:H:W) |
| Pooling | learned-probe multi-head attention head |
| Pre-training resolution | images 224–448 (variable), video 256 |
| Weights dtype | bfloat16 |
| License | Apache-2.0 |