A vision-language model fine-tuned to act as a live game commentator — it looks at a gameplay screenshot and generates a short, natural, spoken-style comment reacting to the moment, similar to how a live streamer or esports caster would react in real time.

Developed by: TK17250
License: apache-2.0
Finetuned from model: unsloth/Llama-3.2-11B-Vision-Instruct
Model type: Vision-Language Model (mllama architecture, 11B parameters)
Language: English

This model was trained 2x faster with Unsloth and Hugging Face's TRL library.

Supported Games

The model is trained specifically on gameplay screenshots from three titles:

Game	Genre
Fortnite	Battle Royale
Rocket League	Vehicular Soccer
VALORANT	Tactical FPS

Commentary style and vocabulary (e.g. weapon names, map callouts, in-game terminology) are tailored to these three games. Performance on other games or genres outside this scope is not guaranteed, since the training data does not cover them.

Training Data

Fine-tuned on a synthetic dataset of ~1,600+ gameplay screenshots paired with commentator-style captions.
Captions were generated using Minimax-M3 as a teacher model, then quality-filtered using a 0–5 relevance/quality score before training.
Each training example consists of: a gameplay screenshot, a system persona prompt ("passionate live game commentator"), and a target caption written in a natural, spoken commentary style.

How to Use

python

from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import torch

processor = AutoProcessor.from_pretrained("TK17250/game-comment-vlm")
model = AutoModelForVision2Seq.from_pretrained(
    "TK17250/game-comment-vlm",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

image = Image.open("your_gameplay_screenshot.jpg").convert("RGB")

system_prompt = (
    "You are a passionate live game commentator. You watch a screenshot from a "
    "video game and give a short, natural spoken-style comment on the gameplay "
    "moment, reacting to how good or bad the play looks."
)
user_prompt = "Comment on this gameplay moment."

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": f"{system_prompt}\n\n{user_prompt}"}
    ]}
]

input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, input_text, add_special_tokens=False, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=128, temperature=0.7, min_p=0.1)
print(processor.decode(output[0], skip_special_tokens=True))

Note: This model's chat template does not support a separate system role alongside image inputs. Combine the persona/system instruction directly into the user message text, as shown above.

Using Unsloth (faster inference)

python

from unsloth import FastVisionModel
from transformers import TextStreamer

model, tokenizer = FastVisionModel.from_pretrained("TK17250/game-comment-vlm", load_in_4bit=False)
FastVisionModel.for_inference(model)

# ... build messages/inputs as above using `tokenizer` instead of `processor`
_ = model.generate(**inputs, streamer=TextStreamer(tokenizer, skip_prompt=True), max_new_tokens=128)

Intended Use

Generating live-style commentary/captions for gameplay clips or screenshots (Fortnite, Rocket League, VALORANT).
Powering auto-commentary features for streaming tools, highlight reels, or gameplay analysis apps.

Limitations

Trained only on the three games listed above; output quality and terminology accuracy drop significantly for other games.
As a fine-tune of an 11B vision-language model on a relatively small synthetic dataset (~1,600 examples), the model may occasionally produce generic or repetitive commentary, or misread fine-grained HUD/UI details (e.g. exact kill counts, health values).
Captions are stylistic/entertainment-oriented and should not be relied on for precise, factual game-state reporting.