MERaLiON-Omni-GRPO-10B is the culturally-aligned variant, fine-tuned with dual-reward Group Reward Optimization for enhanced cultural density and multimodal coherence. Accepts image, video, and audio inputs. Multilingual and reasoning-enhanced variants of the model will be released in upcoming updates.

Architecture: Same base as MERaLiON-Omni-MM-10B, fine-tuned with LoRA (r=8, alpha=32) via dual-reward GRPO (Cultural Density λ=0.2 + Multimodal Coherence).

This model is especially suited for tasks requiring cultural nuance in Southeast Asian contexts.

Important Notice

Alpha Release. This is MERaLiON2-Omni (Alpha). The model was trained with SEA images + SEA audios + general videos, so its video understanding for Southeast Asia is zero-shot and research-oriented.
Not Production-Ready. This model has not undergone safety alignment or red-teaming post-training. It may produce hallucinated, biased, or otherwise unreliable outputs. It is not suitable for real-world deployment.
Research Use Only. This release is intended for academic research and evaluation purposes only. Commercial use is not permitted. See LICENSE for full terms.

License

This model is released under a Research-Only License. Commercial use is prohibited without explicit authorization.

Demo

Usage

Installation and inference interface is identical to MERaLiON-Omni-MM-10B:

Requirements

Tested with the following environment:

Dependency	Version
Python	3.10
torch	2.6.0
transformers	4.52.0
flash-attn	2.7.4
torchaudio	2.6.0
pillow	11.1.0
triton	3.2.0
CUDA	12.7

pip install transformers torch pillow torchaudio flash-attn triton

Load model

python

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
from mm_utils_mm import process_mm_info  # bundled in this repository

model_path = "zzlynxSG/MERaLiON-Omni-GRPO-10B"

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
processor.tokenizer.padding_side = "left"
if processor.tokenizer.pad_token is None:
    processor.tokenizer.pad_token = processor.tokenizer.eos_token

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_path, trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    use_safetensors=True,
    device_map="auto"
)

MIN_PIX = 4 * 28 * 28
MAX_PIX = 8192 * 28 * 28
MAX_PIX_VIDEO = 8600 * 28 * 28

Image

python

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=[[{"role": "user", "content": "Follow the text instruction based on the following image: <ImageHere> \n Describe the cultural elements in this image."}]],
    tokenize=False, add_generation_prompt=True
)[0]

mm_input = [{"role": "user", "content": [
    {"type": "image", "image": "your_image.jpg", "min_pixels": MIN_PIX, "max_pixels": MAX_PIX}
]}]
audios, images, videos = process_mm_info(mm_input)
inputs = processor(text=[chat_prompt], audios=audios, images=images, videos=videos).to(model.device).to(model.dtype)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(processor.decode(outputs[0], skip_special_tokens=True))

Audio

python

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=[[{"role": "user", "content": "Follow the text instruction based on the following audio: <SpeechHere> \n Transcribe this audio."}]],
    tokenize=False, add_generation_prompt=True
)[0]

mm_input = [{"role": "user", "content": [
    {"type": "audio", "audio": "your_audio.mp3"}
]}]
audios, images, videos = process_mm_info(mm_input)
inputs = processor(text=[chat_prompt], audios=audios, images=images, videos=videos).to(model.device).to(model.dtype)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(processor.decode(outputs[0], skip_special_tokens=True))

Video + audio

python

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=[[{"role": "user", "content": "Follow the text instruction based on the following video: <VideoHere> \n Summarize this video."}]],
    tokenize=False, add_generation_prompt=True
)[0]

mm_input = [{"role": "user", "content": [
    {"type": "video", "video": "your_video.mp4", "fps": 2, "min_frames": 32, "max_frames": 64,
     "min_pixels": MIN_PIX, "total_pixels": MAX_PIX_VIDEO}
]}]
audios, images, videos = process_mm_info(mm_input, use_audio_in_video=True)
inputs = processor(text=[chat_prompt], audios=audios, images=images, videos=videos).to(model.device).to(model.dtype)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(processor.decode(outputs[0], skip_special_tokens=True))