Parameter	Value
Learning Rate	1e-5
LR Schedule	Cosine decay
Warmup Fraction	5%
Min LR	1e-6
Optimizer	Adam (β₁=0.9, β₂=0.95, ε=1e-8)
Weight Decay	0.1
Gradient Clipping	1.0
Global Batch Size	32
Micro Batch Size	1
Epochs	1
Max Sequence Length	200,000
Attention Backend	Flash Attention

Frozen Modules

Vision encoder (visual.visual)
Vision-language aligner (visual.visual.merger)

Gradient Checkpointing

Granularity: full
Method: uniform
Recomputed modules: core_attn

Hardware & Parallelism

Configuration	Value
GPUs	8
Data Parallel Size	4
Tensor Parallel Size	2
Sequence Parallel	Enabled
Distributed Optimizer	Enabled
Optimizer CPU Offload	Enabled

Usage

python

from transformers import AutoModelForVision2Seq, AutoProcessor

model = AutoModelForVision2Seq.from_pretrained(
    "Estwld/Qwen3.5-9B-rednote-200K",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("Estwld/Qwen3.5-9B-rednote-200K")

# Chat with the model
messages = [
    {"role": "user", "content": [
        {"type": "image", "image": "https://example.com/image.jpg"},
        {"type": "text", "text": "Describe this image."}
    ]}
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=1024)
print(processor.decode(outputs[0], skip_special_tokens=True))

GPU Memory Requirements

Inference (bf16): ~18 GB VRAM (single GPU)
Fine-tuning (full): ~80+ GB VRAM per GPU (with full parameters, optimizer states, and activations at 200K context)

For long-context inference at 200K tokens, ensure sufficient KV-cache memory. Using Flash Attention helps reduce the memory footprint significantly.

Limitations

This is a research checkpoint at iteration 309. Further training steps may improve performance.
The model is fine-tuned specifically for rednote (Xiaohongshu)-style tasks as defined in the REDSearcher dataset.

License

This model inherits the Apache 2.0 License from the base Qwen3.5-9B model.