You can deploy this model and start earning money today!

Fr0zencr4nE/
UniCoT-7B-MoT

Name: UniCoT-7B-MoT
Rating: 2.5 (15 reviews)

Views

No views yet

Uni-CoT

Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision

This repository contains the UniCoT-7B-MoT model, a unified Chain-of-Thought (CoT) framework for multimodal reasoning across text and vision. It was introduced in the paper Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision.

Project Page: https://sais-fuxi.github.io/projects/uni-cot/ Code: https://github.com/Fr0zenCrane/UniCoT

Overview

Chain-of-Thought (CoT) reasoning has been widely adopted to enhance Large Language Models (LLMs) by decomposing complex tasks into simpler, sequential subtasks. Uni-CoT extends these principles to vision-language reasoning, enabling coherent and grounded multimodal reasoning within a single unified model. The key idea is to leverage a model capable of both image understanding and generation to reason over visual content and model evolving visual states.

The Uni-CoT framework adopts a novel two-level hierarchical reasoning architecture:

Macro-Level CoT: Decomposes a complex task into simpler subtasks and synthesizes their outcomes to derive the final answer. This includes strategies like Sequential, Parallel, and Progressive Refinement Decomposition.
Micro-Level CoT: Focuses on executing individual subtasks, incorporating a Self-Check (Self-Reflection) Mechanism to ensure stable and high-quality results.

This design significantly reduces computational overhead, allowing Uni-CoT to perform scalable and coherent multi-modal reasoning. It aims to solve complex multimodal tasks, including:

🎨 Reliable image generation and editing
🔍 Visual and physical reasoning
🧩 Visual planning
📖 Multimodal story understanding

🧠 Reasoning Pipeline

Quickstart

Installation

The environment setup of Uni-CoT is consistent with its base model, Bagel.

bash

git clone https://github.com/Fr0zenCrane/UniCoT.git
cd UniCoT
conda create -n unicot python=3.10 -y
conda activate unicot
pip install -r requirements.txt
pip install flash_attn==2.5.8 --no-build-isolation

Model Download

You may directly download the huggingface checkpoint or use the following script:

python

from huggingface_hub import snapshot_download

save_dir = "models/UniCoT-7B-MoT"
repo_id = "Fr0zencr4nE/UniCoT-7B-MoT"
cache_dir = save_dir + "/cache"

snapshot_download(cache_dir=cache_dir,
  local_dir=save_dir,
  repo_id=repo_id,
  local_dir_use_symlinks=False,
  resume_download=True,
  allow_patterns=["*.json", "*.safetensors", "*.bin", "*.py", "*.md", "*.txt"],
)

Self-check Reasoning

To perform evaluation or general inference using UniCoT-7B-MoT, you need at least one GPU with 40GB or more VRAM.

Evaluation on WISE benchmark

To reproduce the results on WISE benchmark, you can use the script ./scripts/run_wise_self_reflection.sh. Specify your local checkpoint of UniCoT-7B-MoT and output directory using --model_path and outdir.

bash

gpu_num=8

for i in $(seq 0 $((gpu_num-1)));
do
    CUDA_VISIBLE_DEVICES=$i python inference_mdp_self_reflection_wise.py \
        --group_id $i \
        --group_num $gpu_num \
        --model_path "Fr0zencr4nE/UniCoT-7B-MoT" \
        --data_path "./eval/gen/wise/final_data.json" \
        --outdir "./results" \
        --cfg_text_scale 4 > process_log_$i.log 2>&1 &
done

wait
echo "All background processes finished."

General Inference

For general inference, prepare your prompts by formatting them into a .txt file, with one prompt per line (e.g., test_prompts.txt). Then, use the script ./scripts/run_user_self_reflection.sh to generate images from your prompts with the added benefit of the self-reflection mechanism.

bash

gpu_num=8

for i in $(seq 0 $((gpu_num-1)));
do
    CUDA_VISIBLE_DEVICES=$i python inference_mdp_self_reflection.py \
        --group_id $i \
        --group_num $gpu_num \
        --model_path "Fr0zencr4nE/UniCoT-7B-MoT" \
        --data_path "./test_prompts.txt" \
        --outdir "./results" \
        --cfg_text_scale 4 > process_log_$i.log 2>&1 &
done

wait
echo "All background processes finished."

Preliminary Results

Qualitative Results for Image Generation

Qualitative Results for Image Editing

Quantitative Results on WISE

We first conduct experiments on the WISE dataset to evaluate the reasoning capabilities of our method. As shown in the table below, our model achieves state-of-the-art (SOTA) performance among existing open-source unified models. Our results are averaged over five independent runs to ensure robustness and reliability.

	Culture↑	Time↑	Space↑	Biology↑	Physics↑	Chemistry↑	Overall↑
Janus	0.16	0.26	0.35	0.28	0.30	0.14	0.23
MetaQuery	0.56	0.55	0.62	0.49	0.63	0.41	0.55
Bagel-Think	0.76	0.69	0.75	0.65	0.75	0.58	0.70
Uni-CoT	0.76±0.009	0.70±0.0256	0.76±0.006	0.73±0.021	0.81±0.018	0.73±0.020	0.75±0.013
GPT4O	0.81	0.71	0.89	0.83	0.79	0.74	0.80

Furthermore, we apply our self-check mechanism to the images generated by the original Bagel model with think mode, aiming to evaluate our method’s ability to calibrate erroneous outputs. The results in the table below demonstrate that our model effectively refines the imperfect outputs generated by Bagel.

	Culture↑	Time↑	Space↑	Biology↑	Physics↑	Chemistry↑	Overall↑
Bagel-Think	0.76	0.69	0.75	0.65	0.75	0.58	0.70
Bagel-Think+Uni-CoT	0.75	0.70	0.75	0.71	0.74	0.69	0.73
Uni-CoT	0.76±0.009	0.70±0.0256	0.76±0.006	0.73±0.021	0.81±0.018	0.73±0.020	0.75±0.013
GPT4O	0.81	0.71	0.89	0.83	0.79	0.74	0.80

Quantitative Results on KRIS Bench

We also achieve state-of-the-art (SOTA) performance on the KRIS benchmark, even surpassing the closed-source model Gemini2.0.

Model	Attribute Perception	Spatial Perception	Temporal Perception	Factual Avg	Social Science	Natural Science	Conceptual Avg	Logical Reasoning	Instruction Decomposition	Procedural Avg	Overall Score
Gemini 2.0 (Google)	66.33	63.33	63.92	65.26	68.19	56.94	59.65	54.13	71.67	62.90	62.41
Step 3∅ vision (StepFun)	69.67	61.08	63.25	66.70	66.88	60.88	62.32	49.06	54.92	51.99	61.43
Doubao (ByteDance)	70.92	59.17	40.58	63.30	65.50	61.19	62.23	47.75	60.58	54.17	60.70
BAGEL (ByteDance)	64.27	62.42	42.45	60.26	55.40	56.01	55.86	52.54	50.56	51.69	56.21
BAGEL-Think (ByteDance)	67.42	68.33	58.67	66.18	63.55	61.40	61.92	48.12	50.22	49.02	60.18
Uni-Cot	72.76	72.87	67.10	71.85	70.81	66.00	67.16	53.43	73.93	63.68	68.00
GPT-4o (OpenAI)	83.17	79.08	68.25	79.80	85.50	80.06	81.37	71.56	85.08	78.32	80.09

Citation

If you find Uni-CoT useful for your research, please consider citing the paper:

bibtex

@misc{qin2025unicot,
      title={Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision}, 
      author={Luozheng Qin and Jia Gong and Yuqing Sun and Tianjiao Li and Mengping Yang and Xiaomeng Yang and Chao Qu and Zhiyu Tan and Hao Li},
      year={2025},
      eprint={2508.05606},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.05606}, 
}

Acknowledgement

We acknowledge the contributions of the following projects that inspired and supported Uni-CoT:

Bagel proposed by ByteDance-Seed team.
WISE proposed by PKU-YuanGroup.
KRIS-Bench proposed by Stepfun.
RISE-Bench proposed by Shanghai AI Lab.