Thinking with Tables (TWT)
Model summary
Thinking with Tables (TWT) is a multi-modal model family trained for Tabular-Vision Multi-Modal Understanding (TVMU): jointly using table-related images, text, and executable Python in a sandbox to solve multi-modal table question answering and multi-modal table prediction tasks.
The approach is program-aided neuro-symbolic reasoning: the model emits structured outputs (<analy>, <code>, <answer>), code is executed in a controlled environment, and results are fed back in <code_result> for multi-turn reasoning until a final answer is produced.
Code: See the project README for clone URL, environment setup, ms-swift training scripts, and evaluation pipelines.
Base model and training
- Typical base: multi-modal vision-language model (e.g. Qwen3-VL-8B in the paper).
- Stage 1 — Task-Oriented SFT (TO-SFT): supervised fine-tuning so the model follows the
<analy> / <code> / <answer> protocol; <code_result> is masked during training (loss on model-generated tokens only).
- Stage 2 — Adaptive Loss-Scaled GRPO (AL-GRPO): reinforcement learning with multi-turn rollouts (code execution in sandbox, reward on final answer); only successfully executed code segments contribute to the GRPO loss (
default+code loss scaling).
Approximate data scale (from the paper / README): SFT on the order of ~1.5K table QA + ~1.2K table prediction + ~5K additional samples; GRPO on the order of ~0.5K table QA + ~0.4K table prediction (exact counts may vary by release).
Intended use
Primary use cases
- Table QA: Given a table header/sample image, natural-language question, and path to the full CSV, the model generates code (e.g. pandas) to read and reason over the full table in a sandbox.
- Table prediction: Given a task image, metadata, and paths to tabular data / images, the model may call task-specific tools (e.g.
ResNet50Predict for vision features) plus tabular code to produce a classification or regression answer.
Out-of-scope / not guaranteed
- Tasks that require no code or no environment access may not match the training distribution.
- The model is not a substitute for financial, legal, or safety-critical decisions without human review.
Output format (inference contract)
The model is expected to produce parseable segments:
| Tag | Role |
|---|
<analy>...</analy> | Reasoning |
<code>...</code> | Python for the sandbox |
<code_result>...</code_result> | Filled by the system after execution (not model-generated) |
<answer>...</answer> | Final answer |
Inference is typically multi-turn: generate → execute <code> → append <code_result> → repeat until <answer>.
Evaluation notes
- Table QA: Benchmarks include settings aligned with TAT-QA, FinQA, HiTab, and similar; the evaluation harness uses an OpenAI-compatible API (e.g. vLLM) plus a sandbox loop.
- Table prediction: Tasks such as Pawpularity, Adoption, SkinCA, Paintings require ResNet50 weights and a
tools.py / ResNet50Predict(task, image_path) interface in the sandbox path.
Important: At inference time the full table is usually not pasted into the prompt; the model must read the file via generated code using the provided path.
Limitations
- Depends on a correctly configured sandbox (timeouts, allowed imports, file paths, optional ResNet weights).
- Code execution failures or path errors break the multi-turn loop; robustness depends on deployment.
- Performance varies by table complexity, OCR/image quality, and domain shift from training data.
Citation
1@article{yu2026thinking,
2 title={Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning},
3 author={Yu, Kun-Yang and Zhou, Zhi and Tian, Shi-Yu and Yang, Xiao-Wen and Jia, Zi-Yi and Yang, Ming and Cheng, Zi-Jian and Guo, Lan-Zhe and Li, Yu-Feng},
4 journal={arXiv preprint arXiv:2603.24004},
5 year={2026}
6}
This model is trained based on Qwen3VL, thanks!
Contact
Questions about paths or reproduction: open an issue in the project repository or email
yuky@lamda.nju.edu.cn (as listed in the main README).
