UltraX is a function-calling refinement framework for large-scale pre-training data. Instead of end-to-end text rewriting, UltraX trains a lightweight refinement model to predict structured editing operations — including insertion, deletion, and modification — which are then deterministically executed on the original text.
This repository provides the UltraX refinement models, which are trained via full-parameter SFT to predict cleaning function calls for web-crawled text:
Model
Description
UltraX
Default model: instruction-guided with edit-weighted sampling (main experiment)
UltraX-No-Instruction
Ablation variant: trained without system instruction
UltraX-Preservation-Weighted
Ablation variant: trained with preservation-weighted sampling (60% keep_all)
📢 News
[2026.07.13] UltraX codebase, refinement model, and refined datasets are now available on GitHub and Hugging Face. 🚀🚀🚀
[2026.07.10] UltraX technical report is available on arXiv. 🔥🔥🔥
💡 Highlights
Function-Calling Refinement: Instead of end-to-end text rewriting, UltraX predicts structured editing operations (keep_all, remove_all, remove_lines, replace_str, add_line), enabling fine-grained instance-level editing with deterministic execution.
LAM + DCR Pipeline: Line Alignment and Mapping (LAM) aligns original and refined text at line level, while Dynamic Context Replacement (DCR) converts character-level edits into reliable replace_str operations with unique context anchoring.
Lightweight & Efficient: The refinement model is a standard SFT model that predicts structured function calls, significantly cheaper than end-to-end LLM rewriting while achieving superior performance.
🔬 Pipeline Overview
🔍 Model Details
All models are trained with the following configuration:
Setting
Value
Training Framework
ms-swift with DeepSpeed ZeRO3
Training Type
Full-parameter SFT
Max Sequence Length
20,480 tokens
Learning Rate
3e-5 (cosine decay, min lr 3e-6)
Warmup Ratio
0.03
Precision
bfloat16
Hardware
8x GPU
Model Variants
UltraX (Instruction-Guided, Edit-Weighted): The default configuration. The model is trained with a system instruction that defines the editing operations and cleaning rules, using edit-weighted sampling.
UltraX-No-Instruction: Trained without system instruction. Demonstrates that programmatic refinement brings clear gains even without explicit task protocol (Avg: 45.73 vs Raw 45.08), though instruction guidance further improves performance (Avg: 46.14).
UltraX-Preservation-Weighted: Trained with preservation-weighted sampling where keep_all accounts for 60% of training data. Shows stronger performance on certain tasks (ARC-C, ARC-E, OBQA, SciQ), demonstrating the trade-off between editing aggressiveness and content preservation.
UltraX builds upon insights from ProX and RefineX.
Thanks for their awesome work! Open-source contributions make UltraX possible! 🙌
📖 Citation
If you find our work useful, please consider citing:
bibtex
1@misc{ultrax2026,
2 title={UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing},
3 author={Xinlong Zhao and Dongsheng Liu and Hengyu Zhao and Zixuan Fu and Zheng Wang and Jie Cai and Jie Zhou and Qiang Ma and Xuanhe Zhou and Xu Han and Yudong Wang and Zhiyuan Liu},
4 year={2026},
5 eprint={2607.08646},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8}
📜 License
This project is licensed under the Apache 2.0 license.
