As large language models serve an increasing number of requests, cumulative inference cost is becoming more important relative to the one-time cost of training. Prompt prefill and autoregressive decoding impose different hardware demands: prefill is highly parallel and typically compute-bound, whereas decoding is sequential and often constrained by memory bandwidth. Conventional width, depth, and Mixture-of-Experts scaling increase computation in both phases together, even when additional capacity is primarily needed for continuation prediction.

We introduce the Dual-Flow Transformer, an architecture that allocates learned computation asymmetrically between prefill and decoding. Its primary flow is a complete causal language model that processes the full prompt and alone maintains the persistent key-value cache. An auxiliary flow can be omitted over the prompt and activated only from its final position onward, adding continuation-prediction computation without introducing a second persistent KV cache or altering the primary flow’s cached states. When combined with Mixture-of-Experts layers, the two flows route through the expert pool separately, allowing more expert computation to be activated during decoding while retaining the lower prompt-side expert budget of the primary flow.

We train the two flows jointly using a confidence-weighted likelihood objective, where the primary distribution determines how their token-level losses are combined. Experiments show that Dual-Flow models improve continuation modeling relative to compute-matched conventional scaling, while preserving the primary model’s prefill path and KV-cache footprint. These results suggest that model capacity need not be allocated uniformly across inference phases, and that phase-aware computation provides a practical new scaling dimension for language models.

This repository provides an early public version of the manuscript while the arXiv submission is being processed. The PDF may be updated as the work evolves.

A BibTeX entry will be added once the arXiv identifier is available. For now, please cite the manuscript using its title and authors:

The manuscript is released under the Creative Commons Attribution 4.0 International License.