Ring-lite-linear-preview is a hybrid-linear MoE LLM provided and open-sourced by InclusionAI, which has 17.1B parameters with 3.0B activated parameters. It is a long reasoning model based on hybrid-linear attention, achieving near-linear computational complexity and near-constant space complexity during inference. This model was converted from
Ling-lite-0220, which adopts the softmax attention-based architecture. It matches the performance of DeepSeek-R1-Distill-Qwen-7B on standardized reasoning benchmarks while substantially reducing computational overhead in both training and inference phases. In certain generation speed tests based on vLLM, we observed that the throughput was more than doubled compared to softmax attention models of the same scale (e.g., Ling-lite). To the best of our knowledge, it is the first open-source hybrid-linear reasoning language model.
In terms of the evaluation of reasoning ability， Ring-lite-linear-preview achieves 55.0 on AIME24 and 93.8 on MATH-500.
To evaluate the generation throughput, we deploy Ring-lite-linear and the softmax-attention-based Ring-lite based on vLLM on a single NVIDIA A100 GPU. We conduct two sets of experiments:
-
Long Input Evaluation: We measure the time-to-first-token (TTFT) with varying input sequence lengths (from 512 to 384k tokens) using batch size 1 and TP=1. As shown in the top figure, at 384k input length, Ring-lite-linear achieves 3.5× faster TTFT compared to the softmax-attention-based model.
-
Long Output Evaluation: We fix the input sequence length to 1 and measure the end-to-end (E2E) generation time required for generating output sequences of varying lengths (from 512 to 32k tokens) with batch size 64 and TP=1. As illustrated in the bottom figure, at 32k output length, Ring-lite-linear achieves 2.2× throughput of the softmax-attention-based Ring-lite.
These results demonstrate that our hybrid linear attention mechanism significantly improves both input processing efficiency and generation throughput, especially for long context scenarios.
Additionally, to illustrate the advantage in inference speed, we present a comparison between Ring-lite-linear-preview and softmax-attention-based Ring-lite under a batch size of 64 and an output length of 16k (60x speedup). It can be observed that the KV cache usage of Ring-lite-linear-preview is nearly 1/6 that of Ring-lite, and the E2E time is reduced by 27.24% compared with Ring-lite.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name = "inclusionAI/Ring-lite-linear-preview"
4
5model = AutoModelForCausalLM.from_pretrained(
6 model_name,
7 torch_dtype="auto",
8 device_map="auto"
9)
10tokenizer = AutoTokenizer.from_pretrained(model_name)
11
12prompt = "Give me a short introduction to large language models."
13messages = [
14 {"role": "system", "content": "You are Ring, an assistant created by inclusionAI"},
15 {"role": "user", "content": prompt}
16]
17text = tokenizer.apply_chat_template(
18 messages,
19 tokenize=False,
20 add_generation_prompt=True
21)
22model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
23
24generated_ids = model.generate(
25 **model_inputs,
26 max_new_tokens=8192
27)
28generated_ids = [
29 output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
30]
31
32response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
This code repository is licensed under
the MIT License.
