Views
No views yet
dots.llm1 or visit the dots1 collection, and you will find all you need! Enjoy!dots.llm1 series. Check our report for more details!dots.llm1 model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models.
Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretrained on high-quality corpus without synthetic data. To foster further research, we open-source intermediate training checkpoints spanning the entire training process, providing valuable insights into the learning dynamics of large language models.
dots.llm1 model. which has the following features:dots.llm1 include:dots.llm1 is an open-source model that activates only 14B parameters at inference, delivering both comprehensive capabilities and high computational efficiency.| Model | #Total Params | #Activated Params | Context Length | Download Link |
|---|---|---|---|---|
| dots.llm1.base | 142B | 14B | 32K | 🤗 Hugging Face |
| dots.llm1.inst | 142B | 14B | 32K | 🤗 Hugging Face |
1docker run --gpus all \
2 -v ~/.cache/huggingface:/root/.cache/huggingface \
3 -p 8000:8000 \
4 --ipc=host \
5 rednotehilab/dots1:vllm-openai-v0.9.0.1 \
6 --model rednote-hilab/dots.llm1.inst \
7 --tensor-parallel-size 8 \
8 --trust-remote-code \
9 --served-model-name dots11curl http://localhost:8000/v1/chat/completions \
2 -H "Content-Type: application/json" \
3 -d '{
4 "model": "dots1",
5 "messages": [
6 {"role": "system", "content": "You are a helpful assistant."},
7 {"role": "user", "content": "Who won the world series in 2020?"}
8 ],
9 "max_tokens": 32,
10 "temperature": 0
11 }'1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
3
4model_name = "rednote-hilab/dots.llm1.base"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6
7model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)
8
9text = "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"
10inputs = tokenizer(text, return_tensors="pt")
11outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)
12result = tokenizer.decode(outputs[0], skip_special_tokens=True)
13print(result)1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
3
4model_name = "rednote-hilab/dots.llm1.inst"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6
7model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)
8
9messages = [
10 {"role": "user", "content": "Write a piece of quicksort code in C++"}
11]
12input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
13outputs = model.generate(input_tensor.to(model.device), max_new_tokens=200)
14
15result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
16print(result)vllm serve dots.llm1.inst --port 8000 --tensor-parallel-size 8http://localhost:8000/v1.python -m sglang.launch_server --model-path dots.llm1.inst --tp 8 --host 0.0.0.0 --port 8000http://localhost:8000/v1.dots.llm1 is useful or want to use in your projects, please kindly cite our paper:@misc{huo2025dotsllm1technicalreport,
title={dots.llm1 Technical Report},
author={Bi Huo and Bin Tu and Cheng Qin and Da Zheng and Debing Zhang and Dongjie Zhang and En Li and Fu Guo and Jian Yao and Jie Lou and Junfeng Tian and Li Hu and Ran Zhu and Shengdong Chen and Shuo Liu and Su Guang and Te Wo and Weijun Zhang and Xiaoming Shi and Xinxin Peng and Xing Wu and Yawen Liu and Yuqiu Ji and Ze Wen and Zhenhai Liu and Zichao Li and Zilong Liao},
year={2025},
eprint={2506.05767},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.05767},
}