Doge uses Dynamic Mask Attention as sequence transformation and can use Multi-Layer Perceptron or Cross Domain Mixture of Experts as state transformation. Dynamic Mask Attention allows the Transformer to use self-attention during training and state space during inference, and Cross Domain Mixture of Experts can directly inherit the weights of Multi-Layer Perceptron for further training. This model is trained by
SmallDoge community, for detailed algorithm and model architecture, please refer to
Wonderful Matrices, all training details and code are publicly available on the
small-doge repository.
1from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig, TextStreamer
2
3tokenizer = AutoTokenizer.from_pretrained("SmallDoge/Doge-60M-Instruct")
4model = AutoModelForCausalLM.from_pretrained("SmallDoge/Doge-60M-Instruct", trust_remote_code=True)
5
6generation_config = GenerationConfig(
7 max_new_tokens=100,
8 use_cache=True,
9 do_sample=True,
10 temperature=0.8,
11 top_p=0.9,
12 repetition_penalty=1.0
13)
14steamer = TextStreamer(
15 tokenizer=tokenizer,
16 skip_prompt=True
17)
18
19prompt = "Hi, how are you doing today?"
20conversation = [
21 {"role": "user", "content": prompt}
22]
23inputs = tokenizer.apply_chat_template(
24 conversation=conversation,
25 tokenize=True,
26 return_tensors="pt",
27)
28
29outputs = model.generate(
30 inputs,
31 tokenizer=tokenizer,
32 generation_config=generation_config,
33 streamer=steamer
34)
We build the Doge-Instruct by first SFT on
SmolTalk and then DPO on
UltraFeedback Binarized.
1@misc{shi2024wonderfulmatrices,
2 title={Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture},
3 author={Jingze Shi and Bingheng Wu},
4 year={2024},
5 eprint={2412.11834},
6 archivePrefix={arXiv},
7 primaryClass={cs.LG},
8 url={https://arxiv.org/abs/2412.11834},
9}
