This model was converted from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat to Block FP8 quantization (128x128) to be used as testing for DeepSeek-V3/R1. Dense's intermediate_size was padded to be a multiple of 128.
Below is the original model card.
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
1. Introduction
Last week, the release and buzz around DeepSeek-V2 have ignited widespread interest in MLA (Multi-head Latent Attention)! Many in the community suggested open-sourcing a smaller MoE model for in-depth research. And now DeepSeek-V2-Lite comes out:
16B total params, 2.4B active params, scratch training with 5.7T tokens
Outperforms 7B dense and 16B MoE on many English & Chinese benchmarks
Deployable on single 40G GPU, fine-tunable on 8x80G GPUs
DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation.
2. News
2024.05.16: We released the DeepSeek-V2-Lite.
2024.05.06: We released the DeepSeek-V2.
3. Model Downloads
With DeepSeek-V2, we are open-sourcing base and chat models across two sizes:
Due to the constraints of HuggingFace, the open-source code currently experiences slower performance than our internal codebase when running on GPUs with Huggingface. To facilitate the efficient execution of our model, we offer a dedicated vllm solution that optimizes performance for running our model effectively.
4. Evaluation Results
Base Model
Standard Benchmark
Benchmark
Domain
DeepSeek 7B (Dense)
DeepSeekMoE 16B
DeepSeek-V2-Lite (MoE-16B)
Architecture
-
MHA+Dense
MHA+MoE
MLA+MoE
MMLU
English
48.2
45.0
58.3
BBH
English
39.5
38.9
44.1
C-Eval
Chinese
45.0
40.6
60.3
CMMLU
Chinese
47.2
42.5
64.3
HumanEval
Code
26.2
26.8
29.9
MBPP
Code
39.0
39.2
43.2
GSM8K
Math
17.4
18.8
41.1
Math
Math
3.3
4.3
17.1
For more evaluation details, such as few-shot settings and prompts, please check our paper.
Chat Model
Standard Benchmark
Benchmark
Domain
DeepSeek 7B Chat (SFT)
DeepSeekMoE 16B Chat (SFT)
DeepSeek-V2-Lite 16B Chat (SFT)
MMLU
English
49.7
47.2
55.7
BBH
English
43.1
42.2
48.1
C-Eval
Chinese
44.7
40.0
60.1
CMMLU
Chinese
51.2
49.3
62.5
HumanEval
Code
45.1
45.7
57.3
MBPP
Code
39.0
46.2
45.8
GSM8K
Math
62.6
62.2
72.0
Math
Math
14.7
15.2
27.9
5. Model Architecture
DeepSeek-V2 adopts innovative architectures to guarantee economical training and efficient inference：
For attention, we design MLA (Multi-head Latent Attention), which utilizes low-rank key-value union compression to eliminate the bottleneck of inference-time key-value cache, thus supporting efficient inference.
For Feed-Forward Networks (FFNs), we adopt DeepSeekMoE architecture, a high-performance MoE architecture that enables training stronger models at lower costs.
DeepSeek-V2-Lite has 27 layers and a hidden dimension of 2048. It also employs MLA and has 16 attention heads, where each head has a dimension of 128. Its KV compression dimension is 512, but slightly different from DeepSeek-V2, it does not compress the queries. For the decoupled queries and key, it has a per-head dimension of 64. DeepSeek-V2-Lite also employs DeepSeekMoE, and all FFNs except for the first layer are replaced with MoE layers. Each MoE layer consists of 2 shared experts and 64 routed experts, where the intermediate hidden dimension of each expert is 1408. Among the routed experts, 6 experts will be activated for each token. Under this configuration, DeepSeek-V2-Lite comprises 15.7B total parameters, of which 2.4B are activated for each token.
6. Training Details
DeepSeek-V2-Lite is also trained from scratch on the same pre-training corpus of DeepSeek-V2, which is not polluted by any SFT data. It uses the AdamW optimizer with hyper-parameters set to $\beta_1=0.9$, $\beta_2=0.95$, and $\mathrm{weight_decay}=0.1$. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 after training about 80% of tokens, and again by 0.316 after training about 90% of tokens. The maximum learning rate is set to $4.2 \times 10^{-4}$, and the gradient clipping norm is set to 1.0. We do not employ the batch size scheduling strategy for it, and it is trained with a constant batch size of 4608 sequences. During pre-training, we set the maximum sequence length to 4K, and train DeepSeek-V2-Lite on 5.7T tokens. We leverage pipeline parallelism to deploy different layers of it on different devices, but for each layer, all experts will be deployed on the same device. Therefore, we only employ a small expert-level balance loss with $\alpha_{1}=0.001$, and do not employ device-level balance loss and communication balance loss for it. After pre-training, we also perform long-context extension, SFT for DeepSeek-V2-Lite and get a chat model called DeepSeek-V2-Lite Chat.
7. How to run locally
To utilize DeepSeek-V2-Lite in BF16 format for inference, 40GB*1 GPU is required.
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
34model_name ="deepseek-ai/DeepSeek-V2-Lite"5tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)6model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()7model.generation_config = GenerationConfig.from_pretrained(model_name)8model.generation_config.pad_token_id = model.generation_config.eos_token_id
910text ="An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"11inputs = tokenizer(text, return_tensors="pt")12outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)1314result = tokenizer.decode(outputs[0], skip_special_tokens=True)15print(result)
1from transformers import AutoTokenizer
2from vllm import LLM, SamplingParams
34max_model_len, tp_size =8192,15model_name ="deepseek-ai/DeepSeek-V2-Lite-Chat"6tokenizer = AutoTokenizer.from_pretrained(model_name)7llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True, enforce_eager=True)8sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])910messages_list =[11[{"role":"user","content":"Who are you?"}],12[{"role":"user","content":"Translate the following content into Chinese directly: DeepSeek-V2 adopts innovative architectures to guarantee economical training and efficient inference."}],13[{"role":"user","content":"Write a piece of quicksort code in C++."}],14]1516prompt_token_ids =[tokenizer.apply_chat_template(messages, add_generation_prompt=True)for messages in messages_list]1718outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)1920generated_text =[output.outputs[0].text for output in outputs]21print(generated_text)
LangChain Support
Since our API is compatible with OpenAI, you can easily use it in langchain.
Here is an example:
This code repository is licensed under the MIT License. The use of DeepSeek-V2 Base/Chat models is subject to the Model License. DeepSeek-V2 series (including Base and Chat) supports commercial use.
9. Citation
@misc{deepseekv2,
title={DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model},
author={DeepSeek-AI},
year={2024},
eprint={2405.04434},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
10. Contact
If you have any questions, please raise an issue or contact us at service@deepseek.com.
