You can deploy this model and start earning money today!

RichardErkhov/
deepseek-ai_-_DeepSeek-Coder-V2-Lite-Base-gguf

Name: deepseek-ai_-_DeepSeek-Coder-V2-Lite-Base-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

DeepSeek-Coder-V2-Lite-Base - GGUF

Model creator: https://huggingface.co/deepseek-ai/
Original model: https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Base/

Name	Quant method	Size
DeepSeek-Coder-V2-Lite-Base.Q2_K.gguf	Q2_K	5.99GB
DeepSeek-Coder-V2-Lite-Base.IQ3_XS.gguf	IQ3_XS	6.63GB
DeepSeek-Coder-V2-Lite-Base.IQ3_S.gguf	IQ3_S	6.97GB
DeepSeek-Coder-V2-Lite-Base.Q3_K_S.gguf	Q3_K_S	6.97GB
DeepSeek-Coder-V2-Lite-Base.IQ3_M.gguf	IQ3_M	7.03GB
DeepSeek-Coder-V2-Lite-Base.Q3_K.gguf	Q3_K	7.57GB
DeepSeek-Coder-V2-Lite-Base.Q3_K_M.gguf	Q3_K_M	7.57GB
DeepSeek-Coder-V2-Lite-Base.Q3_K_L.gguf	Q3_K_L	7.88GB
DeepSeek-Coder-V2-Lite-Base.IQ4_XS.gguf	IQ4_XS	8.05GB
DeepSeek-Coder-V2-Lite-Base.Q4_0.gguf	Q4_0	8.29GB
DeepSeek-Coder-V2-Lite-Base.IQ4_NL.gguf	IQ4_NL	8.36GB
DeepSeek-Coder-V2-Lite-Base.Q4_K_S.gguf	Q4_K_S	8.88GB
DeepSeek-Coder-V2-Lite-Base.Q4_K.gguf	Q4_K	9.65GB
DeepSeek-Coder-V2-Lite-Base.Q4_K_M.gguf	Q4_K_M	9.65GB
DeepSeek-Coder-V2-Lite-Base.Q4_1.gguf	Q4_1	9.2GB
DeepSeek-Coder-V2-Lite-Base.Q5_0.gguf	Q5_0	10.1GB
DeepSeek-Coder-V2-Lite-Base.Q5_K_S.gguf	Q5_K_S	10.38GB
DeepSeek-Coder-V2-Lite-Base.Q5_K.gguf	Q5_K	11.04GB
DeepSeek-Coder-V2-Lite-Base.Q5_K_M.gguf	Q5_K_M	11.04GB
DeepSeek-Coder-V2-Lite-Base.Q5_1.gguf	Q5_1	11.0GB
DeepSeek-Coder-V2-Lite-Base.Q6_K.gguf	Q6_K	13.1GB
DeepSeek-Coder-V2-Lite-Base.Q8_0.gguf	Q8_0	15.56GB

Original model description:

license: other license_name: deepseek-license license_link: LICENSE

DeepSeek-V2

Discord

Wechat

Twitter Follow

API Platform | How to Use | License |

Paper Link👁️

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

1. Introduction

We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially enhances the coding and mathematical reasoning capabilities of DeepSeek-V2, while maintaining comparable performance in general language tasks. Compared to DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in various aspects of code-related tasks, as well as reasoning and general capabilities. Additionally, DeepSeek-Coder-V2 expands its support for programming languages from 86 to 338, while extending the context length from 16K to 128K.

In standard benchmark evaluations, DeepSeek-Coder-V2 achieves superior performance compared to closed-source models such as GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks. The list of supported programming languages can be found here.

2. Model Downloads

We release the DeepSeek-Coder-V2 with 16B and 236B parameters based on the DeepSeekMoE framework, which has actived parameters of only 2.4B and 21B , including base and instruct models, to the public.

Model	#Total Params	#Active Params	Context Length	Download
DeepSeek-Coder-V2-Lite-Base	16B	2.4B	128k	🤗 HuggingFace
DeepSeek-Coder-V2-Lite-Instruct	16B	2.4B	128k	🤗 HuggingFace
DeepSeek-Coder-V2-Base	236B	21B	128k	🤗 HuggingFace
DeepSeek-Coder-V2-Instruct	236B	21B	128k	🤗 HuggingFace

3. Chat Website

You can chat with the DeepSeek-Coder-V2 on DeepSeek's official website: coder.deepseek.com

4. API Platform

We also provide OpenAI-Compatible API at DeepSeek Platform: platform.deepseek.com, and you can also pay-as-you-go at an unbeatable price.

5. How to run locally

Here, we provide some examples of how to use DeepSeek-Coder-V2-Lite model. If you want to utilize DeepSeek-Coder-V2 in BF16 format for inference, 80GB*8 GPUs are required.

Inference with Huggingface's Transformers

You can directly employ Huggingface's Transformers for model inference.

Code Completion

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base", trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()
input_text = "#write a quick sort algorithm"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Code Insertion

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base", trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()
input_text = """<｜fim▁begin｜>def quick_sort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[0]
    left = []
    right = []
<｜fim▁hole｜>
        if arr[i] < pivot:
            left.append(arr[i])
        else:
            right.append(arr[i])
    return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜>"""
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True)[len(input_text):])

Chat Completion

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct", trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()
messages=[
    { 'role': 'user', 'content': "write a quick sort algorithm in python."}
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
# tokenizer.eos_token_id is the id of <｜end▁of▁sentence｜>  token
outputs = model.generate(inputs, max_new_tokens=512, do_sample=False, top_k=50, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True))

The complete chat template can be found within tokenizer_config.json located in the huggingface model repository.

An example of chat template is as belows:

bash

<｜begin▁of▁sentence｜>User: {user_message_1}

Assistant: {assistant_message_1}<｜end▁of▁sentence｜>User: {user_message_2}

Assistant:

You can also add an optional system message:

bash

<｜begin▁of▁sentence｜>{system_message}

User: {user_message_1}

Assistant: {assistant_message_1}<｜end▁of▁sentence｜>User: {user_message_2}

Assistant:

Inference with vLLM (recommended)

To utilize vLLM for model inference, please merge this Pull Request into your vLLM codebase: https://github.com/vllm-project/vllm/pull/4650.

python

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

max_model_len, tp_size = 8192, 1
model_name = "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True, enforce_eager=True)
sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])

messages_list = [
    [{"role": "user", "content": "Who are you?"}],
    [{"role": "user", "content": "write a quick sort algorithm in python."}],
    [{"role": "user", "content": "Write a piece of quicksort code in C++."}],
]

prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list]

outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)

generated_text = [output.outputs[0].text for output in outputs]
print(generated_text)

6. License

This code repository is licensed under the MIT License. The use of DeepSeek-Coder-V2 Base/Instruct models is subject to the Model License. DeepSeek-Coder-V2 series (including Base and Instruct) supports commercial use.

7. Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.

RichardErkhov/deepseek-ai_-_DeepSeek-Coder-V2-Lite-Base-gguf

Original model description:

license: other license_name: deepseek-license license_link: LICENSE

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

1. Introduction

2. Model Downloads

3. Chat Website

4. API Platform

5. How to run locally

Inference with Huggingface's Transformers

Code Completion

Code Insertion

Chat Completion

Inference with vLLM (recommended)

6. License

7. Contact

RichardErkhov/
deepseek-ai_-_DeepSeek-Coder-V2-Lite-Base-gguf