A LoRA fine-tuned adapter for deepseek-ai/deepseek-coder-6.7b-base, specialized for Fill-in-the-Middle (FIM) code completion on the reth Rust codebase.

On a held-out reth test split it more than doubles the base model's exact-match rate (see Evaluation).

Want plug-and-play (Ollama / llama.cpp)? Use the GGUF build (q4_k_m / q8_0) — no base model or PEFT needed.

Model Details

Base Model: deepseek-ai/deepseek-coder-6.7b-base
Fine-tuning Method: LoRA (Low-Rank Adaptation) with 4-bit quantization (QLoRA)
Training Data: AST-extracted FIM samples from the reth Rust codebase
Task: Fill-in-the-Middle code completion (predict missing code between prefix and suffix)

Training Configuration

Parameter	Value
LoRA Rank / Alpha	64 / 128
LoRA Dropout	0.05
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Quantization	4-bit (nf4, double quant)
Compute precision	bf16
Learning Rate	2e-5 (cosine)
Epochs	1
Training samples	5,000 AST-mined FIM examples
Max Sequence Length	4096
Stack	transformers 4.49.0, trl 0.17.0, peft 0.19.1

FIM Format

Uses the DeepSeek Coder FIM token format:

<｜fim▁begin｜>[code before cursor]<｜fim▁hole｜>[code after cursor]<｜fim▁end｜>[generated completion]

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/deepseek-coder-6.7b-base", device_map="auto", load_in_4bit=True)
model = PeftModel.from_pretrained(base, "viplismism/deepseek-coder-6.7b-fim-reth-v1")
tokenizer = AutoTokenizer.from_pretrained("viplismism/deepseek-coder-6.7b-fim-reth-v1")

prompt = "<｜fim▁begin｜>fn add(a: i32, b: i32) -> i32 {\n    <｜fim▁hole｜>\n}<｜fim▁end｜>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Evaluation

Evaluated on a held-out reth test split (200 samples never seen in training), tuned adapter vs. the unmodified base model, same FIM prompts:

Metric	Tuned (base + adapter)	Base model	Δ
pass@1 (exact match)	31.0%	13.5%	+17.5 pts (2.3×)
Edit similarity	0.650	0.453	+0.197
BLEU	0.459	0.294	+0.165
Token overlap	0.559	0.374	+0.185

The fine-tune wins on every metric. (pass@1 is strict exact-string match; edit-similarity/BLEU capture near-correct completions.)

Limitations

Tuned on Rust code from a single repository (reth) — generalizes less to other languages/repos.
FIM-only: expects the FIM prompt format above, not free-form chat.
Inherits the biases/limitations of deepseek-coder-6.7b-base.

License

Use of this adapter is subject to the DeepSeek License Agreement governing the base model.

Framework versions

transformers 4.49.0 · trl 0.17.0 · PEFT 0.19.1