Caveat on the metric. Eval loss measures token prediction, not whether generated code runs. Treat it as a training-health signal rather than a capability benchmark; pass@1 on HumanEval or MBPP would be the meaningful measure and is not yet wired up.

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "openbmb/MiniCPM5-1B"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "HIMANSHUKUMARJHA/minicpm5-1b-code-lora")

messages = [{'role': 'user', 'content': 'Write a Python function that reverses a linked list.'}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
inputs.pop("token_type_ids", None)   # this architecture's generate() rejects it
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Requires transformers>=4.51, which is the first version that reads MiniCPM's standalone chat_template.jinja.

Training


Base model	`openbmb/MiniCPM5-1B`
Dataset	`m-a-p/CodeFeedback-Filtered-Instruction`
Training rows	60,000
Steps	6,000 (~1.6 epochs at effective batch 16)
LoRA rank / alpha	32 / 64
Target modules	attention + MLP projections
LR schedule	0.0002 cosine, 3% warmup
Precision	bfloat16
Hardware	1x A100

Reproduced with AutoTune:

modal run finetune.py --profile code

Data handling

The dataset is shuffled with a fixed seed, then a 200-row holdout is taken before training. Rows the formatter cannot parse are dropped and exact duplicates removed, because these datasets contain repeats that would otherwise leak holdout examples into training.

12,515 exact duplicates were removed from this dataset before training.

Note on the metric. Eval loss measures token prediction, not whether generated code runs. Treat it as a training-health signal, not a capability benchmark. Pass@1 on HumanEval or MBPP would be the meaningful measure.

Limitations

This is a 1B parameter model. It is useful for coding assistance at small scale and for on-device or cost-sensitive settings, but it will not match a large general model. Outputs should be validated before use. The adapter inherits any bias present in the training dataset.