Views
No views yet
1pip install transformers torch accelerate
2python inference.pyinference.py for a complete working example.1from inference import build_prompt, generate, FileChunk, DIFF_FORMAT
2
3prompt, code_block, block_start, relative_cursor = build_prompt(
4 file_path="example.py",
5 file_contents=edited_contents,
6 cursor_position=cursor_position,
7 recent_changes=recent_changes,
8 retrieval_chunks=[FileChunk("utils.py", "def helper(): ...")],
9 changes_above_cursor=False,
10)
11
12completion = generate(model, tokenizer, prompt, device="cuda")<|file_sep|> delimiters and a <|cursor|> marker:<|file_sep|>{file_path}
{file_contents}
{retrieval_chunks}
{recent_changes_as_diffs}
<|file_sep|>original/{file_path}:{start}:{end}
{code_block_before_last_edit}
<|file_sep|>current/{file_path}:{start}:{end}
{code_block_with_cursor_marker}
<|file_sep|>updated/{file_path}:{start}:{end}
{prefill}updated/ section with the predicted new code block.original:/updated: format<|cursor|> inserted at cursor positionupdated/ section is seeded with a prefill to constrain generation:changes_above_cursor=False): Prefill everything up to the cursor line. The model only generates from the cursor line onward.changes_above_cursor=True): Prefill only the first line + trailing blank lines. Gives the model freedom to rewrite lines between the insertion point and cursor.<|file_sep|>{file_path}:{start_line}:{end_line}
original:
{old_code}
updated:
{new_code}| Base model | Qwen2.5-Coder-7B |
| Fine-tuning | SFT → GRPO → DPO |
| Parameters | 7B |
| Precision | bfloat16 |
| Context length | 32,768 tokens |
| Architecture | Qwen2 (28 layers, hidden dim 3584) |
| Stop tokens | <|endoftext|>, <|file_sep|> |
| Max output tokens | 1024 |
| Decoding | Greedy (temperature=0) |