This model is derived from OpenCoder-1.5B-Base by applying an additional context extension fine-tuning with an Adjustment of the Base Frequency parameter of RoPE from 10,000 to 500,000. The number of optimization steps is 512 with a batch size of 128 on sequences up to 4,096 in length. The repository context is not used to obtain this checkpoint. More details on the training procedure and other aspects, including all code used, can be found on the Home Page of the project. Note that this model is created with the intent to answer specific research questions and not to gain the maximum possible performance on the repository-level code completion setup. Consider it more as a baseline.

The associated research was initialized and conducted by the JetBrains Research association.

Exact Match on the inproject lines of the large-context subset of the Project-Level Code Completion task from the Long Code Arena benchmark. This checkpoint (dashed orange curve) demonstrates its best performance at a context length of 16,384. "1K" refers to 1,024 tokens. The star markers denote the context length used during the repository-level pre-training stage.

Quickstart

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "sapromak/OpenCoder-1.5B-Base-16K-via-4K"
tokenizer_name = "infly/OpenCoder-1.5B-Base"

model = AutoModelForCausalLM.from_pretrained(model_name,
                                             torch_dtype=torch.bfloat16,
                                             device_map="auto",
                                             trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, trust_remote_code=True)

inputs = tokenizer("# write a quick sort algorithm", return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_new_tokens=256)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)