from transformers import AutoConfig

config = AutoConfig.from_pretrained(
    "micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5"
)

print(config.max_position_embeddings)
print(config.rope_scaling)

Current configuration:

python

{
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 8192,
    "rope_theta": 1000000.0
}

Note: Long-context performance beyond the original context length should be evaluated carefully for specific workloads.

Dataset

The model was trained on converted Claude Fable 5 coding traces formatted into OpenAI-style conversations.

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are CodeMate, an expert programming assistant."
    },
    {
        "role": "user",
        "content": "Write a Python function to compute edit distance."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

Experimental research model.
Long-context capabilities require further evaluation.
May generate incorrect or insecure code.

Acknowledgements

Built upon:

Qwen2
Transformers
PEFT
Hugging Face
llama.cpp
Claude Fable traces

Disclaimer

This project is an independent research effort and is not affiliated with or endorsed by Anthropic, Claude, Alibaba, or Qwen.

Author

Built by micymike 🇰🇪