from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-1124-7B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-2-1124-7B-Instruct")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "olmo-code-sft-7b-lr0.0005")

# Generate code
prompt = "Write a Python function to calculate fibonacci numbers"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was fine-tuned on Python code data with instruction-response pairs.

Training Procedure

Training Hyperparameters

Training regime: LoRA fine-tuning
Learning rate: 0.0005
LoRA rank: 64
LoRA alpha: 128
LoRA dropout: 0.05
Target modules: q_proj, k_proj, o_proj, down_proj, up_proj, gate_proj, v_proj

Speeds, Sizes, Times

Model size: 7B
Training time: Varies by experiment
Checkpoint size: LoRA adapter only (~2GB)

Evaluation

The model was evaluated on Python code generation tasks with focus on:

Code quality
Instruction following
Python syntax correctness

Technical Specifications

Model Architecture and Objective

Architecture: LoRA adapter on top of allenai/OLMo-2-1124-7B-Instruct
Objective: Causal language modeling for code generation
Task type: CAUSAL_LM

Compute Infrastructure

Hardware: GPU cluster
Software: PEFT, Transformers, PyTorch

Citation

If you use this model, please cite:

bibtex

@misc{olmo-code-sft-7b,
  author = {OLMo Code SFT Team},
  title = {OLMo Code SFT - 7B Model},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  howpublished = {\url{https://huggingface.co/olmo-code-sft-7b-lr0.0005}},
}

Model Card Authors

OLMo Code SFT Team

Model Card Contact

For questions about this model, please open an issue in the repository.

dipikakhullar/olmo-code-sft-7b-lr0.0005

OLMo Code SFT - 7B Model

Model Details

Model Description

Model Sources

Uses

Direct Use

Downstream Use

Out-of-Scope Use

Bias, Risks, and Limitations

How to Get Started with the Model