CodeLlama Embedded Test Generator (v10)
This model generates production-grade unit tests for embedded C/C++ code.
It's a merged adapter of CodeLlama-7B fine-tuned with:
- 8-bit quantization
- Flash Attention 2
- Linear RoPE scaling (factor=2.0)
- Custom instruction tuning on embedded unit tests
Key Features
- Generates framework-agnostic test cases
- Optimized for embedded systems constraints
- Strict output formatting (no boilerplate)
- Special tokens for structured prompting
- 8192 context window support
Technical Specifications
| Component | Configuration |
|---|
| Base Model | CodeLlama-7B-HF |
| Fine-tuning | LoRA (r=64, alpha=32) |
| Quantization | 8-bit (llm_int8_threshold=6.0) |
| Attention | Flash Attention 2 |
| Context Window | 8192 tokens (RoPE scaled) |
| Training Epochs | 2 |
| Batch Size | 2 (effective 8 with grad accum) |
| Learning Rate | 1.5e-4 |
| Optimizer | Paged AdamW 8-bit |
🧪 Prompt Structure
<|system|>
Generate comprehensive, framework-agnostic unit tests for C/C++ code. Focus on:
Testing all functions and edge cases
Avoiding redundant headers
Covering boundary conditions and error scenarios
Using clear test names without repetitions
Generate ONLY test logic without framework-specific macros.
<|user|>
Generate unit tests for:
{your_function_here}
<|assistant|>
🚀 Inference Example
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4model_id = "Utkarsh524/codellama_utests_full_new_ver10"
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(
7model_id,
8torch_dtype=torch.float16,
9device_map="auto",
10attn_implementation="flash_attention_2" # Recommended for speed
11)
12
13def generate_tests(function_code):
14prompt = f"""<|system|>
15Generate comprehensive, framework-agnostic unit tests for C/C++ code. Focus on:
16
17Testing all functions and edge cases
18
19Avoiding redundant headers
20
21Covering boundary conditions and error scenarios
22
23Using clear test names without repetitions
24Generate ONLY test logic without framework-specific macros.
25
26<|user|>
27Generate unit tests for:
28{function_code}
29
30<|assistant|>
31"""
32inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=8192).to("cuda")
33outputs = model.generate(
34**inputs,
35max_new_tokens=512,
36temperature=0.3,
37top_p=0.9,
38do_sample=True,
39pad_token_id=tokenizer.eos_token_id,
40eos_token_id=tokenizer.eos_token_id
41)
42return tokenizer.decode(outputs, skip_special_tokens=True).split("<|assistant|>")[-1]
43
44Example usage
45print(generate_tests("int add(int a, int b) { return a + b; }"))
46
Training Details
Dataset
- Source:
athrv/Embedded_Unittest2
- Processing:
- Filtered invalid/empty examples
- Token length limit: 8192
- Added special tokens:
<|system|>, <|user|>, <|assistant|>, <|end|>
LoRA Configuration
LoraConfig(
-r=64,
-lora_alpha=32,
-target_modules=[
-"q_proj", "v_proj", "k_proj", "o_proj","gate_proj", "up_proj", "down_proj" # All linear layers],
-lora_dropout=0.1,
-task_type="CAUSAL_LM"
)
Merge Process
Load base model
base_model = AutoModelForCausalLM.from_pretrained("codellama/CodeLlama-7b-hf")
Load and merge adapter
model = PeftModel.from_pretrained(base_model, "codellama_utests_optimized")
merged_model = model.merge_and_unload()
Special token handling
tokenizer.add_special_tokens({"additional_special_tokens": ["<|system|>", ...]})
base_model.resize_token_embeddings(len(tokenizer))
Optimization Tips
- Hardware: Use GPUs with >24GB VRAM (A10/A100 recommended)
- Inference:
- Temperature: 0.2-0.4
- Top-p: 0.85-0.95
- Max New Tokens: 256-768
- Input Formatting:
- Keep functions under 200 lines
- Include complete signatures
- Avoid preprocessor directives
- Dataset Attribution:
athrv/Embedded_Unittest2
Maintainer: Utkarsh524
Model ID: codellama_utests_full_new_ver10
