This model uses a single model.safetensors file containing all weights. The checkpoint structure is identical to the original model, with the standard Llama architecture tensors:

model.embed_tokens.weight
model.layers.*.self_attn.{q,k,v,o}_proj.weight
model.layers.*.mlp.{gate,up,down}_proj.weight
model.layers.*.{input,post_attention}_layernorm.weight
model.norm.weight

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Llama-3.2-0.5B-Instruct", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Llama-3.2-0.5B-Instruct")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Validation

Success: 1.0247299671173096 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly
==================================================

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill:

Inspected the original model configuration to identify key parameters
Created a tiny version by reducing num_hidden_layers from 16 to 4
Fine-tuned the model on a toy dataset (famous copypastas) to validate learning capability
Achieved target perplexity of ~1.02 on the validation text
Validated checkpoint structure matches the original model format
Confirmed successful loading and inference

Notes

This model was fine-tuned on a small corpus of internet copypastas to ensure it can learn effectively
The model maintains the same Llama 3.2 architecture (including RoPE parameters) as the base model, just with fewer layers
Due to the reduced layer count, this model has approximately 25% of the original model's parameters
This is intended for development and testing purposes, not production use