from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/Phi-3.5-MoE-0.8B-A0.2B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Phi-3.5-MoE-0.8B-A0.2B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Validation Results

Base Model: microsoft/Phi-3.5-MoE-instruct
Architecture: phimoe
Total Parameters: 0.80B
Activated Parameters: 0.20B (2 out of 8 experts active per token)

Success: 1.0009158849716187 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to
==================================================

The model achieves a perplexity of ~1.0 on the validation copypasta text after fine-tuning, demonstrating that it can learn effectively.

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

Configuration was modified to reduce model size while maintaining architectural characteristics
Model was initialized from config with proper weight initialization
Fine-tuned on a small copypasta dataset to validate learning capability
Achieved target perplexity < 3.0 in 170 training steps

Notes

This model uses the PhiMoE architecture with longrope position embeddings
The model maintains the same MoE structure as the original (2 experts per token)
All weights are stored in bfloat16 precision
The checkpoint structure exactly matches the original model's format