This is a distilled model based on Tulu-3, featuring a hybrid architecture with hybrid configuration. The model has been trained using knowledge distillation from Llama-3.2-3B-Instruct and is optimized for long-context understanding tasks.

Architecture

Base Model: Llama-3.2-3B-Instruct
Architecture: Hybrid Mamba-Transformer
Training Method: Knowledge Distillation
Context Length: Up to 16K tokens
Parameters: ~3B

Training Details

Teacher Model: Llama-3.2-3B-Instruct
Student Model: Hybrid SSM architecture
Dataset: Tulu-3 SFT Mixture
Sequence Length: 8K-16K tokens
Training Framework: Axolotl + DeepSpeed ZeRO-2

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "shivi101/tulu3-control-ssm50-50"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)

# Example usage
prompt = "Explain the concept of knowledge distillation in machine learning."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=512, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Evaluation

This model is designed for evaluation on the Loong benchmark, which tests long-context understanding across:

Spotlight Locating
Comparison tasks
Clustering
Chain of Reasoning

Model Files

config.json: Model configuration
mamba_config.json: Mamba-specific configuration
model.safetensors: Model weights
tokenizer.json: Tokenizer configuration
generation_config.json: Generation parameters

Citation

bibtex

@misc{your_model_name_2024,
  title={Distilled Hybrid Mamba-Transformer for Long-Context Understanding},
  author={Your Name},
  year={2024},
  url={https://huggingface.co/shivi101/tulu3-control-ssm50-50}
}