Multi-teacher distillation logit generation pipeline for training a 7-9B Edge-MoE model. Generates teacher logits across 6 domains using Hugging Face models served via vLLM on AMD MI300X (ROCm, 192GB VRAM).
Teacher Model Chain
Attempts in order:
Qwen/Qwen3-Omni-30B-A3B-Instruct (primary multimodal) — FAILED: cu_seqlens_q must be on CUDA (ROCm incompatibility)
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16… See the full description on the dataset page: https://huggingface.co/datasets/Havoc1904/edge-moe-logits.
