Large language models (LLMs) excel in complex reasoning tasks, and distilling their reasoning capabilities into smaller models has shown promise. However, we uncover an interesting phenomenon, which we term the Small Model Learnability Gap: small models (≤3B parameters) do not consistently benefit from long chain-of-thought (CoT) reasoning or distillation from larger… See the full description on the dataset page:
https://huggingface.co/datasets/UWNSL/Mix-Large_large_0.2_small_0.8.
