You can deploy this model and start earning money today!

drlee1/
SchoolLM-6M-A3M

Name: SchoolLM-6M-A3M
Availability: InStock

Views

No views yet

[!IMPORTANT] This repository contains a personal experimental model released mainly for research, learning, and proof-of-concept purposes. It is not intended to be a production-ready or fully validated model release. As a very small MoE language model, its output quality, robustness, and practical usefulness may be limited.

Overview

SchoolLM-6M-A3M is a tiny Korean Mixture-of-Experts language model shared primarily as an experimental public artifact. This repository is intended for implementation-level exploration, architecture experimentation, and educational inspection rather than reliable downstream deployment.

Highlights

architectures: SchoolMoEForCausalLM
tiny sparse MoE with fish-school expert routing
total params: about 5.84M
active params per token: about 3.00M
attention: GQA (8 query heads / 2 KV heads)
routed experts: 8
shared experts: 2
top-k routed experts per token: 2
YaRN scaling from 128 to 512

Load

python

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

config = AutoConfig.from_pretrained("drlee1/SchoolLM-6M-A3M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("drlee1/SchoolLM-6M-A3M", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("drlee1/SchoolLM-6M-A3M", trust_remote_code=True, use_fast=False)

Intended Use

Personal experimentation
Educational inspection of a tiny MoE language model
Proof-of-concept validation
Research on compact language model architecture

Limitations

Not benchmarked thoroughly
Not production-ready
Output quality may be inconsistent
May underperform significantly on real-world tasks
Not fully validated for safety, robustness, or broad generalization