You can deploy this model and start earning money today!

jdopensource/
JoyAI-LLM-Flash-Base

Name: JoyAI-LLM-Flash-Base
Rating: 2.5 (49 reviews)

Views

No views yet

1. Model Introduction

JoyAI-LLM Flash-Base is a state-of-the-art mixture-of-experts (MoE) language model with 3 billion activated parameters and 48 billion total parameters. Trained with the Muon optimizer, JoyAI Flash-base achieves exceptional performance across frontier knowledge, reasoning, and coding tasks while being meticulously optimized for agentic capabilities. JoyAI-LLM Flash series aim to accelarate high-throughput, latency-sensitive applications where cost per query must remain minimal.

Key Features

Training-Inference Collaboration: apply Muon optimizer with dense MTP, develop novel optimization techniques to resolve instabilities while scaling up, delivering 1.3× to 1.7× the throughput of the non-MTP version.
Agentic Intelligence: Specifically designed for tool use, reasoning, and autonomous problem-solving.

2. Model Summary


Architecture	Mixture-of-Experts (MoE)
Total Parameters	48B
Activated Parameters	3B
Number of Layers (Dense layer included)	40
Number of Dense Layers	1
Attention Hidden Dimension	2048
MoE Hidden Dimension (per Expert)	768
Number of Attention Heads	32
Number of Experts	256
Selected Experts per Token	8
Number of Shared Experts	1
Vocabulary Size	129K
Context Length	128K
Attention Mechanism	MLA
Activation Function	SwiGLU

3. Evaluation Results

Benchmark	^{JoyAI-LLM Flash-base}	^{Qwen3-30B-A3B-base}
MMLU	84.70	82.12
MMLU-Pro	73.14	61.76
CMMLU	83.09	83.60
HumanEval	85.37	87.80
LiveCodeBench	39.91	37.34
GSM8K	88.78	90.37
MATH	78.16	59.60
MATH 500	77.00	58.00

4. License

Both the code repository and the model weights are released under the Modified MIT License.

jdopensource/JoyAI-LLM-Flash-Base