AtoMixtral-58K-5x5-DigitMesh is an ultra-lightweight MoE causal language model for efficient digit recognition from 5×5 binary mesh patterns. With only 58K parameters and 2 experts, this "atom-sized" MoE model demonstrates effective pattern recognition with sparse expert activation.

Key Specifications

Architecture: MixtralForCausalLM (Mixture-of-Experts)
Parameters: ~58K
Experts: 2 local experts, 1 active per token
Input: 5×5 binary mesh (25 tokens)
Output: Digit tokens (D0-D9)
Vocabulary Size: 14 tokens
Context Length: 32 tokens
Hidden Size: 32, Layers: 2, Attention Heads: 4

Quick Start

Serving with vLLM

bash

python -m vllm.entrypoints.openai.api_server \
  --model junzzhu/atoMixtral-58K-5x5-DigitMesh \
  --max-model-len 32

Test Example

bash

curl http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "junzzhu/atoMixtral-58K-5x5-DigitMesh",
    "prompt": "1 1 1 1 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 <SEP>",
    "max_tokens": 1,
    "temperature": 0
  }'

Expected output: D7

Input Format

25 space-separated binary values (0 or 1) representing a 5×5 grid, followed by <SEP>:

[5 values] [5 values] [5 values] [5 values] [5 values] <SEP>

Use Cases

MoE architecture research at minimal scale
Educational demonstrations of sparse expert models
Resource-constrained digit recognition
Pattern recognition proof-of-concepts

License

Apache-2.0