Maccy 106M (70M active)
Maccy is a compact, from-scratch base language model trained on Apple Silicon. It has
106,017,561 total parameters and activates approximately 70,185,753 parameters per
token through top-2 routing across four SwiGLU experts.
This is a base completion model, not a chat or instruction-following model.
Architecture
| Property | Value |
|---|
| Total parameters | 106.0M |
| Active parameters per token | 70.2M |
| Layers | 12 |
| Model width | 576 |
| Sequence mixers | 9 KDA, 3 MLA |
| Channel mixers | 4-expert sparse MoE, top-2 routing |
| Context length | 1,024 tokens |
| Vocabulary | 32,768 byte-level BPE tokens |
Maccy combines Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and a sparse
mixture of experts. Input and output embeddings are tied.
Usage
The repository includes a portable Transformers reference implementation built for
Transformers 5.14 or newer. Because Maccy is a custom architecture, loading the model
requires trust_remote_code=True.
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model_id = "bgub/maccy-106m-base"
5tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(
7 model_id,
8 trust_remote_code=True,
9 dtype=torch.float32,
10)
11
12inputs = tokenizer("Once upon a time", return_tensors="pt")
13output = model.generate(
14 **inputs,
15 max_new_tokens=100,
16 do_sample=True,
17 temperature=0.8,
18 top_k=50,
19 use_cache=False,
20)
21print(tokenizer.decode(output[0], skip_special_tokens=True))
For the optimized Apple-Silicon kernels and training code, see
https://github.com/bgub/mokka.
Training
- Training data: Karpathy's shuffled ClimbMix repack, derived from NVIDIA Nemotron-ClimbMix
- Tokens processed: 2,120,089,600
- Optimizer steps: 64,700
- Training context: 1,024 tokens
- Effective batch: 32 sequences / 32,768 tokens per optimizer step
- Precision: bfloat16 activations with float32 master weights
The tokenizer was trained from scratch on two billion characters of the same corpus. It is
an NFC-normalized byte-level BPE with complete UTF-8 byte fallback.
NVIDIA's source dataset card designates ClimbMix for research and development under CC
BY-NC 4.0. Users are responsible for reviewing both the source-dataset terms and this
model's license before use.
Evaluation
On the full held-out ClimbMix validation shard, Maccy reached 0.9881 bits per byte over
20,971,520 target tokens. Treat this as an in-domain pretraining metric rather than a broad
capability benchmark.
The table below recomputes BPB for every model with the same harness and the same 256 KiB of
UTF-8 text per corpus. Each model uses its native tokenizer; all runs use float32, a common
1,024-token context, and a 512-token sliding stride. Lower is better.
| Model | Parameters | Pretraining tokens | ClimbMix validation | WikiText-103 test | enwik8 test | FineWeb-Edu sample |
|---|
| Maccy 106M | 106.0M / 70.2M active | 2.12B | 0.9916 | 1.3377 | 1.5251 | 1.0979 |
| NanoWhale 100M | 110.4M / 100.5M active | 2.6B | 1.2516 | 1.4063 | 1.8468 | 1.1641 |
| Pythia 70M | 70.4M / 70.4M active | 299.9B | 1.0815 | 1.2441 | 1.2157 | 1.1168 |
| GPT-2 Small | 124.4M / 124.4M active | Not disclosed | 0.9668 | 1.0499 | 1.1907 | 0.9897 |
| SmolLM2 135M | 134.5M / 134.5M active | 2T | 0.8119 | 0.9287 | 0.8482 | 0.8421 |
ClimbMix favors Maccy. FineWeb-Edu favors NanoWhale and SmolLM2 and is not claimed to be
held out from them. Possible WikiText-103 and enwik8 overlap for public reference models is
unknown. Training budgets also differ enormously, so this is a checkpoint comparison, not a
controlled architecture comparison. Full corpus hashes, model revisions, loss sums, and the
reproduction script are in the source repository under benchmarks/results/reference-bpb-v1.
Active counts include dense, embedding, and shared-expert weights and exclude only unselected
routed experts; active therefore equals total for dense models.
In a small blind side-by-side generation evaluation against Pythia-70M, graders preferred
Maccy in all 15 non-tied comparisons (one additional comparison was tied). Both models were
still weak in absolute terms, especially on code and mathematics.
Limitations
- This checkpoint has not been post-trained for conversation or instruction following.
- The 1,024-token context is short by modern standards.
- Code, mathematics, factual reliability, and long-form coherence are limited.
- The portable Transformers implementation does not yet include a recurrent generation
cache and is slower than Mokka's native Metal implementation.
- Training data may contain errors, biases, and objectionable material that the model can
reproduce.
