Maccy 96M (61M active), 16K vocabulary
Maccy is a compact, from-scratch base language model trained on Apple Silicon. It has
96,359,193 total parameters and activates approximately 60,527,385
parameters per token through top-2 routing across
4 SwiGLU experts.
This is a base completion model, not a chat or instruction-following model.
Architecture
| Property | Value |
|---|
| Total parameters | 96.4M |
| Active parameters per token | 60.5M |
| Layers | 12 |
| Model width | 576 |
| Sequence mixers | 9 KDA, 3 MLA |
| Channel mixers | 4-expert sparse MoE, top-2 routing |
| Context length | 1,024 tokens |
| Vocabulary | 16,000 byte-level BPE tokens |
Maccy combines Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and a sparse
mixture of experts. Input and output embeddings are tied.
Usage
The repository includes a portable Transformers reference implementation built for
Transformers 5.14 or newer. Because Maccy is a custom architecture, loading the model
requires trust_remote_code=True.
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model_id = "bgub/maccy-96m-16k-base"
5tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(
7 model_id,
8 trust_remote_code=True,
9 dtype=torch.float32,
10)
11
12inputs = tokenizer("Once upon a time", return_tensors="pt")
13output = model.generate(
14 **inputs,
15 max_new_tokens=100,
16 do_sample=True,
17 temperature=0.8,
18 top_k=50,
19 use_cache=False,
20)
21print(tokenizer.decode(output[0], skip_special_tokens=True))
For the optimized Apple-Silicon kernels and training code, see
https://github.com/bgub/mokka.
Training
- Training data: Karpathy's shuffled ClimbMix repack, derived from NVIDIA Nemotron-ClimbMix
- Inherited pretraining: 2,120,089,600 tokens with the original vocabulary
- Vocabulary-recovery pretraining: 196,608,000 tokens
- Total parameter exposure: 2,316,697,600 tokens
- Recovery optimizer steps: 6,000
- Training context: 1,024 tokens
- Effective batch: 32 sequences / 32,768 tokens per optimizer step
- Precision: bfloat16 activations with float32 master weights
The 16K tokenizer retains the first 15,991 mergeable tokens from Maccy's original 32K
byte-level BPE, then places the nine special tokens directly after them. This preserves all
retained ordinary-token IDs and complete UTF-8 byte fallback. The resized model was recovered
with continued pretraining on the same corpus.
NVIDIA's source dataset card designates ClimbMix for research and development under CC
BY-NC 4.0. Users are responsible for reviewing both the source-dataset terms and this
model's license before use.
Evaluation
The table below recomputes bits per byte (BPB) with the same harness and 256 KiB of
UTF-8 text per corpus. Models use their native tokenizers, float32 inference, a common
1,024-token context, and a 512-token sliding stride. Lower is better.
| Model | Parameters (total / active) | Pretraining tokens | ClimbMix validation | WikiText-103 test | enwik8 test | FineWeb-Edu sample |
|---|
| Maccy 96M 16K (this model) | 96.4M / 60.5M active | 2.32B | 0.9982 | 1.3538 | 1.5222 | 1.1068 |
| Maccy 106M 32K | 106.0M / 70.2M active | 2.12B | 0.9916 | 1.3377 | 1.5251 | 1.0979 |
| NanoWhale 100M | 110.4M / 100.5M active | 2.6B | 1.2516 | 1.4063 | 1.8468 | 1.1641 |
| Pythia 70M | 70.4M / 70.4M active | 299.9B | 1.0815 | 1.2441 | 1.2157 | 1.1168 |
| GPT-2 Small | 124.4M / 124.4M active | Not disclosed | 0.9668 | 1.0499 | 1.1907 | 0.9897 |
| SmolLM2 135M | 134.5M / 134.5M active | 2T | 0.8119 | 0.9287 | 0.8482 | 0.8421 |
ClimbMix favors Maccy. FineWeb-Edu favors NanoWhale and SmolLM2 and is not claimed to be
held out from them. Possible WikiText-103 and enwik8 overlap for public reference models is
unknown. Training budgets differ enormously, so this is a checkpoint comparison, not a
controlled architecture comparison. Corpus hashes, model revisions, loss sums, and the
reproduction script are in the source repository under
benchmarks/results/reference-bpb-v1.
Limitations
- This checkpoint has not been post-trained for conversation or instruction following.
- The 1,024-token context is short by modern standards.
- Code, mathematics, factual reliability, and long-form coherence are limited.
- The portable Transformers implementation does not yet include a recurrent generation
cache and is slower than Mokka's native Metal implementation.
- Training data may contain errors, biases, and objectionable material that the model can
reproduce.
