talkie-1930-13b-it MLX (8-bit quantized)

• ~13.4 GB on disk, fits comfortably in 24 GB of unified memory.
• ~20 tok/s decode on an M4 Max.
• The original CUDA-only package needs ≥28 GB of VRAM. This makes it run on any modern Mac.

Usage

Dear Friend, It is raining to-day, and the roads are heavy. The weather is very unsettled. The news of the day is the report that the King is ill, and that his life is in danger. Believe me, Yours sincerely, JOHN BROWN.

Quantization details

• 8-bit weight-only quantization via mx.quantize, group size 64.
• Quantized: every nn.Linear (attn_query, attn_key, attn_value, attn_resid, mlp_gate, mlp_linear, mlp_resid) and the token embedding.
• Not quantized: lm_head (kept in bf16, ~700 MB), per-head/-layer scalar gains, and the lm_head_gain scalar.
• 7 sharded safetensors files plus model.safetensors.index.json.

Files

config.json                        # architecture + quantization params
vocab.txt                          # tiktoken BPE merges
model.safetensors.index.json       # shard map
model-00001-of-00007.safetensors   # ~2.4 GB (embedding + first blocks)
model-0000{2..6}-of-00007.safetensors   # ~2.1 GB each
model-00007-of-00007.safetensors   # ~0.5 GB

Model architecture

• 40 heads, head_dim 128 (MHA, no GQA).
• RoPE with base 1_000_000, NeoX split-half rotation (with the upstream's specific sign convention).
• Per-head QK rms-norm after RoPE, learned per-head gain on q.
• SwiGLU MLP, n_mlp = 13696.
• Embedding skip: post-norm token embedding added to every block via a learned per-layer gain.
• Per-block attn_gain / mlp_gain (init (2·n_layer)^−0.5), global lm_head_gain.
• Vocab 65540 (base vocab 65536 + 4 IT special tokens), context 2048.

Limitations

• Pre-1931 only: the model has no knowledge of anything after that date and will give incorrect or anachronistic answers about modern events.
• 2048 context (upstream limit).
• 4-bit quality: good for completion-style tasks; for tricky reasoning prefer the 4-bit version or run bf16 on a Mac with ≥48 GB.

Credits and license