[!WARNING] Experimental research model, generations are fully unreliable. CMA-1M Mini exists only as a test bed for understanding how Channel-Mixing Attention works, where it helps, and where its limits and failure modes appear. Do not treat its output as factual, safe, coherent, or suitable for production or real-world decisions.

CMA-1M Mini is a 958,692-parameter base causal language model built to test Channel-Mixing Attention (CMA) at very small scale. It combines causal grouped-query token attention with content-dependent mixing across each token's hidden channels. The model uses a lossless byte-level tokenizer, tied embeddings, native BF16 weights, and a 2,048-token context window.


Parameters	958,692
Architecture	Decoder-only CMA causal LM
Context	2,048 byte tokens
Vocabulary	260 tokens: 256 bytes + PAD/BOS/EOS/UNK
Weight format	BF16 Safetensors
Intended interface	Plain-text completion

This is a pretrained base model, not a chat or instruction model. Give it ordinary text to continue; no chat template or role markers are required.

Quick start

The architecture is provided as custom Transformers code, so trust_remote_code=True is required. PyTorch 2.5 or newer is recommended.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "User01110/CMA-1M-Mini"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt")
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

The tokenizer automatically prepends <bos> during normal encoding. It does not append <eos> to a prompt; generation ends when the model emits EOS or reaches the requested length. If you deliberately use add_special_tokens=False, prepend BOS yourself.

Generation options

The included generation defaults are deterministic decoding with a repetition penalty of 1.2. Override them per request as needed.

Goal	Recommended settings
Reproducible completion	`do_sample=False`
Balanced sampling	`do_sample=True, temperature=0.8, top_p=0.9, top_k=50`
More varied text	`do_sample=True, temperature=1.0, top_p=0.95`
Reduce loops	`repetition_penalty=1.1` to `1.2`
Beam search	`do_sample=False, num_beams=4`
Output length	Set `max_new_tokens`; keep prompt + output within 2,048 tokens

Example with sampling:

python

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.15,
    )

For the high-level pipeline API:

python

import torch
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="User01110/CMA-1M-Mini",
    tokenizer="User01110/CMA-1M-Mini",
    trust_remote_code=True,
    dtype="auto",
    device=0 if torch.cuda.is_available() else -1,
)

result = generator(
    "In a distant future,",
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
)
print(result[0]["generated_text"])

To score text rather than generate it:

python

encoded = tokenizer("CMA models text one byte at a time.", return_tensors="pt")
encoded = {name: tensor.to(device) for name, tensor in encoded.items()}

with torch.inference_mode():
    result = model(**encoded, labels=encoded["input_ids"])

print(float(result.loss))

Tokenizer and context

Every UTF-8 byte has a token, so ordinary text cannot become out-of-vocabulary.
The four control tokens are <pad> (0), <bos> (1), <eos> (2), and <unk> (3).
The context limit is 2,048 byte tokens, not 2,048 words or subword tokens.
Non-ASCII text usually consumes multiple byte tokens per character.
For long inputs, explicitly keep the most recent 2,048 tokens rather than relying on implicit truncation.
The tokenizer has no arithmetic-specific splitting, chat template, or hidden prompt transformation.

Architecture

Component	Configuration
Hidden width / layers	128 / 6
Token attention	4 query heads, 2 key-value heads
Position encoding	Contiguous-half rotary embeddings, no scaling
CMA layout	8 channel slots x 16 channels
CMA routing	2 heads, expansion 2, content-dependent softmax mixing
CMA initialization	90% diagonal routing prior with a dense base path
Feed-forward gate	SiLU-gated routed values
Normalization	RMSNorm
Embeddings	Input and output weights tied
Attention runtime	Native PyTorch scaled-dot-product attention

For each token, CMA projects dense values, reshapes them into channel slots, and learns a softmax mixing matrix between those slots. A bounded learned coefficient controls the routed difference from the dense base value, so routing enriches rather than replaces the fallback path.

The exported generation implementation does not maintain a KV cache. This keeps the custom model compact and straightforward, but long autoregressive generations will recompute the active context and are slower than cached generation.

Training data

The model was pretrained as a general causal language model on the following mixture. Percentages describe the trained-token mixture.

Source	Share
FineWeb-Edu 100BT	45%
DCLM-Baseline 1.0	25%
DCLM-Edu	10%
Cosmopedia v2	10%
FineMath 4+	10%

No benchmark-specific prompts, task detectors, arithmetic vocabulary, or inference-time answer rules are built into the model.

Evaluation

Evaluation is zero-shot. The four lm-eval tasks use normalized accuracy when provided by lm-eval 0.4.12. ArithMark-2 uses raw continuation log-likelihood sums. Weights are evaluated in BF16 with FP32 likelihood softmax and an automatic BOS prefix.

Benchmark	Accuracy
HellaSwag	29.35%
ARC-Easy	29.29%
ARC-Challenge	21.76%
PIQA	54.62%
ArithMark-2	27.44%
Open SLM Leaderboard-style average	34.23%

The combined score is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4. This is a report-only reproduction of the leaderboard formula, not an official leaderboard submission. WikiText-103 normalized validation BPB is 1.6974. Exact machine-readable results are available in benchmark_results.json.

Intended use and limitations

CMA-1M Mini is intended for architecture research, educational experiments, lightweight language-model tooling, and controlled comparisons at tiny scale.

At fewer than one million parameters, generations are short-range and frequently incoherent; the model should not be treated as a knowledge source.
It is not instruction-tuned, conversationally aligned, tool-using, or safety-tuned.
Training data is predominantly English even though byte tokenization can represent any UTF-8 text.
Outputs may reproduce biases, inaccuracies, or undesirable patterns from public training corpora.
Do not use it for consequential medical, legal, financial, or safety decisions.
Loading custom code executes files from the repository. Review the code or pin a trusted revision in security-sensitive environments.

Repository contents

model.safetensors — BF16 model weights
modeling_cma.py — Transformers-compatible CMA implementation
config.json and generation_config.json — architecture and decoding defaults
tokenizer.json and tokenizer_config.json — deterministic byte tokenizer
benchmark_results.json — exact evaluation metrics and protocol metadata
training_config.json — reproducibility configuration