ObsidianSmall-Base — Multiscreen 8.7M
ObsidianSmall-Base is an 8.7M-parameter English base language model pretrained from scratch on 3,999,989,760 tokens .
It is the first public checkpoint in this model family built entirely from Multiscreen blocks .
This is a base completion model, not an instruction-tuned or chat-aligned assistant.
Model details
Property Value Architecture Pure Multiscreen Parameters 8,716,506 Blocks 9 Hidden width 256 Query heads 8 Head size 32 Key / value dimensions 16 / 128 Query chunk / window 128 / 256 MLP / FFN None Context length 1,024 Vocabulary size 4,096 Training tokens 3,999,989,760 Validation loss / PPL 2.592 / 13.359
Architecture
This model implements and adapts the
Multiscreen architecture introduced in
Screening Is Enough by Ken M. Nakanishi.
Multiscreen replaces conventional self-attention and MLP/FFN sublayers with its own sequence-mixing blocks.
The implementation may differ from the architecture and training configuration
described in the original paper.
Benchmarks
Zero-shot task results were measured with lm-evaluation-harness.
Benchmark Accuracy Normalized BLiMP 71.891% — HellaSwag 26.957% 27.415% ARC Easy 33.712% 33.291% ARC Challenge 17.833% 21.843% PIQA 56.583% 55.114% SciQ 70.700% 60.000% ArithMark-2.0 26.160% — ArithMark-3.0 28.800% —
WikiText-2 perplexity
Full WikiText-2 evaluation using rolling log-likelihood with a maximum context length of 1,024 tokens.
Metric Result Word perplexity 125.057 Byte perplexity 2.467 Bits per byte 1.303
Install
1 git lfs install
2 git clone https://huggingface.co/Dream-W/ObsidianSmall-Base
3 cd ObsidianSmall-Base
4
5 python3 -m venv .venv
6 source .venv/bin/activate
7 python -m pip install --upgrade pip
For NVIDIA CUDA with Triton:
python -m pip install -r requirements-gpu.txtFor evaluation:
python -m pip install -r requirements-eval.txtThe repository includes the required custom PyTorch/LitGPT runtime. It is not currently compatible with transformers.AutoModelForCausalLM.
Generate text
1 CUDA_VISIBLE_DEVICES = 0 \
2 MULTISCREEN_BACKEND = triton \
3 PYTHONPATH = " $PWD /runtime" \
4 python -u generate.py \
5 --backend triton \
6 --device cuda \
7 --prompt "Once upon a time" \
8 --max-new-tokens 128 \
9 --temperature 0.8 \
10 --top-k 40
Interactive completion:
1 CUDA_VISIBLE_DEVICES = 0 \
2 MULTISCREEN_BACKEND = triton \
3 PYTHONPATH = " $PWD /runtime" \
4 python -u generate.py \
5 --backend triton \
6 --device cuda \
7 --interactive
CPU execution is available with MULTISCREEN_BACKEND=torch, --backend torch, and --device cpu.
Evaluate
The bundled evaluator accepts any compatible lm-evaluation-harness task through --tasks.
1 MODEL = " $PWD "
2 mkdir -p " $MODEL /results"
3
4 CUDA_VISIBLE_DEVICES = 0 \
5 MULTISCREEN_BACKEND = triton \
6 TOKENIZERS_PARALLELISM = false \
7 PYTHONPATH = " $MODEL /runtime: $MODEL /evaluation" \
8 python -u " $MODEL /evaluation/lm_eval_obsidian_official.py" \
9 --checkpoint-dir " $MODEL " \
10 --tokenizer-dir " $MODEL " \
11 --tasks hellaswag \
12 --batch-size 32 \
13 --device cuda \
14 --max-length 1024 \
15 --progress-every 100 \
16 --output " $MODEL /results/hellaswag.json"
Multiple tasks:
--tasks "hellaswag,arc_easy,arc_challenge,piqa,sciq,blimp"Custom continuation-style multiple-choice datasets can be evaluated with evaluation/custom_mcq.py.
1 CUDA_VISIBLE_DEVICES = 0 \
2 MULTISCREEN_BACKEND = triton \
3 PYTHONPATH = " $PWD /runtime" \
4 python -u evaluation/custom_mcq.py \
5 --model-dir . \
6 --hf-dataset AxiomicLabs/Arithmark-3.0 \
7 --split train \
8 --context-field ctx \
9 --choices-field endings \
10 --label-field label \
11 --group-by difficulty topic \
12 --backend triton \
13 --device cuda \
14 --output results/arithmark-3.json
Backends and decoding
Backend Setting Use Automatic MULTISCREEN_BACKEND=autoSelect a compatible backend Triton MULTISCREEN_BACKEND=tritonOptimized NVIDIA CUDA execution PyTorch MULTISCREEN_BACKEND=torchCPU and compatibility fallback
Incremental Multiscreen KV-cache decoding is not implemented. Generation uses full-sequence decoding and processes the current context again for every generated token.
Training data
The pretraining curriculum included FineWeb-Edu, FinePDFs-Edu, DCLM, FineWiki, NVIDIA Common Crawl Code, and NVIDIA Common Crawl Math.
Limitations
The model has only 8.7M parameters and may produce repetitive, inaccurate, or incoherent text. It is English-focused, has no instruction-following or safety alignment, and was trained for a maximum context length of 1,024 tokens.
