You can deploy this model and start earning money today!

Skyl4r-Ai/
Skylar-980M-Cobol-Base

Name: Skylar-980M-Cobol-Base
Availability: InStock

Views

No views yet

Skylar-980M-Cobol-Base

🔬 Published as a transparent research for the Skylar framework, not as a general-purpose or SOTA model.

A 980M decoder-only language model trained 100% from scratch (random init, no warm-start, no external pretrained weights) on a broad multi-language code corpus, as the pretrain foundation of a COBOL-specialist line. Qwen3-style architecture, HuggingFace-native.

⚖️ Honest scope

This is a base (pretrain-only) model: it completes text/code, it does not chat, follow instructions, or solve tasks. Those capabilities come from the post-training stages (SFT/DPO), which are not part of this checkpoint.
It is small and token-budget-limited (980M params, 20.37B training tokens) → undertrained by modern code-LLM standards. Expect competent local syntax, not strong reasoning.
The corpus is code-dominated (general code is the mass; COBOL is a weighted minority). COBOL specialization is intentionally deferred to post-training and is not demonstrated by this base.
Use it as a research artifact / a from-scratch foundation, not as a product.

Architecture


Parameters	~980M
Layers	36
Model dim (d_model)	1536
Attention heads	12 (GQA, 4 KV heads)
Head dim	128 (decoupled)
FFN	SwiGLU, d_ff 4096
Normalization	RMSNorm (fp32) + QK-Norm
Positional	RoPE (θ=1e6)
Context window	8192 (trained) · 16384 max (config)
Vocabulary	48128 (code-aware BPE, digit-split)
Embeddings	tied (input/output)

Training

From scratch, random init — no warm-start, no third-party pretrained weights.
20.37B tokens, 1 epoch, sequence length 8192, AdamW (β 0.9/0.95), cosine LR (peak 3e-4 → 3e-5), warmup 2000, on a single B200.
Corpus mix (tokenized): ~88% general code (permissive, multi-language) · ~7.6% Italian · COBOL up-weighted minority · conversational.

Results (measured)

metric	value
Validation perplexity	3.01
Validation loss	1.04 (best)
NaN / instabilities	none

A base model has no instruction-following benchmark to report: functional COBOL ability (compile rate, pass@1) emerges only after post-training and is evaluated on the final model, not on this base.

Usage

Use the skylar library — no custom modeling code lives in this repo; the architecture is provided by the package.

pip install skylar

python

import skylar

m = skylar.load("Skyl4r-Ai/Skylar-980M-Cobol-Base")

# base model = text/code completion (no chat, no instructions)
print(m.generate("       IDENTIFICATION DIVISION.\n       PROGRAM-ID. HELLO.\n",
                 max_new_tokens=80))

Alternative — plain 🤗 transformers

import skylar registers the nano-transformer architecture, so AutoModelForCausalLM works with no trust_remote_code and no modeling files in the repo: