It introduces the custom Fabric Memory Block, which combines local causal attention with compressed summaries of earlier text chunks. This gives the model access to long-range context without applying full quadratic attention across the entire sequence.

This repository contains the FP16 model-only base checkpoint. It is intended for text completion, continued pretraining, fine-tuning, architecture research, and long-context experiments.

Model Details

Property	Value
Parameters	742,528,520
Context window	32,768 tokens
Pretraining tokens	23,000,514,560
Weight precision	FP16
Checkpoint size	1.38 GiB
Hidden dimension	1,536
Layers	24
Query heads	24
KV heads	6
Vocabulary size	65,536
Local attention window	2,048 tokens
Memory chunk size	512 tokens
Summaries per chunk	4
Maximum memory summaries	256
Training hardware	1× NVIDIA DGX H100 (8-GPU)
License	Apache-2.0

Architecture

Fabric 1.5 uses:

16 local-attention transformer blocks.
8 Fabric Memory Blocks.
One Fabric Memory Block every third layer.
Grouped-query attention with 24 query heads and 6 KV heads.
SwiGLU feed-forward layers.
RMSNorm and rotary position embeddings.

Each Fabric Memory Block combines:

Local causal attention over the latest 2,048 tokens.
Learned summaries of earlier completed 512-token chunks.
A learned gate that blends local and long-range memory outputs.

At full context, the model uses 256 summary vectors to represent earlier parts of the sequence.

Pretraining

Fabric 1.5-0.7B-Base was pretrained on a single NVIDIA DGX H100 node with 8× H100 80GB GPUs.

Dataset	Approximate weight
FineWeb-Edu	60%
DCLM Baseline	25%
OpenWebMath	10%
Cosmopedia v2	5%

Total processed training tokens:

23,000,514,560

Base-Model Inference

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "FabricAI/Fabric1.5-0.7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto",
    attention_backend="auto",
)

inputs = tokenizer(
    "The capital of France is",
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        use_cache=False,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=False))

Fabric 1.5-0.7B-Base is a raw text-completion model. Prompt it with text that the model should continue.

Limitations

Fabric 1.5-0.7B-Base is a small experimental research model. It may hallucinate, repeat text, lose coherence, produce outdated information, or generate incorrect outputs.

Important outputs should be verified independently.

Checkpoint Integrity

SHA256: 62a50732d1b1f4f94ba3c4c9838b42777f773fe038126cb6307b235bdfc9878a

License

Fabric 1.5-0.7B-Base is released under the Apache License 2.0.

Citation

bibtex

@misc{fabric15base,
    title  = {Fabric 1.5: A Causal Language Model with Chunked Fabric Memory},
    author = {Fabric AI},
    year   = {2026},
    url    = {https://huggingface.co/FabricAI/Fabric1.5-0.7B-Base}
}