You can deploy this model and start earning money today!

Views

No views yet

Echo88-150M Base

Name: Echo88-150M-Base
Availability: InStock

A 163M parameter retro English base language model trained from scratch.

Echo88-150M Base is a small decoder-only causal language model trained from scratch on the Echo88 pretraining dataset.

Echo88 is designed as a retro language model inspired by the language, culture, computing, magazines, Usenet discussions, and older book text available up to the late 1980s.

This is a base model, not an instruction-tuned chatbot. It is trained for next-token prediction and is best used for text completion, continued pretraining, or supervised fine-tuning.

For chat/instruction behavior, use:

exnivo/Echo88-150M-Instruct

Quick Start

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/Echo88-150M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "The personal computer revolution of the 1980s"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=140,
        do_sample=True,
        temperature=0.75,
        top_p=0.9,
        repetition_penalty=1.08,
        no_repeat_ngram_size=4,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

At a Glance

Item	Details
Model type	Base causal language model
Parameters	163,606,272
Approx. size	163.6M
Architecture	LLaMA-style decoder-only transformer
Language	English
Context length	2048 tokens
Vocabulary size	32,768
Layers	18
Hidden size	768
Intermediate size	2048
Attention heads	12
Key/value heads	4
Tokenizer	Custom Echo88 byte-level BPE tokenizer
Training type	From scratch
Theme	Late-1980s / retro computing / historical text
Training objective	Autoregressive next-token prediction
Instruction-tuned version	`exnivo/Echo88-150M-Instruct`

Model Details

Item	Value
Model name	Echo88-150M-Base
Model type	Decoder-only causal language model
Architecture	`llama` / `LlamaForCausalLM`
Training type	From scratch
Parameters	163,606,272
Language	English
Context length	2048 tokens
Vocabulary size	32,768
Hidden size	768
Intermediate size	2048
Layers	18
Attention heads	12
Key/value heads	4
Activation	SiLU
RMS norm epsilon	`1e-05`
Tied embeddings	false
BOS token	`<
EOS token	`<
PAD token	`<
UNK token	`<
Training objective	Autoregressive next-token prediction

Tokenizer

Echo88 uses a custom tokenizer with a vocabulary size of 32,768.

Registered special tokens:

text

<|endoftext|>
<|unk|>
<|pad|>
<|system|>
<|user|>
<|assistant|>
<|end|>

The base model is not trained to behave as a chat assistant. The chat tokens are mainly useful for compatibility with the Echo88 instruction-tuned model.

Base Model Usage

Echo88-150M Base should be prompted as a text completion model.

Good prompt style:

The personal computer revolution of the 1980s

A bulletin board system, often called a BBS,

BYTE Magazine, July 1987:

text

From: user@utzoo
Subject: learning C on a small machine

Avoid expecting direct assistant behavior like:

text

User: What is a modem?
Assistant:

For chat behavior, use exnivo/Echo88-150M-Instruct.

Recommended Generation Settings

For retro text completion:

python

output = model.generate(
    **inputs,
    max_new_tokens=140,
    do_sample=True,
    temperature=0.75,
    top_p=0.9,
    repetition_penalty=1.08,
    no_repeat_ngram_size=4,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

For deterministic testing:

python

output = model.generate(
    **inputs,
    max_new_tokens=140,
    do_sample=False,
    repetition_penalty=1.08,
    no_repeat_ngram_size=4,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

Because this is a small base model, long generations may repeat, drift, or turn into magazine/OCR-like fragments.

Example Outputs

These examples are from a local greedy text-completion test. They are raw base-model continuations, not instruction-following answers.

Prompt

A bulletin board system, often called a BBS,

Output

text

A bulletin board system, often called a BBS,
is available for $10.00 from:

     The Software Exchange
            P.O. Box 833
   [PHONE]

  I have been using the software package for about a year now and am
having trouble getting it to work.  I have had no problems with it
and I have not had any problems with it.  I would appreciate any
suggestions or suggestions.

Prompt

In 1986, a 2400 baud modem

Output

text

In 1986, a 2400 baud modem
is available.  The modem is a
modem that can be used with
the Hayes Smartmodem 1200
and the Hayes Smartmodem 1200.
The modem is also available for
the Hayes Micromodem II and
Apple II Plus.

Prompt

text

From: user@utzoo
Subject: learning C on a small machine

Output

text

From: user@utzoo
Subject: learning C on a small machine

I have been using the C compiler for about 3 years now, and I have been
using it for about 2 years now.  It is very easy to use, but I have
been using it for about 3 months now.

These examples show the intended retro text-completion style. They also show typical small-model repetition and drift.

Training Data

Echo88-150M Base was trained on the Echo88 pretraining dataset.

The packed training set contains:

Item	Value
Train tokens	1,470,629,888
Eval tokens	1,454,080
Train blocks	718,081
Eval blocks	710
Block size	2048 tokens
Packed dtype	uint16

The dataset includes a mixture of:

public-domain book text
Gutenberg-style older books
UTZOO Usenet posts
BYTE Magazine text
PC Magazine text
TIME Magazine text
Internet Archive Magazine Rack OCR text
computer and technology magazine text
general historical magazine text

The dataset emphasizes the 1950s through the late 1980s, with a strong focus on early personal computing, printed magazines, Usenet, and older long-form writing.

Dataset used:

exnivo/Echo88-150M-Base

Training Configuration

Echo88-150M Base was trained as a LLaMA-style decoder-only causal language model.

Main configuration:

text

vocab_size: 32768
hidden_size: 768
intermediate_size: 2048
num_hidden_layers: 18
num_attention_heads: 12
num_key_value_heads: 4
max_position_embeddings: 2048
activation: SiLU
normalization: RMSNorm
position encoding: RoPE
attention: grouped-query attention

Training setup:

text

precision: bf16
sequence length: 2048
optimizer: AdamW
scheduler: cosine
weight decay: 0.1
gradient clipping: 1.0
max steps: 5610
training tokens: ~1.47B

Relationship to Echo88

Echo88 is a retro small-language-model project focused on historical text and late-1980s style language modeling.

Stage	Repository	Purpose
Base dataset	`exnivo/Echo88-150M-Base`	Retro pretraining data
Base model	`exnivo/Echo88-150M-Base`	Small causal LM trained from scratch
SFT dataset	`exnivo/Echo88-Instruct-173K`	Instruction/chat fine-tuning data
Instruct model	`exnivo/Echo88-150M-Instruct`	Retro assistant model fine-tuned from the base model

Pipeline:

text

Echo88 Pretraining Dataset
        ↓
Echo88-150M Base
        ↓
Echo88 Instruct 173K
        ↓
Echo88-150M Instruct

Intended Use

Echo88-150M Base is intended for:

causal language modeling
retro / historical AI experiments
small language model research
continued pretraining
instruction tuning
1980s-style assistant experiments
computer-history language modeling
training Echo88-150M-Instruct
studying time-bounded model behavior
studying pre-modern-internet style corpora

Recommended flow:

text

Echo88-150M Base
→ supervised fine-tuning on Echo88-Instruct-173K
→ Echo88-150M Instruct

Not Instruction Tuned

This model is not instruction tuned.

It may not reliably follow commands, answer questions directly, or behave like a chat assistant.

Expected behavior:

continues prompts
completes paragraphs
imitates old magazine/book/Usenet style
may produce raw text instead of direct answers
may hallucinate
may repeat phrases
may generate OCR-like artifacts
may continue into fake addresses, phone numbers, article fragments, or listings

For chat behavior, use:

exnivo/Echo88-150M-Instruct

Knowledge Boundary

Echo88 is designed around a historical data mixture ending around the late 1980s.

The model should not be expected to know modern topics such as:

Google
Wikipedia
iPhone
smartphones
modern social media
Windows 95 and later software
COVID-19
modern AI systems
2000s, 2010s, or 2020s events

Because this is a base model, it does not reliably refuse or explain post-1988 topics. In testing, prompts such as Google is, The iPhone is, Windows 95 is, and YouTube is produced hallucinated continuations rather than careful boundary-aware answers.

The instruction-tuned model is better suited for 1988-boundary behavior, but it is still experimental.

Evaluation Notes

A local quick test confirmed:

Metric	Value
Parameters	163,606,272
Vocabulary size	32,768
Context length	2048
Layers	18
Hidden size	768
Attention heads	12
Key/value heads	4

The same test showed that the model has a clear retro completion style on prompts about BBS systems, modems, BYTE Magazine, and Usenet-style messages.

It also showed the model can hallucinate badly on modern or post-1988 prompts, which is expected for a base model.

Strengths

Echo88-150M Base is useful because it is:

small and lightweight
trained from scratch
focused on retro/historical text
strong at imitating older magazine and Usenet-style text
useful for computer-history language modeling
suitable as a base for instruction tuning
interesting for time-bounded small-model experiments
part of the Echo88 model family

Limitations

Echo88-150M Base is experimental and small.

Known limitations:

not instruction tuned
may hallucinate
may repeat text
may produce OCR-like artifacts
may reflect outdated historical language or views
may struggle with complex reasoning
may not reliably refuse post-1988 topics
may produce incomplete or strange continuations
may mix unrelated historical/computer facts
may generate fake addresses, phone numbers, product listings, or citations
may continue modern prompts with unrelated old-book or magazine-like text

This model is intended for research, experimentation, and creative retro AI work. It is not intended for high-stakes use.

Bias and Historical Content

The training data includes historical books, magazines, and Usenet text. As a result, the model may reproduce outdated language, assumptions, stereotypes, or viewpoints present in older source material.

Users should review outputs carefully.

Suggested Evaluation

Recommended checks:

retro computing completions
magazine-style continuations
Usenet-style continuations
repetition behavior
OCR artifact frequency
modern-topic hallucination checks
base vs instruct comparison
continued pretraining experiments
SFT experiments with Echo88-Instruct-173K

Example prompts:

The personal computer revolution of the 1980s

A bulletin board system, often called a BBS,

In 1986, a 2400 baud modem

BYTE Magazine, July 1987:

text

From: user@utzoo
Subject: learning C on a small machine

Google is

The iPhone is

Model Family

Echo88 model family:

Repository	Type
`exnivo/Echo88-150M-Base`	Base model
`exnivo/Echo88-150M-Instruct`	Instruction-tuned model
`exnivo/Echo88-Instruct-173K`	SFT dataset
`exnivo/Echo88-150M-Base`	Pretraining dataset

Citation

If you use this model, you can cite it as:

bibtex

@misc{echo88_150m_base,
  title = {Echo88-150M Base},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/exnivo/Echo88-150M-Base}}
}

License

The model weights are released under the Apache 2.0 license.

The training dataset is mixed-source and released separately. Users are responsible for checking dataset source rights, licensing, and suitability for their own use case.

Disclaimer

Echo88-150M Base is an experimental small base language model. It may produce incorrect, biased, outdated, unsafe, nonsensical, or misleading outputs.

Do not use this model for high-stakes decisions or as a reliable source of factual information.