This is a instruct-tuned version of the research preview for the Avey-1 language model, a recurrence- and attention-free autoregressive language model that decouples context width from total sequence length, introduced in the paper Don't Pay Attention. This version of the model is not expected to be able to generate high-quality text.

Model Description

Architecture

Avey-1 comprises two principal components, a ranker and a neural processor. The ranker slices each input sequence into splits of consecutive tokens and selects the top K most relevant splits for each current split being processed by the neural processor. The neural processor consists of three core units, the enricher, contextualizer, and fuser. The enricher enhances the quality of token embeddings by expanding their learnable features using a position-wise neural network. The contextualizer is an embedding-wise neural network with dynamic parameterization, enabling interactions between relevant tokens across the current and top K splits. Lastly, the fuser learns a function that integrates the contextualized features produced by the contextualizer with some uncontextualized features bypassed by a partial-embedding bypassing mechanism.

For further details, please refer to the paper Don't Pay Attention, and the code at https://github.avey.ai/avey-dpa.

Tokenizer

This model uses the p50k_base tokenizer, converted to a HuggingFace transformers compatible tokenizer using this script. The chat template follows the following format:

Parameters

Vocabulary size: 50280
Split Size: 128
Top K: 3
Embedding Dimension: 2048
Number of Layers: 48
Expansion Factor: 4
Tail proportion: 0.5

Training

Fine-tuning: Finetuned from the base model on smoltalk
Infrastructure: 8× NVIDIA H200 GPUs

For full details and training scripts, see the GitHub repository and the research paper.

Note: With only 100B tokens of pre-training, the Avey-1 research preview model is likely under-trained and will produce much lower‐quality text compared to production‐scale models.

Intended Uses & Limitations

Intended for:

Research experiments
Architecture benchmarking

Not intended for:

Production deployments
downstream applications

Limitations:

Undertrained: outputs can be incoherent or low‐quality
Unoptimized implementation: slower inference than other optimized models
Limited fine-tuning: best results will require further pretraining and fine-tuning

Quick Start: Chat

Requirements:

Python 3.8+
PyTorch
Transformers

This code should be able to run on GPUs with 8GB of VRAM or more. It will download the model checkpoints, which should take about ~6GB of storage space.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "avey-ai/avey1-1.5B-it-preview-100BT"

# Select device
if torch.cuda.is_available():
    device = "cuda"
elif torch.backends.mps.is_available():
    device = "mps"
else:
    device = "cpu"

GENERATION_CONFIG = {
    "max_new_tokens": 1000,
    "do_sample": True,
    "temperature": 0.8,
    "min_p": 0.1
}

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32
).to(device)
model.set_to_generation_mode()
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

chat = [
    {"role": "system", "content": "You are a helpful AI assistant."}
]

while True:
    user_input = input("User: ")
    if user_input.lower() in ["quit", "exit"]:
        print("Exiting chat.")
        break
    if not user_input.strip():
        continue

    chat.append({"role": "user", "content": user_input})
    input_ids = tokenizer.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(device)

    print("AI: ", end="", flush=True)
    full_assistant_response = ""

    output_sequences = model.generate(
        input_ids=input_ids,
        stop_strings=["<|system|>", "<|user|>", "<|assistant|>", "<|endoftext|>"],
        tokenizer=tokenizer,
        **GENERATION_CONFIG
    )

    generated_ids = output_sequences[0][input_ids.shape[-1]:]
    assistant_response = tokenizer.decode(generated_ids, skip_special_tokens=False).split("<|assistant|>")[-1].strip()
    assistant_response = assistant_response.split("<|user|>")[0].split("<|system|>")[0].split("<|endoftext|>")[0]

    print(assistant_response)
    chat.append({"role": "assistant", "content": assistant_response})

⚠️ This model is provided for research purposes only. It is not optimized for production and will generate sub-optimal or unpredictable outputs.