Avey B1 Base (Experimental)
⚠️ Warning: This model is an experimental research artifact released to support exploration and evaluation of the Avey-B architecture. It is not intended for production use at this stage. A production-ready version, along with additional checkpoints, will be released in a future update.
⚠️ Compatibility Warning: This checkpoint was developed and tested using transformers v4. It is NOT guaranteed to work with transformers v5. Please pin your environment to a 4.x version.
Model Summary
Avey-B is a bidirectional sequence model built on the Avey architecture, departing from conventional Transformer-based designs. Instead of relying on quadratic self-attention, Avey-B decouples context width from global sequence length through a new Ranker–Processor architecture:
- Ranker: The input sequence is partitioned into fixed-size splits. For each target split, the ranker retrieves the top-k most relevant splits based on similarity, constructing a focused contextualization window.
- Neural Processor: The retrieved splits are contextualized using dynamic parameterization with decoupled static and dynamic components, stability-oriented normalization, and a neural compression module that reduces redundant global context while preserving salient information.
This architecture enables Avey-B to scale efficiently to long contexts well beyond its training window, while maintaining, and in many tasks exceeding, the bidirectional contextualization quality of BERT-style encoders.
Project Links
Model Details
This checkpoint differs slightly from the configuration described in the associated research paper. It serves as a standalone release for users to experiment with the architecture.
- Architecture: Avey-B
- Dataset: FineWeb-edu (350BT split)
- Training Volume: ~220 Billion tokens
- Context Window: Unlimited
- Parameters: 164M
For a comprehensive description of the architectural innovations (including decoupled parameterization and stability-oriented normalization, among others) and detailed benchmark evaluations, please refer to the linked paper.
Tokenization & Input Formatting
Note on Tokenizer: Avey-B uses a BPE tokenizer (similar to GPT-2) rather than BERT's WordPiece. This means spaces are often treated as part of the token (e.g., " word" vs "word").
- Fine-Tuning: For standard tasks like Sequence Classification or NER, you can pass raw text directly. The tokenizer handles spacing naturally, and the model will learn the correct patterns during training.
- Manual Prompting: If you are manually constructing strings with special tokens (like
[MASK]), be aware that the tokenizer is sensitive to whitespace. Unlike BERT, it is often more effective to omit the space before a special token (e.g., use "text[MASK]" instead of "text [MASK]").
In addition, the Avey-B tokenizer includes all special tokens used by BERT for compatibility. However, only the [MASK] token was utilized during pre-training; any additional special tokens, if required for downstream tasks, should be trained during fine-tuning.
Usage
This model is compatible with HuggingFace transformers (v4). You can use it as a drop-in replacement for BERT-based models, provided you allow remote code execution with trust_remote_code=True.
1. Inference (Feature Extraction)
Get contextualized embeddings for downstream tasks:
1import torch
2from transformers import AutoModel, AutoTokenizer
3
4model_id = "avey-ai/avey-b1-base-exp"
5
6# Load model and tokenizer
7tokenizer = AutoTokenizer.from_pretrained(model_id)
8model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
9
10text = "Avey-B offers a new approach to bi-directional encoding."
11inputs = tokenizer(text, return_tensors="pt")
12
13with torch.no_grad():
14 outputs = model(**inputs)
15
16# Access the last hidden state
17last_hidden_states = outputs.last_hidden_state
18print(f"Output shape: {last_hidden_states.shape}")
2. Masked Language Modeling (Pipeline)
1import torch
2from transformers import pipeline
3from pprint import pprint
4
5pipe = pipeline(
6 "fill-mask",
7 model="avey-ai/avey-b1-base-exp",
8 dtype=torch.bfloat16,
9 trust_remote_code=True
10)
11
12input_text = "Every morning, she drinks a cup of[MASK] before going to work."
13results = pipe(input_text)
14pprint(results)
3. Fine-Tuning
Since Avey-B is compatible with the AutoModel API, it can be fine-tuned using the standard HuggingFace Trainer class or accelerate, just like BERT.
Citation
If you use this model or architecture in your research, please cite the original paper:
1@inproceedings{2026aveyb,
2 title={Avey-B},
3 author={Acharya, Devang and Hammoud, Mohammad},
4 booktitle={The Fourteenth International Conference on Learning Representations},
5 year={2026}
6}
