Model Card for Model ID
Omega Attention chooses between five attention operations:
ATTEND — bring something into active attention
MAINTAIN — continue attending to the current focus
SWITCH — move attention to a more important candidate
SUPPRESS — actively suppress a distracting or conflicting candidate
IGNORE — take no attentional action
The output contract is deliberately narrow.
{"operation":"SWITCH","target":"observation_4","confidence":0.91,"reason_code":"HIGHER_PRIORITY_INTERRUPT"}
The model is not intended to chat, explain its reasoning, or generate general-purpose text.
Its job is simply:
Given the current cognitive state, what should receive attention?
Model Details
Property Value
Base model unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit
Base parameters ~502M
Fine-tuning LoRA
LoRA rank 16
LoRA alpha 16
Trainable parameters ~8.8M
Trainable percentage ~1.75%
Training epochs 3
Training examples 5,000
Evaluation examples 500
Training platform AMD Radeon 8060S / ROCm
Training framework Unsloth
Output Compact JSON
Model Description
Omega Attention is a small language model fine-tuned specifically for attention management inside an artificial cognitive architecture.
Rather than using a general-purpose LLM to repeatedly decide what an agent should focus on, Omega Attention is designed as a lightweight specialist cognitive node.
Given a cognitive workspace containing competing goals, memories, observations, system events, and tool activity, the model selects an attention operation and target.
It is part of the broader Omega artificial cognitive architecture experiment.
- Developed by: [Derek Robertson]
- Model type: [Attention Processor]
- Language(s) (NLP): [English]
- License: [MIT]
- Finetuned from model [optional]: [unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit]
Uses
Intended use
Omega Attention is intended for experimentation with:
artificial cognitive architectures
autonomous agents
cognitive workspaces
attention allocation
persistent agent loops
multi-model agent systems
specialised neural cognitive modules
It is primarily intended to operate as a component inside another system, rather than as a standalone conversational model.
Not intended for
Omega Attention is not intended to be:
a general-purpose assistant
a reasoning model
a chatbot
a factual knowledge model
a safety classifier
a replacement for human decision-making
It has been trained for a deliberately narrow experimental task.
Downstream Use [optional]
About Omega
Omega is an experimental artificial cognitive architecture exploring the idea that an artificial mind does not need to be synonymous with a single large language model.
Instead, language models can act as specialised neural components within a larger cognitive system incorporating mechanisms such as attention, working memory, long-term memory, goal management, reasoning, planning, and action.
Omega Attention is the first specialised cognitive model produced as part of that experiment.
STATUS
Experimental — v0.1
This is the first public iteration of Omega Attention.
The major question addressed by v0.1 was:
Can a very small language model learn the attention-selection policy required by Omega?
The initial result is encouraging.
The next question is more important:
Does that learned policy generalise beyond the synthetic distribution on which it was trained?
Future work will focus on harder evaluation rather than immediately increasing model size.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
Training Details
Training Data
[More Information Needed]
Training Procedure
Training Hyperparameters
- Training regime: [More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
[More Information Needed]
Factors
[More Information Needed]
Metrics
Evaluation was performed against 500 held-out synthetic scenarios.
Metric 200 steps 1,875 steps / 3 epochs
Valid JSON 100% 100%
Operation accuracy 74.2% 96.0%
Target accuracy 45.2% 92.2%
Results
The architecture, dataset, LoRA configuration, and model size were unchanged between these runs.
The primary difference was training duration.
Target accuracy increased from 45.2% to 92.2% without increasing model size or inference cost.
Summary
These results should not be interpreted as demonstrating 96% accuracy on arbitrary real-world attention-management tasks.
The evaluation examples are held out from training, but they are generated by the same synthetic scenario-generation system used to produce the training dataset.
The results therefore demonstrate that a 0.5B model can successfully learn the attention policy represented by this dataset.
Generalisation to different distributions, adversarial scenarios, and real Omega cognitive traces remains to be evaluated.
More Information [optional]
Model Card Authors [optional]
Model Card Contact
Framework versions
