Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

erebus-487m-base – AI Model by soyrsoyr | AlphaNeural AI

You can deploy this model and start earning money today!

soyrsoyr/
erebus-487m-base

like

0

HuggingFaceFW/fineweb-edu

Views

No views yet

Erebus-487M-base

Erebus-487M is a custom-trained 487M parameter language model based on the Llama architecture.

Model Details

Architecture: Llama-style transformer
Parameters: 487.8M
Context Length: 1024 tokens
Training Data: FineWeb-Edu (529M tokens)
Training Steps: 10,000
Final Loss: 3.0528

Architecture Specifics

Layers: 12
Hidden Size: 1536
Attention Heads: 16 (Query) / 4 (Key/Value)
Intermediate Size: 6144
Grouped Query Attention (GQA): 4:1 ratio
RoPE: Rotary Position Embeddings
Activation: SwiGLU
Normalization: RMSNorm

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("erebus-487m-base")
tokenizer = AutoTokenizer.from_pretrained("erebus-487m-base")

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))

Training

Trained from scratch on FineWeb-Edu using:

AdamW optimizer (lr=3e-4)
Batch size: 128 (8 batch × 16 grad accum)
Context: 1024 tokens

License

Apache 2.0

Developer's Note

This model is moreso a bit of practice for me, but maybe there could be some application somewhere