Zenyx-Vanta is a modernized Bidirectional Encoder (BERT-style) model. This iteration uses the Omni-Mix dataset strategy, designed to provide the encoder with a balance of high-quality educational text, general web knowledge, Pythonic logic, and mathematical reasoning.

Architecture Details

Model Type: Masked Language Model (MLM)
Parameters: ~350 Million
Tokenizer: Qwen 2.5 (151,646 vocab size)
Positioning: Rotary Positional Embeddings (RoPE) with 10k base
Activation: SwiGLU (SiLU-gated MLP)
Attention: Grouped Query Attention (GQA) with 12 Heads (4 KV Heads)

Training Data: The "Omni-Mix"

Vanta was trained on a balanced 4-way distribution to maximize cross-domain reasoning:

FineWeb-Edu (25%): High-signal educational content.
FineWeb (25%): General linguistic context from broad web crawls.
StarCoderData - Python (25%): Source code for logic and syntax understanding.
Open-Web-Math (25%): Mathematical text and LaTeX for symbolic reasoning.

Technical Specifications

Parameter	Value
`hidden_size`	768
`num_hidden_layers`	12
`num_attention_heads`	12
`num_key_value_heads`	4
`intermediate_size`	3072
`max_position_embeddings`	2048
`hidden_act`	SwiGLU (SiLU)

Quick Start / Inference

To use Zenyx-Vanta for mask filling, you can use the following snippet (requires jax, flax, and transformers):

python

from transformers import AutoTokenizer
import jax.numpy as jnp
# Note: Ensure your local ZenyxVanta architecture definition matches the model weights
# model = ZenyxVanta(vocab_size=151646)

tokenizer = AutoTokenizer.from_pretrained("Arko007/zenyx-vanta-bert")
text = "The powerhouse of the cell is the ___."
prompt = text.replace("___", "<|MASK|>")

inputs = tokenizer(prompt, return_tensors="np")
# logits = model.apply({'params': params}, inputs['input_ids'])
# ... (Standard JAX inference logic)

Credits

Developed by Arko007 and the Zenyx team.