NeuraNET Zero is a large language model designed for conversational interaction and general text generation. This repo contains the public NeuraNET Zero 18B Preview checkpoint, which includes:

the RiXIS 1 configuration and modelling implementation;
BF16 model weights;
small X\NeuraNET tokenizer and chat template;
eager attention and PyTorch SDPA support;
standard generation and KV-cache support

NeuraNET Zero is released as a preview for non-commercial research and evaluation.

Model Specifications

Property	Value
Model	NeuraNET Zero 18B Preview
Architecture	RiXIS 1
Parameters	17,711,116,288
Transformer layers	80
Vocabulary size	32,001
Maximum context length	32,768 tokens
Normalisation	RMSNorm
Primary language	English
Licence	CC BY-NC-ND 4.0

The model uses grouped-query attention with 32 query heads and 8 key-value heads.

Usage

Installation

pip install -U "transformers==5.14.1" accelerate safetensors torch

Example

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "rubenroy/NeuraNET-Zero-18B-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype="auto",
    device_map="cuda"
).eval()

messages = [
    {
        "role": "user",
        "content": "Hello there! Can you introduce yourself?"
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True
).to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=False,
        use_cache=True,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id
    )

new_tokens = generated_ids[
    :,
    inputs["input_ids"].shape[1]:
]

response = tokenizer.batch_decode(
    new_tokens,
    skip_special_tokens=True
)[0]

print(response.strip())

Sampling

The above example uses greedy decoding for stable output:

do_sample=False

For more varied generation, sampling can be enabled:

python

generated_ids = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
    repetition_penalty=1.05,
    use_cache=True,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id
)

RiXIS

RiXIS 1 [PREVIEW] is contained in this repository via:

configuration_rixis1.py
modeling_rixis1.py

For this reason:

trust_remote_code=True

Please review repository code before enabling remote execution.

DISCLAIMER: Authorised public RiXIS 1 model weights release ("NeuraNET Zero"). Source files are a reference implementation for loading and inference. proprietary development infrastructure and implementation details are omitted.

Evaluation

This release does not currently publish a benchmark suite or verified evaluation results. Community evaluation is welcome within the terms of the licence. Evaluation reports must state:

the exact model revision;
the prompt or chat format;
decoding parameters;
hardware and software versions;
any preprocessing or postprocessing;
whether the model was modified

Licence

Licensed under the Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License (CC BY-NC-ND 4.0); you may not use this file except in compliance with the License. You may obtain a copy of the License at

CC BY-NC-ND 4.0

Unless required by applicable law or agreed to in writing, this work is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.