NeuroBLAST‑1.9B Instruct - Early Preview (Research‑Only)
Status: 🚧 Early preview checkpoint - not production‑ready.
Core message: This model is released to showcase the NeuroBLAST (Brain‑Like Architecture with Stacked Transformers) design and to catalyze community research. Do not rely on its outputs for clinical, legal, financial, or safety‑critical decisions.
CRITICAL MEDICAL DISCLAIMER: This model is NOT for medical diagnosis, treatment, triage, or as a source of medical knowledge/reference. Do not use its outputs to inform clinical decisions, guidelines, or patient education. Always consult qualified healthcare professionals and verified medical resources.
TL;DR
Parameters: 1.9B
Objective: Instruction‑tuned conversational model demonstrating the NeuroBLAST architecture.
Training tokens: 100B tokens of permissively licensed, open‑source text.
Context length: trained on 16k-token sequences; RoPE configured to 132k positions (effective max).
Sources: Mix of general web text (FineWeb, FineWeb‑Edu) and permissive/open medical corpora (e.g., PMC Open Access subset, PubMedQA, MedMCQA).
License: Apache‑2.0 for model weights & code. Downstream users must also respect underlying data licenses.
Use scope: Research, benchmarking, and architecture exploration only. No medical advice, patient care use, or use as a source of medical knowledge/reference.
1. Model Details
1.1 Architecture
NeuroBLAST (Brain‑Like Architecture with Stacked Transformers) arranges transformer modules in stacked, semi‑independent “blocks” with explicit interfaces intended to mimic hierarchical processing pathways. This checkpoint keeps the stack compact (1.9B params) to ease experimentation and ablation.
Architecture
Diagramy summary:
Input tokens are embedded with a token embedding layer plus Rotary Positional Embeddings (RoPE).
The stream first enters an Association Cortex block (× N layers) composed of standard self‑attention followed by a SwiGLU MLP.
From here, two pooler layers branch out (Assoc → Sensory Pooler and Assoc → Motor Pooler) that compress/route state to the downstream modules.
A Sensory Cortex stack (× M layers, green) receives:
Its own self‑attention.
Gated cross‑attention to the Association Cortex (σ gating).
A SwiGLU MLP.
It can also cross‑attend to the Motor Cortex.
A Motor Cortex stack (× K layers) includes:
Self‑attention.
Gated cross‑attention to the Sensory Cortex (σ gate) and ungated cross‑attention to the Association Cortex.
A SwiGLU MLP.
A third pooler (Sensory → Motor Pooler) provides a condensed sensory representation to the motor block.
After these specialized cortex modules, activations pass through a LayerNorm and a Linear head for next‑token prediction.
Memory Layer: WIP
1.2 Checkpoint Type
Pretraining: Next‑token prediction on mixed‑domain corpora.
Instruction tuning: Lightweight SFT/DPO pass for chat usability.
Precision: BF16 weights.
1.3 Files & Formats
Weights in safetensors format.
trust_remote_code=True is required due to custom generation utilities.
2. Intended Use
2.1 Primary Intended Use
Research on model architectures (e.g., stacked transformers), data curation, and instruction‑tuning techniques.
Benchmarking against similarly sized open models.
Educational demos of training/inference pipelines.
2.2 Out‑of‑Scope / Prohibited Uses
Medical, diagnostic, or therapeutic decision support.
Any clinical workflow or patient‑facing chatbot.
Automated decision‑making in safety‑critical or regulated contexts (finance, law, aviation, etc.).
Generation of disallowed content per applicable laws/policies (e.g., hate speech, harassment, illegal acts) without appropriate safeguards.
3. Safety & Disclaimer (Read This!)
NOT for medical diagnosis, treatment, triage, or as a source of medical knowledge/reference. Outputs may be wrong, outdated, or dangerously misleading; never use them to make or inform clinical decisions. Always consult qualified healthcare professionals and authoritative medical resources.
Hallucinations happen. The model can produce confident but incorrect statements and fabricated citations.
No warranty. The model is provided “AS IS,” without guarantees of accuracy or fitness for purpose.
Handle sensitive data carefully. Do not input Protected Health Information (PHI) or other confidential data unless you have legal permission and proper safeguards.
Regulatory compliance is your responsibility. If you fine‑tune, deploy, or integrate this model, ensure compliance with relevant laws (e.g., HIPAA, GDPR, FDA/EMA rules for clinical decision support).
Emergency disclaimer: If you or someone else may be experiencing a medical emergency, call your local emergency number immediately. Do not rely on this model.
4. Training Data
4.1 High‑Level Composition (~100B tokens)
General web corpus: Curated slices of FineWeb and FineWeb‑Edu.
Medical/biomedical corpora (permissive licenses): Examples include the PMC Open Access subset (Creative Commons family), PubMedQA (MIT), MedMCQA (MIT/CC0 variants), and similar open QA/MCQ datasets.
Curated SFT data: Created by medical doctors and authors of the Medcases Application.
Synthethic SFT data
4.2 Licensing Summary
Source
License (examples)
Notes
FineWeb / FineWeb‑Edu
ODC‑By 1.0
Subject to CommonCrawl ToS
PMC Open Access subset
CC‑BY / CC0 / CC‑BY‑NC, etc. (varies per article)
Only OA commercially allowed articles with machine‑readable licenses were used
PubMedQA
MIT
QA over biomedical abstracts
MedMCQA
MIT/CC0 (check source)
Multiple‑choice medical exam questions
Downstream users must respect these licenses when redistributing data derivatives.
