QED-B1-Instruction-v3-300M
Model Description
QED-B1-Instruction-v3-300M is an instruction-tuned version of QED-B1-Instruction-v2, a decoder-only Transformer language model trained entirely from scratch.
The underlying foundation model was pretrained using original model weights and an independently created dataset. QED-B1-Instruction-v2 further trains the QED-B1-Instruction-v2 on an original instruction-following dataset to improve its ability to understand prompts, follow instructions, answer questions, write text, generate code, and perform general assistant-style tasks.
No external pretrained model weights were used at any stage of development.
Model Details
| Property | Value |
|---|
| Model Name | QED-B1-Instruction-v3-300M |
| Base Model | Qarvexium/QED-B1-Instruction-v2 |
| Model Type | Decoder-only Transformer |
| Parameters | 299.82M |
| Hidden Dimension | 1024 |
| Layers | 12 |
| Attention Heads | 16 |
| Activation Function | SwiGLU |
| Normalization | RMSNorm |
| Position Encoding | RoPE |
| Vocabulary Size | 96,000 |
| Tokenizer | SentencePiece |
| Training Method | Instruction Tuning |
Architecture
QED-B1-Instruction-v3 retains the same architecture as QED-B1-Instruction-v2, including:
- Rotary Positional Embeddings (RoPE)
- RMSNorm
- SwiGLU feed-forward networks
- Multi-head causal self-attention
- Weight-tied token embeddings
- Decoder-only Transformer architecture
Instruction tuning updates the model weights while preserving the original architecture.
Training Data
QED-B1-Instruction-v3 was instruction-tuned using a fully original dataset created specifically for this project.
The dataset was independently authored and includes a diverse collection of instruction-response pairs covering:
- General knowledge
- Reasoning
- Coding
- Mathematics
- Writing
- Summarization
- Text transformation
- Question answering
- Conversation
- Creative generation
The data creation process included:
- Original prompt creation
- Original responses
- Custom formatting
- Custom preprocessing
- SentencePiece tokenization
No public instruction datasets or existing assistant conversations were used.
Intended Use
QED-B1-Instruction-v3 is intended for:
- General-purpose AI assistants
- Instruction following
- Text generation
- Code generation
- Research
- Fine-tuning
- Educational projects
- Experimentation
Limitations
Although instruction-tuned, QED-B1-Instruction-v3 has limitations:
- It can still produce incorrect or fabricated information.
- Responses may vary in quality depending on prompt complexity.
- It has not undergone reinforcement learning from human feedback (RLHF).
- It should not be relied upon for high-risk applications without human verification.
License
MIT License.
