QVAC Genesis II Pretrained Model - Combined (Failure Analysis + Option-Level Reasoning)

Key Highlights

• Pretrained on QVAC Genesis II
  This model has been pretrained on Tether’s QVAC Genesis II dataset.
  Dataset card: https://huggingface.co/datasets/qvac/GenesisII
  The checkpoint combines the Failure Analysis prompt and Option-Level Reasoning prompt formats (equal-weight mixture) and was pretrained on approximately 106B tokens, using BF16 mixed precision and a 4,096-token context window, with a Qwen3-family 1.7B-parameter decoder-only transformer architecture.
• Checkpoints in Hugging Face format
  Checkpoints are provided in standard Hugging Face format for inference, continual pretraining, and fine-tuning.
• Educational coverage
  QVAC Genesis II includes the following domains:
  • Machine learning
  • High school statistics
  • High school chemistry
  • Econometrics
  • College chemistry
  • College physics
  • Geography
  • Astronomy
  • College computer science
  • Electrical engineering
  • High school computer science

Intended Uses

• Continual pretraining or fine-tuning for educational applications (STEM tutoring, QA systems, curriculum support)
• Benchmarking reasoning and subject-specific QA performance
• Research into synthetic dataset-driven LLM pretraining

Model Details

Model Description

• Developed by: QVAC by Tether
• Model type: Decoder-only Transformer (causal LM)
• Language(s) (NLP): Primarily English
• License: Apache-2.0
• Finetuned from model: None (randomly initialized)
• Intended stage: Base pretrained model (no SFT / RLHF)

Dataset Details

• Dataset: qvac/GenesisII
• Prompt formats: Failure Analysis prompt + Option-Level Reasoning prompt

Uses

Direct Use

• General language modeling: next-token prediction, continuation, summarization, drafting.
• Research baseline for scaling, data ablations, or tokenizer studies.

Downstream Use (recommended)

• Continued Pre-Training (CPT) on more tokens.
• SFT for assistants, domain experts, or task-specific models.
• Preference optimization / RLHF for safer, more helpful behavior.
• Adapters/LoRA for efficient domain specialization.

Out-of-Scope Use

• High-stakes decision-making (medical/financial/legal)
• Safety-critical or autonomous control systems
• Unfiltered end-user deployment without alignment / safety layers
• Any use that violates applicable laws or platform policies.

Bias, Risks, and Limitations

• Bias & toxicity: May reflect or amplify biases present in web text.
• Hallucinations: Can produce confident but incorrect statements or citations.
• Security / privacy: May emit continuous random strings.
• Context limit: 4,096 tokens; longer inputs require chunking.

Recommendations

• Disclose limitations to downstream users.
• Research model: not intended for production use cases.

How to Get Started

Training Details

Training Data

• Dataset: qvac/GenesisII
• Prompt formats: Failure Analysis prompt + Option-Level Reasoning prompt
• Total pretraining tokens: ~106B tokens
• Tokenizer: Qwen3 tokenizer

Training Procedure

• Framework: Megatron-LM (Megatron-Core)
• Sequence length: 4,096
• Precision: BF16
• Optimizer: AdamW (β₁=0.9, β₂=0.95), weight decay 0.01
• Learning rate: 2e-4 → 2e-5
• Warmup: 1,215 steps
• Training steps: 12,158
• Gradient clipping: 1.0
• Seed: 42
• Logging: every 100 steps
• Checkpointing: every 500 steps
• Blending: equal weights (50:50)

Multi-Node GPU Setup

• Cluster: 8 nodes, each with 8 NVIDIA H100 80GB (64 GPUs total)
• Scheduler: Slurm
• Launch: srun + torchrun
• Framework: Megatron-LM

Technical Specifications

Model Architecture and Objective

• Architecture: Qwen3-style decoder-only Transformer
• Parameters: ~1.7B
• Context length: 4,096 tokens
• Objective: Causal LM (next-token prediction)

Conversion & Inference

Changelog

• v0.1 (2025-12-16): Initial release - combined prompt-format variant