Phase 5 Hugging Face deployment from slm-train-scratch.
Model summary
Model type: decoder-only causal language model (LlamaForCausalLM).
Model size: 150M-class; measured checkpoint size is 154.42M parameters.
Instruction tuning:Non-IT / non-instruction-tuned. This is a base language model, not a chat model and not instruction aligned.
Tokenizer: byte-level BPE-style tokenizer with vocabulary size 32000.
Precision: bfloat16 checkpoint.
Training data
The model was pretrained on a cleaned mixture of general educational text and medical/health-domain text.
General/education source: HuggingFaceFW/FineWeb-Edu, cleaned and filtered subset: 2.00M rows.
Medical/health source: TheBlueScrubs/the_blue_scrubs-v1, cleaned and filtered subset: 2.00M rows.
Packed train tokens: 5.72B tokens.
Packed validation tokens: 18.22M tokens.
Training run
Epochs configured: 1.
Approximate logged wall-clock training time: 73.3 hours (3.1 days). This is based on recorded metrics and may include evaluation, resume, and pause gaps.
Intended use
This checkpoint is intended for:
research experiments with small/base language models;
continued pretraining;
domain adaptation experiments;
evaluation of a compact medical-plus-education pretrained base model.
Limitations
This is not an instruction-tuned or chat-aligned model.
Outputs may be incomplete, incorrect, or unsafe without downstream alignment and evaluation.
The model must not be used as a source of medical advice.
The training mixture contains web and domain text; users should evaluate bias, factuality, memorization, and domain safety before downstream use.
