Llama-3.1-8B-MilitaryCPT
Model Description
This is a specialized, continuously pre-trained (CPT) version of Meta's Llama 3.1 8B model. It has been fine-tuned on a proprietary dataset consisting of raw military doctrine, tactical manuals, and Situation Reports (SITREPs). The goal of this model is to deeply integrate military terminology, acronyms, and operational contexts into the base model's internal knowledge representation.
IMPORTANT: This is a foundation model that has only completed the Continuous Pre-Training phase. It is not an instruction-following assistant yet. It is designed to be the base weights for a subsequent Supervised Fine-Tuning (SFT) phase.
Training Details
- Base Model:
unsloth/Meta-Llama-3.1-8B
- Architecture: 8 Billion Parameters, Transformer
- Training Method: LoRA (Low-Rank Adaptation)
- LoRA Rank (r): 128
- LoRA Alpha: 32
- Target Modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, embed_tokens, lm_head
- Quantization: Trained in 4-bit, merged and exported to 16-bit (
bfloat16)
- Hardware: NVIDIA A100 (Ampere, 80GB) via Slurm HPC
- Frameworks: Unsloth, Hugging Face Transformers, TRL, PyTorch
Intended Use
This model is intended for developers and researchers building military or tactical AI assistants. It provides a robust, domain-specific foundation for further fine-tuning (SFT/DPO).
Limitations
Because this model was trained on raw text using next-token prediction, prompting it with questions will likely result in it autocompleting the text like a manual rather than providing conversational answers.
