gerasmark/

Meta-Llama-3.1-8B-Instruct-Forensics-v1

Summary

Model Details

Model Description

• Model type: Instruction-following Language Model (LoRA-based fine-tuning)
• Language(s): English
• Fine-tuned from model: Meta-Llama-3.1-8B-Instruct

Training Details

Training Data

• Dataset: Advanced_Forensic_Investigations_Knowledge_Library_v1
• Data size: ~100 high-quality, domain-specific QA pairs

Training Procedure

Preprocessing

• Template: llama3
• Token truncation/cutoff: 2048
• No vocab resizing or prompt packing

Hyperparameters

• Finetuning type: LoRA
• Precision: bf16
• LoRA rank: 16
• LoRA alpha: 32
• Batch size: 4
• Gradient accumulation: 8
• Learning rate: 3e-4
• Epochs: 35
• LR scheduler: cosine
• Quantisation: 4-bit (bitsandbytes)
• Cutoff length: 2048

Compute

• Training time: close to 30 minutes
• Framework: LLaMA-Factory

Evaluation

Testing Data, Factors & Metrics

• Metrics Used: BLEU-4, ROUGE-1, ROUGE-2
• Results:
  • BLEU-4: 99%
  • ROUGE-1: 99%
  • ROUGE-2: 99%

Technical Specifications

Model Architecture and Objective

• Base: Transformer (Meta-Llama-3.1-8B-Instruct architecture)
• Fine-tuning method: LoRA
• Objective: Instruction-following with forensic legal knowledge adaptation

Compute Infrastructure

• Hardware: 2xL40s
• Software: LLaMA-Factory, PyTorch, Transformers, bitsandbytes