🤖 Thoth Text Model

📘 Overview

🧠 Base Model

• Base: Qwen/Qwen2.5-7B-Instruct
• Adapter: LoRA fine-tuned using Axolotl
• Architecture: Transformer Decoder (Causal LM)
• Precision: bfloat16
• Frameworks: PyTorch + Transformers + PEFT

🏋️ Fine-tuning Details

• Library: Axolotl
• Adapter Type: LoRA
• Learning Rate: 2e-4
• LoRA α: 16
• LoRA r: 8
• Dropout: 0.05
• Batch Size: 16
• Epochs: 1
• Optimizer: adamw_bnb_8bit
• Sequence Length: 4096
• Compute: RunPod GPU Instance

📂 Dataset

⚠️ Note:
The dataset used for fine-tuning is private and locally stored at
/workspace/fine-tuning/data/trump.json

It follows the Alpaca-style JSON format: