Athenas-Primal-8B is a LoRA fine-tune of Qwen3-8B (4-bit quantized) trained on synthetic agentic software engineering traces. The model is designed to investigate, diagnose, and fix production-grade bugs across multiple programming languages and technology stacks.

Model Description

Developed by: kodjaoglanian
Base model: unsloth/Qwen3-8B-unsloth-bnb-4bit
License: Apache 2.0
Finetuned from: Qwen3-8B (4-bit NF4, bitsandbytes)
Adapter type: LoRA (rank 16, alpha 16)
Max sequence length: 4096
Languages: English, Portuguese

Training Methodology

The model was fine-tuned using Supervised Fine-Tuning (SFT) with LoRA adapters via the Unsloth framework and TRL's SFTTrainer.

LoRA Configuration

Parameter	Value
Rank	16
Alpha	16
Dropout	0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Gradient checkpointing	Unsloth

Training Hyperparameters

Parameter	Value
Epochs	3
Learning rate	2e-4
LR scheduler	Linear
Effective batch size	8 (4 × 2 accumulation)
Warmup steps	10
Optimizer	adamw_8bit
Weight decay	0.01
Packing	Enabled

Infrastructure

Platform: AWS SageMaker Studio
GPU: NVIDIA A10G (24 GB)
Framework: Unsloth + TRL
Precision: bfloat16

Dataset

Training data: kodjaoglanian/fable-traces-mini

A synthetic dataset of multi-turn agentic conversations simulating real-world debugging sessions. Each trace follows an iterative investigation pattern:

A production incident is reported with symptoms, logs, and context
The assistant reasons through the problem using structured thinking
Tool calls are made to inspect and modify source code
Failures and compilation errors are encountered and resolved
The root cause is identified and a fix is applied

The dataset covers 29 scenarios across 8 languages (Python, Node.js, Rust, Go, Java, C++, TypeScript, C) with bug categories including memory leaks, race conditions, goroutine leaks, N+1 queries, thread pool starvation, SQL injection, use-after-free, and more.

Evaluation

HellaSwag

Model	acc	acc_norm
Qwen3-8B base (4-bit)	0.5140	0.6580
Athenas-Primal-8B	0.5625	0.7460

Fine-tuning yielded +8.8 percentage points improvement in acc_norm over the base model.

ARC Challenge

Model	acc	acc_norm
Qwen3-8B base (4-bit)	0.5375	0.5623
Athenas-Primal-8B	0.5418	0.5648

Context

Model	acc_norm	Precision
GPT-4	~95.3%	16-bit
Llama-3-70B	~88.7%	16-bit
Qwen2.5-7B	~83.0%	16-bit
Llama-3-8B	~80.7%	16-bit
Athenas-Primal-8B	74.6%	4-bit

Market figures are from full-precision models. The 4-bit quantization accounts for approximately 5–8pp of the gap.

Intended Use

Agentic debugging: Multi-step investigation and resolution of production incidents
Code review: Detection of concurrency bugs, resource leaks, and security vulnerabilities
Cross-language engineering: Python, Node.js, Rust, Go, Java, C++, TypeScript, C

Limitations

Trained on synthetic data — real-world bug patterns may differ
4-bit quantization introduces a measurable accuracy penalty versus full precision
No RLHF or safety alignment was applied
Not optimized for general conversation, creative writing, or mathematical reasoning
Tool calls in training are simulated; the model does not execute code

Quick Start

python

from unsloth import FastLanguageModel
from transformers import TextStreamer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Kodjaoglanian/athenas-primal-8b",
    max_seq_length=4096,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "<your system prompt>"},
    {"role": "user", "content": "Describe your bug or engineering problem here."},
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt",
).to("cuda")

_ = model.generate(
    input_ids=inputs,
    streamer=TextStreamer(tokenizer, skip_prompt=True),
    max_new_tokens=2048,
    temperature=0.7,
    top_p=0.9,
)

Citation

bibtex

@misc{athenas_primal_8b,
  author = {kodjaoglanian},
  title = {Athenas-Primal-8B: Agentic Software Engineering via LoRA Fine-Tuning of Qwen3-8B},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Kodjaoglanian/athenas-primal-8b}
}

Acknowledgments

Unsloth — efficient fine-tuning framework
Qwen Team — base model
TRL — SFTTrainer