# ============================================================================
# HMX ORM - Qwen3.6 27B Reasoning Preservation Fine-tune
# Optimized for:
# - ERP reasoning specialization
# - Preserve general reasoning capability
# - Preserve conversational ability
# - Reduce catastrophic forgetting
# - Avoid Odoo overbias
# - Long-context business reasoning
# ============================================================================

# ─── Base Model ──────────────────────────────────────────────────────────────
base_model: Qwen/Qwen3.6-27B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

# ─── Dataset ────────────────────────────────────────────────────────────────
datasets:
  - path: HashMicro/test-accounting
    revision: main
    data_files: "data/v3.0.0/**/*.jsonl"
    type: chat_template
    field_messages: messages
    field_role: role
    field_content: content

# IMPORTANT:
# Delete this folder whenever:
# - changing dataset version
# - changing sequence length
# - changing tokenizer
dataset_prepared_path: /workspace/data/last_run_prepared
chat_template: chatml

# Long-context reasoning
sequence_len: 4096
# Better cognition preservation
sample_packing: false
# Avoid unnecessary padding
pad_to_sequence_len: false

# ─── LoRA / PEFT ────────────────────────────────────────────────────────────
adapter: lora
# Resume / merge adapter if needed
lora_model_dir:
# Safer adaptation for reasoning preservation
lora_r: 16
lora_alpha: 32
lora_dropout: 0.03
# More stable scaling
peft_use_rslora: true
# DoRA unnecessary for this objective
peft_use_dora: false
# Keep empty unless custom norm handling needed
peft_kwargs:
  layer_norm_names: []

# IMPORTANT:
# Attention-only tuning first.
# Safer for preserving reasoning + personality.
#
# If ERP reasoning still weak later:
# add:
# - gate_proj
# - up_proj
# - down_proj
#
# But MLP tuning increases overwrite risk.
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj

# ─── Quantization / QLoRA ──────────────────────────────────────────────────
load_in_4bit: true
# Best-practice QLoRA settings
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: true
bnb_4bit_compute_dtype: bfloat16
# Precision
bf16: true
fp16: false
tf32: true

# ─── Training Hyperparameters ──────────────────────────────────────────────
micro_batch_size: 2
# Large effective batch for smoother updates
gradient_accumulation_steps: 16
# Low epochs to reduce catastrophic forgetting
num_epochs: 1
optimizer: adamw_torch_fused
lr_scheduler: cosine

# VERY IMPORTANT:
# Low LR preserves pretrained reasoning manifold
learning_rate: 1e-5
warmup_ratio: 0.03
weight_decay: 0.01
# Only train assistant responses
train_on_inputs: false

# Efficient batching
group_by_length: true
# Helps reduce overfitting / memorization
neftune_noise_alpha: 5
# ─── Performance / Stability ───────────────────────────────────────────────
max_grad_norm: 0.3

# Correct modern FA2 config for Axolotl
attn_implementation: flash_attention_2
gradient_checkpointing: true
# ─── Checkpointing & Persistence ───────────────────────────────────────────
output_dir: /workspace/outputs/hmx-orm-v3
# Automatically resume interrupted runs
auto_resume_from_checkpoints: true
# Save twice per epoch
saves_per_epoch: 2

# Keep latest checkpoints only
save_total_limit: 3

# Dynamic checkpoint trigger
#
# Trigger manually:
# touch /workspace/outputs/hmx-orm-v3/axolotl_checkpoint.save
dynamic_checkpoint:
  enabled: true
  check_interval: 25
  trigger_file_path: axolotl_checkpoint.save

# ─── Evaluation ────────────────────────────────────────────────────────────
eval_batch_size: 2

# Evaluate once per epoch
evals_per_epoch: 1
# Small validation split
val_set_size: 0.05

# ─── HuggingFace Hub ───────────────────────────────────────────────────────
hub_model_id: HashMicro/hmx-acc-v3-qwen27b-lora

# Push checkpoints if enabled
hub_strategy: checkpoint

# Manual push recommended after validation
push_to_hub: false

# ─── Logging ───────────────────────────────────────────────────────────────
logging_steps: 10

logging_dir: /workspace/logs/tensorboard

report_to:
  - tensorboard

# ─── Debug ─────────────────────────────────────────────────────────────────
debug: true

# ─── Multi GPU / Distributed ───────────────────────────────────────────────
ddp_find_unused_parameters: false
ddp_timeout: 18000

hmx-acc-v3-qwen27b-lora

This model is a fine-tuned version of Qwen/Qwen3.6-27B on the HashMicro/test-accounting dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 884
training_steps: 29493

Training results

Framework versions

PEFT 0.18.1
Transformers 5.3.0
Pytorch 2.9.1+cu128
Datasets 4.5.0
Tokenizers 0.22.1

HashMicro/hmx-acc-qwen27b-v3-m

hmx-acc-v3-qwen27b-lora

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

HashMicro/
hmx-acc-qwen27b-v3-m