Hemlock-Qwen2.5-Coder-32B – AI Model by nbeerbower | AlphaNeural AI

You can deploy this model and start earning money today!

nbeerbower/
Hemlock-Qwen2.5-Coder-32B

like

0

text-generation

nbeerbower/hemlock-sft-v0.1

Qwen/Qwen2.5-Coder-32B-Instruct

text-generation-inference

endpoints_compatible

Views

No views yet

image/png

Hemlock-Qwen2.5-Coder-32B

An attempt to train Qwen2.5-Coder on The Hemlock Programming Language using SFT.

Results

These models work! Qwen2.5-Coder goes from a completely made up syntax to mostly one-shotting runnable programs.

However, they do still hallucinate. Retrained models will come later with better data and higher LoRA rank and alpha.

Training Configuration

Parameter	Value
Training Mode	SFT
Base Model	`Qwen/Qwen2.5-Coder-32B-Instruct`
Learning Rate	9e-05
Epochs	2
Batch Size	1
Gradient Accumulation	16
Effective Batch Size	16
Max Sequence Length	2048
Optimizer	paged_adamw_8bit
LR Scheduler	cosine
Warmup Ratio	0.05
Weight Decay	0.01
Max Grad Norm	0.25
Seed	42
LoRA Rank (r)	128
LoRA Alpha	64
LoRA Dropout	0.05
Target Modules	up_proj, down_proj, gate_proj, k_proj, q_proj, v_proj, o_proj
Quantization	4-bit (NF4)
GPU	NVIDIA RTX A6000

Trained with Merlina

Merlina on GitHub