HazLLM 1.5B
HazLLM is a highly specialized, locally fine-tuned 1.5B parameter language model engineered explicitly for Roblox Luau development and modern web development (HTML, CSS, JS).
Built on top of Qwen2.5-Coder-1.5B-Instruct, this model was surgically tuned to eliminate the repetition degeneration bug commonly found in smaller models. It delivers concise, expert-level code without hallucinating or looping endlessly.
Training Statistics
- Base Model:
unsloth/Qwen2.5-Coder-1.5B-Instruct
- Dataset: 6,636 highly-curated instruction pairs
- Focus Areas: Roblox Luau, Modern Web Dev, Anti-Repetition logic
- Final Loss:
0.5201 (Dropped from 1.325)
- Hardware: Trained locally on a single NVIDIA RTX 4060 (8GB VRAM)
- Peak VRAM Usage: 7.82 GB (97.7% absolute optimization limit)
- LoRA Rank: 64 (Massive capacity for complex API retention)
- Precision:
bfloat16
- Optimization Framework: Unsloth Engine
Architecture and Methodology
The core problem with 1.5B parameter models is their tendency to fall into repetition loops or output generic, unoptimized code. HazLLM solves this through a highly specialized dataset and aggressive hyperparameter tuning.
The training data consists of 6,636 manually structured JSONL objects. We explicitly trained the model on modern Luau paradigms, such as TweenService, DataStoreService, strict type-checking, and responsive GUI constraints. For web development, the model was fed modern semantic HTML5, CSS Grid/Flexbox architectures, and vanilla ES6+ JavaScript.
To kill the repetition bug, the dataset included dedicated "anti-repetition" conversations where the assistant was explicitly rewarded for stopping its generation cleanly. During inference, we run a repetition penalty of 1.2 paired with a no_repeat_ngram_size of 4, ensuring absolute conciseness.
Training Pipeline
The training pipeline bypassed standard web-UI fine-tuning limits by using custom Python scripts built directly around the unsloth library. The trainer successfully bypassed PyTorch sub-byte bugs and HuggingFace pickling crashes to deliver a seamless, ultra-optimized 2.5 hour training run on consumer hardware.
The optimizer used was adamw_8bit with a learning rate of 3e-4 over 2 complete epochs, allowing the model to quickly converge to a loss of 0.5201 without overfitting.
Capabilities
- Roblox Luau Expert: Fully understands TweenService, ScreenGui, DataStoreService, and modern Luau type-checking.
- Web Developer: Writes semantic HTML5, CSS3 (Flexbox/Grid), and vanilla JS efficiently.
- Strictly Concise: Trained with custom system prompts and data specifically designed to force the AI to stop generating once the correct answer is given. No endless looping.
Usage
You can download the GGUF model directly from this repository to run in Ollama, LM Studio, or llama.cpp. Alternatively, you can run the raw 16-bit weights using standard HuggingFace pipelines.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("c7s89r/HazLLM-1.5B")
4model = AutoModelForCausalLM.from_pretrained("c7s89r/HazLLM-1.5B", device_map="auto")
5
6prompt = "Write a Roblox script to fade a GUI in using TweenService."
7inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
8outputs = model.generate(**inputs, max_new_tokens=512)
9print(tokenizer.decode(outputs[0]))
Links
- GitHub Repository: c7s89r/HazLLM
Contributors
Creator and Lead Developer: @c7s89r
