⚠️ This is an early/intermediate checkpoint. The final, recommended model is Aslam-13/aelora-qwen3-4b — use that one unless you specifically want to study how the project evolved.

This checkpoint covers only Level 1 (Velari language) + Level 2 (Thex-Kron base-8 math) of a larger 5-level project. It does not contain logic rules (Vel-Rith), advanced grammar (comparatives, superlatives, imperatives, possession), or world/lore knowledge — those were added in subsequent training runs.

Why a synthetic world? Benchmarks leak into pre-training. By fine-tuning on a domain that provably cannot exist in any pre-training corpus, every correct answer is unambiguous evidence of learning — not retrieval from prior knowledge.

It's kept public because the iteration history is part of the story: training in stages let me observe catastrophic forgetting in real time and validate that combining Level 1 + Level 2 data in a single run prevented the model from "losing" Velari vocabulary while learning math.

🧪 Full project (code, datasets, RAG comparison, writeup): github.com/Aslam-13/Fine_tuning_RAG
🏁 Final model: Aslam-13/aelora-qwen3-4b
📓 Training notebook: kaggle.com/code/syed13/fine-tune

What this checkpoint knows

Domain	Coverage
Velari — vocabulary	30-word lexicon (nouns, verbs, pronouns, adjectives)
Velari — grammar (basic)	Plurals (`-an`), past tense (`ta-`), negation (`ne`), basic SVO sentences
Thex-Kron math	Base-8 numerals (`nul, ek, doi, tri … hep, ek-nul …`), addition with carry, basic multiplication

Does NOT know: Vel-Rith logic rules, comparatives/superlatives, imperatives, possession, world/society lore, multi-step word problems. For those use the final model.

Why a base-8 system?

The base model (Qwen3-4B) "knows" base-10 from pre-training extremely well. Teaching it that 5 + 3 = 10 (in Aeloran base-8) directly contradicts that prior. If this checkpoint correctly answers fen vor tri = ek-nul (5 + 3 = 10₈), it's evidence that LoRA fine-tuning successfully overrode a deeply-baked behavior — not just memorized a surface pattern.

Training details (this checkpoint)


Base model	`unsloth/qwen3-4b-unsloth-bnb-4bit`
Method	LoRA (4-bit) via Unsloth + TRL `SFTTrainer`
Dataset	Combined Level 1 (language) + Level 2 (math) JSONL
License	Apache 2.0

Full hyperparameters and training logs are in the GitHub repo.

Quick start

Option A — with Unsloth (fastest, requires GPU)

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Aslam-13/velari-level2-qwen3-4b",
    max_seq_length = 2048,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

prompt = "What is fen vor tri in Aeloran math?"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Option B — with plain `transformers`

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Aslam-13/velari-level2-qwen3-4b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "What is fen vor tri in Aeloran math?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Expected output (approximate): "ek-nul" (5 + 3 = 8 in base-10 → 10 in base-8 → ek-nul). Exact phrasing may vary.

Limitations

Intermediate checkpoint, superseded by aelora-qwen3-4b.
No logic, no advanced grammar, no world knowledge.
Single-domain research artifact — not a general-purpose model.

Trained with Unsloth.

Aslam-13/velari-level2-qwen3-4b

Velari-Level2-Qwen3-4B — Intermediate Checkpoint