This report documents the training, adapter merge, and GGUF export process for
TNSA/NGen-4-OW-4B-Thinking. The run used supervised fine-tuning through a
LoRA adapter on top of the Hugging Face base model, with the ArabicQA 2.1M
dataset normalized into ChatML-style training examples.
The final training run completed successfully, saved the adapter to the Modal
adapter volume, merged the adapter back into the base model, converted the
merged Hugging Face model to GGUF, patched the GGUF metadata where required,
and produced a quantized local file:
outputs/gguf/ngen4-ow-4b-thinking-Q4_K_M.gguf
The downloaded quantized GGUF size is 2,708,803,872 bytes, approximately
2.71 GB.
2. Model Identity
The trained model is based on:
TNSA/NGen-4-OW-4B-Thinking
The model is treated as a TNSA multilingual reasoning and chat model. The
fine-tuning objective was to adapt it toward a large Arabic question-answering
corpus while preserving the base model's reasoning and long-context behavior.
The model was not trained from scratch. The training was parameter-efficient:
the original base weights were loaded in quantized form, frozen, and adapted
through trainable LoRA matrices attached to selected projection layers.
3. Dataset
The dataset used for this run was:
riotu-lab/ArabicQA_2.1M
The CLI pulled the Hugging Face dataset using the train split and normalized
it into the local JSONL file:
data/prepared/ngen41_train.jsonl
The local dataset statistics were:
Field
Value
Source dataset
riotu-lab/ArabicQA_2.1M
Split
train
Streaming load
true
Rows kept
2,141,104
Rows skipped as unrecognized
42
Prepared JSONL size
about 2.29 GB
The data preparation CLI supports common Hugging Face formats such as
messages, conversations, prompt/response, instruction/output,
question/answer, and problem/solution. Rows from ArabicQA were converted
into a standard message structure with system, user, and assistant roles.
The prepared examples were rendered during training with a ChatML-like format:
The model was trained with supervised fine-tuning using Hugging Face
Trainer. The base model was loaded in 4-bit quantized form using
BitsAndBytes:
Setting
Value
Quantization
4-bit
Quant type
nf4
Compute dtype
bfloat16
Double quantization
enabled
Attention implementation
sdpa
Gradient checkpointing
enabled
Use cache
disabled during training
The base model was prepared for k-bit training through PEFT's
prepare_model_for_kbit_training. This is important because quantized base
models are not trained directly like full-precision models. Instead, the base
weights remain effectively frozen while LoRA adapter weights receive gradient
updates.
6. Run Hyperparameters
The final run manifest shown in the training screenshot recorded:
Hyperparameter
Value
Base model
TNSA/NGen-4-OW-4B-Thinking
LoRA rank
64
LoRA alpha
128
Learning rate
2e-6
Per-device batch
2
Gradient accumulation
4
Recorded effective batch
64
Max steps
150
Training examples
2,141,104
Final logged training loss
about 2.205
Training wall time in screenshot
about 1:54:58
The screenshot also showed the training loop completed all 150/150 steps,
then saved the final adapter and committed the output to the Modal volume.
Batch Size Note
The run manifest reports effective_batch: 64. The visible training
parameters are per_gpu_batch: 2 and grad_accum: 4. In the training script,
the manifest computes effective batch as:
per_gpu_batch * grad_accum * 8
That records an 8-way multiplier. If the run is executed on a single GPU
configuration, the practical optimizer batch from the visible values is:
2 * 4 = 8 examples per optimizer step
The report keeps the manifest value because it is the value saved by the run,
but this difference should be cleaned up in the training script for future
runs if single-GPU and multi-GPU jobs are both supported.
7. LoRA Configuration
The model was adapted with LoRA rather than full fine-tuning. LoRA inserts
small trainable low-rank matrices into selected linear layers. During training,
only these adapter matrices are updated; the original base model weights remain
frozen.
These targets cover the most important transformation points in a decoder-only
language model:
Target
Function
Why it was trained
q_proj
Query projection in attention
Helps the model learn what to attend to for Arabic questions and instructions
k_proj
Key projection in attention
Adjusts how source tokens are represented for retrieval inside attention
v_proj
Value projection in attention
Changes what content is carried forward once attention is assigned
o_proj
Attention output projection
Lets the adapted attention result be integrated back into the residual stream
gate_proj
MLP gate projection
Controls which feed-forward features activate
up_proj
MLP expansion projection
Allows new task/domain features to be represented in the hidden expansion
down_proj
MLP compression projection
Maps adapted MLP features back into the model hidden size
8. Why These Layers Were Changed
The selected modules balance adaptation strength and stability.
Attention projections (q_proj, k_proj, v_proj, o_proj) are responsible
for token-to-token interaction. For a QA-heavy Arabic dataset, the model must
learn patterns such as question intent, answer grounding, semantic matching,
and multilingual phrasing. Adapting attention projections helps the model
change how it routes information across the context without rewriting the
entire model.
The MLP projections (gate_proj, up_proj, down_proj) are responsible for
feature transformation after attention. These layers are where many factual,
stylistic, and reasoning transformations are expressed. Training them with
LoRA gives the model capacity to adapt answer style, domain conventions, and
Arabic QA patterns.
The combination of attention and MLP LoRA is stronger than attention-only
LoRA, because the model can both attend differently and transform the attended
information differently. At the same time, it is much cheaper and safer than
full fine-tuning, because the base model remains frozen.
9. Layers Intentionally Not Changed
The following parts were intentionally not directly trained:
Component
Reason
Base dense weights
Preserves base model knowledge and reduces training cost
Token embeddings
Avoids destabilizing the tokenizer vocabulary and rare-token behavior
Layer norms
Keeps normalization behavior stable during low-rank adaptation
Router/gating control outside LoRA targets
Reduces risk of routing collapse or expert imbalance
Output head
Avoids unnecessary large adapter memory and preserves base logits geometry
The manifest records:
"router_frozen": true
Freezing router behavior is especially important for architectures with
specialized routing or hybrid blocks. Router changes can create sharp behavior
shifts, unstable expert use, and worse generalization if the dataset is narrow.
For this run, the goal was domain adaptation rather than architectural
re-routing, so the router was kept frozen.
10. Sequence Length and Long Context
The training script was configured for long context:
text
1max_seq_length = 131072
2sample_max_new = 65536
The GGUF metadata later showed:
qwen35.context_length = 262144
This means the base model and exported GGUF retain a very long context
configuration. For practical serving, the actual usable context depends on the
inference backend, GPU VRAM, quantization, KV cache settings, and request
length. The vLLM backend scaffold created separately uses a conservative
--max-model-len 32768 by default so deployment is more likely to fit on
available hardware.
11. Training Flow
The end-to-end training flow was:
Pull the Hugging Face dataset.
Normalize rows into chat messages.
Write the local JSONL dataset.
Upload the JSONL to the Modal data volume.
Load the base model in 4-bit NF4.
Attach LoRA adapters to selected projection layers.
Tokenize the JSONL dataset into causal language modeling sequences.
Train for the configured number of optimizer steps.
Save the final adapter to /out/final.
Commit the adapter volume.
Merge the adapter into the base model.
Save merged Hugging Face weights.
Convert merged model to f16 GGUF.
Patch GGUF block metadata if needed.
Quantize to Q4_K_M.
Download the final GGUF locally.
12. Export and GGUF Conversion
After training, the adapter was merged into the base model using PEFT's
adapter-loading path. This was required because the base model carries PEFT or
adapter-related metadata, and the older manual merge path caused a Transformers
save-time adapter-state error.
This patch was necessary because the converter counted an extra prediction or
auxiliary layer in a way that can confuse GGUF runtimes. The patch prevents
runtime errors where the loader expects weights for a nonexistent layer.
The quantized GGUF was checked again after quantization and already had:
The very small epoch value is expected because the dataset is extremely large
relative to the number of steps. With 2,141,104 examples and only 150
training steps, the run is a short adaptation pass rather than a full epoch
over the dataset.
This is useful as a first successful adapter build because it verifies:
Dataset normalization works.
Modal upload works.
The model can be loaded in 4-bit mode.
LoRA modules can be attached and trained.
The adapter can be saved.
The adapter can be merged and exported.
The GGUF can be quantized and downloaded.
It should not be treated as a fully converged training run. A longer training
run, validation set, and benchmark suite are needed before making quality
claims.
16. Why This Training Strategy Was Reasonable
Full fine-tuning a 4B-class model requires more memory, more time, and greater
risk of catastrophic forgetting. LoRA gives a strong middle path:
The base model remains intact.
Training is cheaper.
The adapter is small and easy to save.
The adapter can be merged later for deployment.
The target modules give enough coverage for meaningful behavior change.
The ArabicQA dataset is large and domain-specific. A short LoRA run is a good
first pass for teaching the model dataset style and Arabic QA response
patterns without overwriting general reasoning ability.
17. Limitations
The run has several limitations:
The final run was only 150 steps.
The dataset was large, so the model saw only a small fraction of one epoch.
No validation metrics are recorded in the screenshot.
The final loss alone is not enough to evaluate answer quality.
The effective batch value in the manifest should be corrected for future
single-GPU runs.
The report does not claim benchmark improvements without evaluation.
ArabicQA rows were normalized automatically, so spot checks should be done
for formatting and answer quality.
18. Recommended Next Steps
Recommended follow-up work:
Run a longer training job, for example 750 to 2000 steps.
Add a held-out validation split from ArabicQA.
Log validation loss every fixed number of steps.
Add Arabic QA evaluation prompts.
Add multilingual sanity prompts to ensure English and general reasoning did
not regress.
Fix the effective_batch manifest calculation so it reflects the actual
GPU count.
Save the exact Modal app run URL and run ID in the manifest.
Save a copy of the export manifest locally with the GGUF.
Test the GGUF in the vLLM RAG backend and with a direct local runtime.
19. Reproducibility Commands
The dataset preparation and training command used for this family of run was:
The training pipeline successfully adapted TNSA/NGen-4-OW-4B-Thinking with a
LoRA adapter over selected attention and MLP projection layers, using
riotu-lab/ArabicQA_2.1M as the training corpus. The adapter was saved,
merged, converted to GGUF, patched for correct Qwen block metadata, quantized
to Q4_K_M, and downloaded locally.
The most important architectural decision was to update projection layers
instead of the full model. This gave the run enough flexibility to learn Arabic
QA behavior while keeping training cost and regression risk low. The router and
core base weights remained frozen to preserve stability.
This should be considered a successful first training and export pass. The next
step is evaluation: compare the base model and the tuned GGUF on Arabic QA,
general chat, reasoning, and RAG tasks before positioning it as a production
model.
