SafeAI-HF/ngen4ow-4b-thinking-gguf

Training Report: NGen-4-OW-4B-Thinking

1. Executive Summary

This report documents the training, adapter merge, and GGUF export process for TNSA/NGen-4-OW-4B-Thinking. The run used supervised fine-tuning through a LoRA adapter on top of the Hugging Face base model, with the ArabicQA 2.1M dataset normalized into ChatML-style training examples.

The final training run completed successfully, saved the adapter to the Modal adapter volume, merged the adapter back into the base model, converted the merged Hugging Face model to GGUF, patched the GGUF metadata where required, and produced a quantized local file:

outputs/gguf/ngen4-ow-4b-thinking-Q4_K_M.gguf

The downloaded quantized GGUF size is 2,708,803,872 bytes, approximately 2.71 GB.

2. Model Identity

The trained model is based on:

TNSA/NGen-4-OW-4B-Thinking

The model is treated as a TNSA multilingual reasoning and chat model. The fine-tuning objective was to adapt it toward a large Arabic question-answering corpus while preserving the base model's reasoning and long-context behavior.

The model was not trained from scratch. The training was parameter-efficient: the original base weights were loaded in quantized form, frozen, and adapted through trainable LoRA matrices attached to selected projection layers.

3. Dataset

The dataset used for this run was:

riotu-lab/ArabicQA_2.1M

The CLI pulled the Hugging Face dataset using the train split and normalized it into the local JSONL file:

data/prepared/ngen41_train.jsonl

The local dataset statistics were:

Field	Value
Source dataset	riotu-lab/ArabicQA_2.1M
Split	train
Streaming load	true
Rows kept	2,141,104
Rows skipped as unrecognized	42
Prepared JSONL size	about 2.29 GB

The data preparation CLI supports common Hugging Face formats such as messages, conversations, prompt/response, instruction/output, question/answer, and problem/solution. Rows from ArabicQA were converted into a standard message structure with system, user, and assistant roles.

The prepared examples were rendered during training with a ChatML-like format:

<|im_start|>system
...
<|im_end|>
<|im_start|>user
...
<|im_end|>
<|im_start|>assistant
...
<|im_end|>

This format keeps the training data aligned with the tokenizer and chat template expected by the base model.

4. Training Infrastructure

Training was run on Modal using the project app and volumes configured for this model:

Resource	Name
Training app	ngen4-ow-4b-thinking-train
Data volume	ngen4-ow-4b-thinking-data
Adapter volume	ngen4-ow-4b-thinking-adapter
Export app	ngen4-ow-4b-thinking-export
Export volume	ngen4-ow-4b-thinking-export

The training image was based on:

nvidia/cuda:12.8.1-devel-ubuntu22.04
Python 3.12

Key Python libraries included:

Library	Purpose
torch	Model execution and training
transformers	Tokenizer, model loading, Trainer
datasets	JSONL training data loading
peft	LoRA adapter creation and merge
bitsandbytes	4-bit NF4 base model loading
accelerate	Device and training support
huggingface_hub / hf_transfer	HF model and dataset access

5. Training Method

The model was trained with supervised fine-tuning using Hugging Face Trainer. The base model was loaded in 4-bit quantized form using BitsAndBytes:

Setting	Value
Quantization	4-bit
Quant type	nf4
Compute dtype	bfloat16
Double quantization	enabled
Attention implementation	sdpa
Gradient checkpointing	enabled
Use cache	disabled during training

The base model was prepared for k-bit training through PEFT's prepare_model_for_kbit_training. This is important because quantized base models are not trained directly like full-precision models. Instead, the base weights remain effectively frozen while LoRA adapter weights receive gradient updates.

6. Run Hyperparameters

The final run manifest shown in the training screenshot recorded:

Hyperparameter	Value
Base model	TNSA/NGen-4-OW-4B-Thinking
LoRA rank	64
LoRA alpha	128
Learning rate	2e-6
Per-device batch	2
Gradient accumulation	4
Recorded effective batch	64
Max steps	150
Training examples	2,141,104
Final logged training loss	about 2.205
Training wall time in screenshot	about 1:54:58

The screenshot also showed the training loop completed all 150/150 steps, then saved the final adapter and committed the output to the Modal volume.

Batch Size Note

The run manifest reports effective_batch: 64. The visible training parameters are per_gpu_batch: 2 and grad_accum: 4. In the training script, the manifest computes effective batch as:

per_gpu_batch * grad_accum * 8

That records an 8-way multiplier. If the run is executed on a single GPU configuration, the practical optimizer batch from the visible values is:

2 * 4 = 8 examples per optimizer step

The report keeps the manifest value because it is the value saved by the run, but this difference should be cleaned up in the training script for future runs if single-GPU and multi-GPU jobs are both supported.

7. LoRA Configuration

The model was adapted with LoRA rather than full fine-tuning. LoRA inserts small trainable low-rank matrices into selected linear layers. During training, only these adapter matrices are updated; the original base model weights remain frozen.

The run used:

LoRA Setting	Value
Rank r	64
Alpha	128
Dropout	0.0
Bias	none
Task type	CAUSAL_LM

The target modules were:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

These targets cover the most important transformation points in a decoder-only language model:

Target	Function	Why it was trained
q_proj	Query projection in attention	Helps the model learn what to attend to for Arabic questions and instructions
k_proj	Key projection in attention	Adjusts how source tokens are represented for retrieval inside attention
v_proj	Value projection in attention	Changes what content is carried forward once attention is assigned
o_proj	Attention output projection	Lets the adapted attention result be integrated back into the residual stream
gate_proj	MLP gate projection	Controls which feed-forward features activate
up_proj	MLP expansion projection	Allows new task/domain features to be represented in the hidden expansion
down_proj	MLP compression projection	Maps adapted MLP features back into the model hidden size

8. Why These Layers Were Changed

The selected modules balance adaptation strength and stability.

Attention projections (q_proj, k_proj, v_proj, o_proj) are responsible for token-to-token interaction. For a QA-heavy Arabic dataset, the model must learn patterns such as question intent, answer grounding, semantic matching, and multilingual phrasing. Adapting attention projections helps the model change how it routes information across the context without rewriting the entire model.

The MLP projections (gate_proj, up_proj, down_proj) are responsible for feature transformation after attention. These layers are where many factual, stylistic, and reasoning transformations are expressed. Training them with LoRA gives the model capacity to adapt answer style, domain conventions, and Arabic QA patterns.

The combination of attention and MLP LoRA is stronger than attention-only LoRA, because the model can both attend differently and transform the attended information differently. At the same time, it is much cheaper and safer than full fine-tuning, because the base model remains frozen.

9. Layers Intentionally Not Changed

The following parts were intentionally not directly trained:

Component	Reason
Base dense weights	Preserves base model knowledge and reduces training cost
Token embeddings	Avoids destabilizing the tokenizer vocabulary and rare-token behavior
Layer norms	Keeps normalization behavior stable during low-rank adaptation
Router/gating control outside LoRA targets	Reduces risk of routing collapse or expert imbalance
Output head	Avoids unnecessary large adapter memory and preserves base logits geometry

The manifest records:

"router_frozen": true

Freezing router behavior is especially important for architectures with specialized routing or hybrid blocks. Router changes can create sharp behavior shifts, unstable expert use, and worse generalization if the dataset is narrow. For this run, the goal was domain adaptation rather than architectural re-routing, so the router was kept frozen.

10. Sequence Length and Long Context

The training script was configured for long context:

max_seq_length = 131072
sample_max_new = 65536

The GGUF metadata later showed:

qwen35.context_length = 262144

This means the base model and exported GGUF retain a very long context configuration. For practical serving, the actual usable context depends on the inference backend, GPU VRAM, quantization, KV cache settings, and request length. The vLLM backend scaffold created separately uses a conservative --max-model-len 32768 by default so deployment is more likely to fit on available hardware.

11. Training Flow

The end-to-end training flow was:

1. Pull the Hugging Face dataset.
2. Normalize rows into chat messages.
3. Write the local JSONL dataset.
4. Upload the JSONL to the Modal data volume.
5. Load the base model in 4-bit NF4.
6. Attach LoRA adapters to selected projection layers.
7. Tokenize the JSONL dataset into causal language modeling sequences.
8. Train for the configured number of optimizer steps.
9. Save the final adapter to /out/final.
10. Commit the adapter volume.
11. Merge the adapter into the base model.
12. Save merged Hugging Face weights.
13. Convert merged model to f16 GGUF.
14. Patch GGUF block metadata if needed.
15. Quantize to Q4_K_M.
16. Download the final GGUF locally.

12. Export and GGUF Conversion

After training, the adapter was merged into the base model using PEFT's adapter-loading path. This was required because the base model carries PEFT or adapter-related metadata, and the older manual merge path caused a Transformers save-time adapter-state error.

The corrected export path does the following:

AutoPeftModelForCausalLM.from_pretrained(adapter_path)
merge_and_unload()
save_pretrained(merged_dir)
convert_hf_to_gguf.py --outtype f16
llama-quantize Q4_K_M

The f16 GGUF was created first:

/export/gguf/ngen4-ow-4b-thinking-f16.gguf

The quantized GGUF was then created:

/export/gguf/ngen4-ow-4b-thinking-Q4_K_M.gguf

The final local file is:

outputs/gguf/ngen4-ow-4b-thinking-Q4_K_M.gguf

13. GGUF Metadata Patch

During conversion, the GGUF converter reported:

qwen35.block_count = 33

The exporter patched this to:

qwen35.block_count = 32

This patch was necessary because the converter counted an extra prediction or auxiliary layer in a way that can confuse GGUF runtimes. The patch prevents runtime errors where the loader expects weights for a nonexistent layer.

The quantized GGUF was checked again after quantization and already had:

qwen35.block_count = 32

14. Exported Model Metadata

The export logs identified the architecture as:

Qwen3_5ForCausalLM
general.architecture = qwen35
general.size_label = 4.2B

Important GGUF metadata included:

Field	Value
Architecture	qwen35
Size label	4.2B
Block count	32
Context length	262144
Embedding length	2560
Feed-forward length	9216
Attention heads	16
KV heads	4
Key length	256
Value length	256
Quantization	Q4_K_M
Quantized size	about 2.71 GB

15. Observed Training Result

The training screenshot shows the run reached completion:

150/150 steps
train_loss: 2.205
epoch: 0.0005605

The very small epoch value is expected because the dataset is extremely large relative to the number of steps. With 2,141,104 examples and only 150 training steps, the run is a short adaptation pass rather than a full epoch over the dataset.

This is useful as a first successful adapter build because it verifies:

1. Dataset normalization works.
2. Modal upload works.
3. The model can be loaded in 4-bit mode.
4. LoRA modules can be attached and trained.
5. The adapter can be saved.
6. The adapter can be merged and exported.
7. The GGUF can be quantized and downloaded.

It should not be treated as a fully converged training run. A longer training run, validation set, and benchmark suite are needed before making quality claims.

16. Why This Training Strategy Was Reasonable

Full fine-tuning a 4B-class model requires more memory, more time, and greater risk of catastrophic forgetting. LoRA gives a strong middle path:

• The base model remains intact.
• Training is cheaper.
• The adapter is small and easy to save.
• The adapter can be merged later for deployment.
• The target modules give enough coverage for meaningful behavior change.

The ArabicQA dataset is large and domain-specific. A short LoRA run is a good first pass for teaching the model dataset style and Arabic QA response patterns without overwriting general reasoning ability.

17. Limitations

The run has several limitations:

1. The final run was only 150 steps.
2. The dataset was large, so the model saw only a small fraction of one epoch.
3. No validation metrics are recorded in the screenshot.
4. The final loss alone is not enough to evaluate answer quality.
5. The effective batch value in the manifest should be corrected for future single-GPU runs.
6. The report does not claim benchmark improvements without evaluation.
7. ArabicQA rows were normalized automatically, so spot checks should be done for formatting and answer quality.

18. Recommended Next Steps

Recommended follow-up work:

1. Run a longer training job, for example 750 to 2000 steps.
2. Add a held-out validation split from ArabicQA.
3. Log validation loss every fixed number of steps.
4. Add Arabic QA evaluation prompts.
5. Add multilingual sanity prompts to ensure English and general reasoning did not regress.
6. Fix the effective_batch manifest calculation so it reflects the actual GPU count.
7. Save the exact Modal app run URL and run ID in the manifest.
8. Save a copy of the export manifest locally with the GGUF.
9. Test the GGUF in the vLLM RAG backend and with a direct local runtime.

19. Reproducibility Commands

The dataset preparation and training command used for this family of run was:

python scripts/ngen_thinking_cli.py train-hf riotu-lab/ArabicQA_2.1M --split train --max-steps 750

The screenshoted completed run itself records:

max_steps = 150

The GGUF export command was:

python scripts/ngen_thinking_cli.py export --checkpoint final --quant Q4_K_M --download

The final GGUF file:

C:\Users\NACHIKETH\Desktop\NGen-4.1\NGen-4-OW-4B-Thinking\outputs\gguf\ngen4-ow-4b-thinking-Q4_K_M.gguf

20. Conclusion

The training pipeline successfully adapted TNSA/NGen-4-OW-4B-Thinking with a LoRA adapter over selected attention and MLP projection layers, using riotu-lab/ArabicQA_2.1M as the training corpus. The adapter was saved, merged, converted to GGUF, patched for correct Qwen block metadata, quantized to Q4_K_M, and downloaded locally.

The most important architectural decision was to update projection layers instead of the full model. This gave the run enough flexibility to learn Arabic QA behavior while keeping training cost and regression risk low. The router and core base weights remained frozen to preserve stability.

This should be considered a successful first training and export pass. The next step is evaluation: compare the base model and the tuned GGUF on Arabic QA, general chat, reasoning, and RAG tasks before positioning it as a production model.