AndreasBis/TinyLlamaTalos-1B

TinyLlama UltraChat Fine-Tune

This repository contains a full fine-tuning run of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on HuggingFaceH4/ultrachat_200k, trained with tinyllama-training.ipynb and saved under tinyllama-ultrachat-rslora/.

Notebook Highlights

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Dataset: HuggingFaceH4/ultrachat_200k
• Full run: quick_test = False
• Conversations are expanded into assistant-turn training rows
• Context length: 512
• LoRA variant: Rank-Stabilized LoRA
• LoRA rank: 128
• LoRA alpha: 256
• LoRA dropout: 0.04
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Precision: bf16
• Gradient checkpointing: enabled
• Per-device train batch size: 256
• Gradient accumulation steps: 4
• Effective batch size: 1024
• Learning rate: 1e-4
• Epochs: 1.0
• Eval subset during training: 4096
• Output directory: ./tinyllama-ultrachat-rslora

Run Summary

Item	Value
Processed training rows	657,771
Processed evaluation rows	73,150
Optimizer steps	643
Epochs	1.0
Effective batch size	1024
Train runtime	6:32:07
Eval runtime	0:00:38.21
Train samples / second	27.958
Train steps / second	0.027
Eval samples / second	107.196
Eval steps / second	0.419
Final train loss	1.0671
Final eval loss	1.0621
Final perplexity	2.8925
Adapter output	./tinyllama-ultrachat-rslora/adapter
Merged output	./tinyllama-ultrachat-rslora/merged

Loss Table

Step	Training Loss	Validation Loss
32	1.088547	1.108700
64	1.098695	1.115909
96	1.099704	1.114013
128	1.093774	1.107312
160	1.087225	1.102051
192	1.082110	1.098298
224	1.080011	1.094395
256	1.075894	1.090381
288	1.070766	1.085714
320	1.064308	1.081495
352	1.064451	1.078283
384	1.058922	1.074668
416	1.055673	1.071584
448	1.050642	1.068871
480	1.049796	1.066419
512	1.047866	1.064614
544	1.046279	1.063277
576	1.040401	1.062529
608	1.043520	1.062152
640	1.045642	1.062093
643	1.045642	1.062105

Output Tree

tinyllama-ultrachat-rslora/
|-- adapter/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   `-- training_args.bin
|-- checkpoint-608/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- optimizer.pt
|   |-- rng_state.pth
|   |-- scheduler.pt
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   |-- trainer_state.json
|   `-- training_args.bin
|-- checkpoint-640/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- optimizer.pt
|   |-- rng_state.pth
|   |-- scheduler.pt
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   |-- trainer_state.json
|   `-- training_args.bin
|-- checkpoint-643/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- optimizer.pt
|   |-- rng_state.pth
|   |-- scheduler.pt
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   |-- trainer_state.json
|   `-- training_args.bin
|-- merged/
|   |-- chat_template.jinja
|   |-- config.json
|   |-- generation_config.json
|   |-- model.safetensors
|   |-- tokenizer.json
|   `-- tokenizer_config.json
|-- all_results.json
|-- eval_results.json
|-- train_results.json
`-- trainer_state.json

Artifact Highlights

• adapter/ contains the saved PEFT adapter weights and tokenizer files.
• merged/ contains the fully merged Hugging Face model, including model.safetensors.
• checkpoint-608/, checkpoint-640/, and checkpoint-643/ are the saved training checkpoints retained by Trainer.
• Root-level JSON files contain the final train metrics, eval metrics, aggregate results, and trainer state.