nano-llm-ft

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on an unknown dataset.

Model description

This model is a fine-tuned version of an open-source Nano LLM designed for general-purpose text generation tasks. It has been adapted using a curated synthetic dataset focused on clarity, neutrality, and regionally relevant topics, primarily related to the United Kingdom and Saudi Arabia. The goal of this fine-tune is to create a lightweight, efficient, and deployable model suitable for educational, research, and small-scale production uses on limited hardware.

Key Features

Optimized for short to medium-form text generation

Data emphasizes clear explanations, helpful tone, and non-biased content

Trained on curated synthetic pairs (input, output)

Suitable for mobile, edge devices, and Colab environments

Fast inference due to Nano-scale architecture

Intended uses & limitations

Text generation

Explanation, paragraph writing, and educational responses

Lightweight LLM experimentation

Fine-tuning tutorials and teaching

Local/offline LLM deployments

Chatbots, assistants, and micro-apps

Not intended for:

High-stakes decision-making

Medical, legal, or financial advice

Producing factual content without verification

Training and evaluation data

How It Was Trained

Training Framework

Framework: Hugging Face Transformers + Datasets

Training Environment: Google Colab (Free Tier)

Precision: FP16 (or BF16 depending on GPU)

Optimizer: AdamW

Training Strategy: Full fine-tuning or LoRA (based on your setup)

Training Arguments (Optimized Version)

training_args = TrainingArguments( output_dir="./nano-llm-ft", per_device_train_batch_size=8, per_device_eval_batch_size=8, gradient_accumulation_steps=1, num_train_epochs=3, learning_rate=5e-5, fp16=True, logging_steps=20, save_steps=1500, eval_steps=1500, save_total_limit=2, report_to="none" )

Dataset Format

The dataset was stored in Hugging Face–compatible DatasetDict:

DatasetDict({ train: Dataset({ features: ['input', 'output'], num_rows: XXXX }) validation: Dataset({ features: ['input', 'output'], num_rows: XXXX }) })

Each sample contains:

{ "input": "Write a short paragraph about renewable energy.", "output": "Renewable energy comes from..." }

Dataset Details

Size: 1,000+ samples

Type: Synthetic instructional dataset

Safety Filters:

No harmful, biased, or inappropriate content

Neutral and regionally accurate (UK + Saudi Arabia)

Purpose:

Teach the model structured text generation

Improve clarity, coherence, and helpfulness

Performance

Evaluated with validation loss / perplexity

Optimized for coherence, neutral tone, and helpful explanations

Not optimized for benchmarks like MMLU or MT-Bench (optional future work)

Limitations

May generate inaccurate facts — verify before use

Not a replacement for large general-purpose LLMs

Limited reasoning depth due to compact architecture

No guarantee of culturally perfect outputs for UK/Saudi topics

Training data is synthetic; diversity is limited

Ethical Considerations

Dataset avoids political, sexual, hateful, violent, or harmful content

Designed to promote clarity, neutrality, and fairness

Not trained on private or sensitive real-world data

Should not be used in contexts requiring expert knowledge or real-world risk

Training procedure

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("your-username/nano-llm-ft") tokenizer = AutoTokenizer.from_pretrained("your-username/nano-llm-ft")

prompt = "Write a short paragraph about solar energy."

inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate(**inputs, max_length=150)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Model Creator

This model was prepared and fine-tuned by Abdur Rahman in a Google Colab environment.

Citation

@misc{nano_llm_ft_2025, title={Fine-Tuned Nano LLM for Text Generation}, author={Your Name}, year={2025}, publisher={Hugging Face}, howpublished={\url{https://huggingface.co/Abdurrahmanesc/nano-llm-ft}}, }

Model Tags (Recommended)

tags:

• text-generation
• nano-llm
• fine-tuned
• transformers
• synthetic-dataset
• educational
• lightweight
• colab

Framework versions

• PEFT 0.18.0
• Transformers 4.57.1
• Pytorch 2.8.0+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1