|
|
--- |
|
|
language: |
|
|
- tr |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
tags: |
|
|
- llama-3 |
|
|
- turkish |
|
|
- tiny-llama |
|
|
- scratch-build |
|
|
datasets: |
|
|
- TFLai/Turkish-Alpaca |
|
|
metrics: |
|
|
- loss |
|
|
model_type: llama |
|
|
--- |
|
|
|
|
|
# Llama-TR-Mini (9M Parameters) |
|
|
|
|
|
Llama-TR-Mini is an experimental, ultra-lightweight Turkish language model with **9.3 million parameters**, trained from scratch using the Llama 3 architecture. |
|
|
|
|
|
This project was developed to explore the limits of small-scale language modeling and to understand the end-to-end pre-training/fine-tuning pipeline on consumer-grade hardware (Apple Silicon). |
|
|
|
|
|
## Model Specifications |
|
|
- **Architecture:** Llama 3 |
|
|
- **Parameters:** 9,343,232 |
|
|
- **Hidden Size:** 256 |
|
|
- **Intermediate Size:** 512 |
|
|
- **Number of Layers:** 8 |
|
|
- **Attention Heads:** 8 |
|
|
- **Vocabulary Size:** 5,000 (Custom Turkish Tokenizer) |
|
|
- **Training Epochs:** 30 |
|
|
- **Device:** MacBook Pro (MPS - Metal Performance Shaders) |
|
|
|
|
|
## Training Data |
|
|
The model was trained on the [Turkish-Alpaca](https://huggingface.co/datasets/TFLai/Turkish-Alpaca) dataset, which contains approximately 52K instruction-following pairs translated into Turkish. |
|
|
|
|
|
## Intended Use & Limitations |
|
|
**Important Note:** Due to its extremely small size (9M parameters), this model is prone to significant hallucinations and may produce nonsensical or repetitive outputs. |
|
|
|
|
|
- **Purpose:** Educational purposes, understanding LLM mechanics. |
|
|
- **Not Suited For:** Production environments, factual information retrieval, or complex reasoning tasks. |
|
|
- **Format:** Optimized for the Llama 3 Instruct template (`<|start_header_id|>user<|end_header_id|>`). |
|
|
|
|
|
## How to Use |
|
|
You can load this model using the `transformers` library: |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
model_id = "k12tr/mini-tr-9m" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = AutoModelForCausalLM.from_pretrained(model_id) |
|
|
|
|
|
prompt = "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nTürkiye'nin başkenti neresidir?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
output = model.generate(**inputs, max_new_tokens=50, temperature=0.1, repetition_penalty=1.5) |
|
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True)) |
