| | --- |
| | language: en |
| | license: apache-2.0 |
| | library_name: transformers |
| | tags: |
| | - text-generation |
| | - pytorch |
| | - gpt |
| | - language-model |
| | --- |
| | |
| | # tinyMind |
| |
|
| | This is a small transformer language model trained from scratch with approximately 17,731,328 parameters. |
| |
|
| | ## Model Details |
| |
|
| | - **Architecture**: GPT-style transformer |
| | - **Parameters**: ~17M |
| | - **Layers**: 6 |
| | - **Attention Heads**: 8 |
| | - **Embedding Dimension**: 256 |
| | - **Max Sequence Length**: 512 |
| | - **Vocabulary Size**: 50257 |
| |
|
| | ## Training Data |
| |
|
| | The model was trained on a diverse mixture of high-quality text data including: |
| | - OpenWebText |
| | - Wikipedia articles |
| | - BookCorpus |
| | - Other curated text sources |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | from transformers import GPT2TokenizerFast, AutoModelForCausalLM |
| | |
| | tokenizer = GPT2TokenizerFast.from_pretrained("HenrySentinel/tinyMind") |
| | model = AutoModelForCausalLM.from_pretrained("HenrySentinel/tinyMind") |
| | |
| | # Generate text |
| | input_text = "The key to artificial intelligence is" |
| | input_ids = tokenizer.encode(input_text, return_tensors="pt") |
| | output = model.generate(input_ids, max_length=100, temperature=0.8, do_sample=True) |
| | generated_text = tokenizer.decode(output[0], skip_special_tokens=True) |
| | print(generated_text) |
| | ``` |
| |
|
| | ## Training Details |
| |
|
| | - **Optimizer**: AdamW with cosine learning rate scheduling |
| | - **Learning Rate**: 0.001 |
| | - **Batch Size**: 8 |
| | - **Sequence Length**: 512 |
| | - **Epochs**: 3 |
| | - **Gradient Clipping**: 1.0 |
| |
|
| | ## Limitations |
| |
|
| | This is a small model designed for experimentation and learning. It may: |
| | - Generate inconsistent or factually incorrect content |
| | - Have limited knowledge compared to larger models |
| | - Require careful prompt engineering for best results |
| |
|
| | ## License |
| |
|
| | Apache 2.0 |
| |
|
