Initial commit with model and tokenizer

README.md

@@ -18,18 +18,39 @@ A small Transformer Decoder model trained from scratch on the Tiny Shakespeare d
 - Dataset: Tiny Shakespeare
 - Epochs: 5
 - Learning Rate: 0.0003
+- Batch Size: 64
+- Block Size: 128
 - Optimizer: AdamW
-- Loss: CrossEntropyLoss
-- wandb project: [Link to wandb project](https://wandb.ai/honcharova-de-hannover/LanguageModel_Project?nw=nwuserhoncharovade)
+- Loss Function: CrossEntropyLoss
+- Dropout Rate: 0.1
+- Embedding Dimension: 256
+- Number of Layers: 4
+- Number of Attention Heads: 4
 
 ## Usage
+To use this model, simply load it using the following code:
+
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
+# Load the model and tokenizer
 model = AutoModelForCausalLM.from_pretrained("NataliaH/tiny_shakespeare_transformer")
 tokenizer = AutoTokenizer.from_pretrained("NataliaH/tiny_shakespeare_transformer")
 
+# Encode input text
 inputs = tokenizer("Once upon a time", return_tensors="pt")
 outputs = model.generate(**inputs)
 print(tokenizer.decode(outputs[0]))
 ```
+
+## Model Architecture
+This model is a Transformer Decoder-based architecture, optimized for text generation. 
+It was trained on the Tiny Shakespeare dataset to generate Shakespeare-like text.
+
+## Training Process
+- Training was performed for 5 epochs.
+- The model uses AdamW optimizer with a learning rate of 0.0003.
+- Dropout rate during training was set to 0.1 to reduce overfitting.
+
+## License
+This model is released under the MIT License.