Update README.md

README.md

@@ -1,50 +1,71 @@
 ---
-        language: en
-        license: mit
-        tags:
-        - pytorch
-        - causal-lm
-        - language-model
-        - flash-attention
-        ---
+language: en
+license: mit
+tags:
+  - pytorch
+  - causal-lm
+  - language-model
+  - flash-attention
+datasets:
+  - Salesforce/wikitext
+pipeline_tag: question-answering
+---
+
+# PurelyUnfunctionalAI/GibberishGPT
+
+A lightweight decoder-only transformer language model trained with Flash Attention on the WikiText dataset.
 
-        # PurelyUnfunctionalAI/GibberishGPT
+## Model Details
 
-        This is a language model trained with Flash Attention. The model is based on a decoder-only transformer architecture.
+- **Model Type:** Causal Language Model
+- **Architecture:** Decoder-only Transformer
+- **Embedding Size:** 512
+- **Hidden Layers:** 8
+- **Attention Heads:** 8
+- **Context Length:** 512
+- **Flash Attention:** Enabled
+- **Training Data:** Salesforce/wikitext
 
-        ## Model Details
+## Usage
 
-        - **Model Type:** Causal Language Model
-        - **Embedding Size:** 512
-        - **Hidden Layers:** 8
-        - **Attention Heads:** 8
-        - **Context Length:** 512
-        - **Flash Attention:** Enabled
+```python
+import torch
+import tiktoken
+from transformers import AutoModelForCausalLM
 
-        ## Usage
+# Load the tokenizer
+tokenizer = tiktoken.get_encoding("gpt2")
 
-        ```python
-        import tiktoken
-        from transformers import AutoModelForCausalLM
+# Load the model
+model = AutoModelForCausalLM.from_pretrained("PurelyUnfunctionalAI/GibberishGPT")
 
-        # Load the tokenizer
-        tokenizer = tiktoken.get_encoding("gpt2")
+# Encode input
+input_text = "Your prompt here"
+input_ids = tokenizer.encode(input_text)
+input_tensor = torch.tensor([input_ids], dtype=torch.long)
 
-        # Load the model
-        model = AutoModelForCausalLM.from_pretrained("PurelyUnfunctionalAI/GibberishGPT")
+# Generate
+output = model.generate(input_tensor, max_length=100)
+generated_text = tokenizer.decode(output[0].tolist())
+print(generated_text)
+```
 
-        # Encode input
-        input_text = "Your prompt here"
-        input_ids = tokenizer.encode(input_text)
-        input_tensor = torch.tensor([input_ids], dtype=torch.long)
+# Limitations
 
-        # Generate
-        output = model.generate(input_tensor, max_length=100)
-        generated_text = tokenizer.decode(output[0].tolist())
-        print(generated_text)
-        ```
+- The model has a context length of 512 tokens
+- It was trained on WikiText data which may not cover specialized domains
+- As a lightweight model, it may not perform as well as larger LLMs on complex tasks
 
-        ## License
+# Citation
+If you use this model in your research, please cite:
 
-        This model is available under the MIT License.
-        
+```
+@misc{GibberishGPT,
+  author = {Gathara, Michael},
+  title = {GibberishGPT: A Lightweight Language Model with Flash Attention},
+  year = {2025},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://huggingface.co/PurelyUnfunctionalAI/GibberishGPT}}
+}
+```