README.md

---
library_name: transformers
datasets:
- sleeping-ai/TEKGEN-Wiki
- lewiswatson/flag_injected_wikipedia_sample
- zekebass/tensor-logic-wikipedia
- Samiro1/Wiki-Corpus-Chat
- JoseHuman/dtConceptosProgramacion
- Parleatacoeur/leyesperuanasactualizadas
- beyarkay/pre-1950s-text
- emreisik/news
- ae5115242430e13/RSL_Maran
- lgrobol/openminuscule
- Dorian2B/french-history-5K
- MultivexAI/Everyday-Language-Corpus-deduped
- yashm/phrases
- 16dvnk/Spirit_Kings_Golden_Textbook
language:
- en
- es
base_model:
- openai-community/gpt2
---

# Model Card for Fu01978/gpt2-mega-wiki-logic
This model is a fine-tuned version of GPT-2 (Base) trained on a diverse "mini-pile" of 14 datasets ranging from French history and Peruvian law to programming concepts and esoteric texts. It is designed to be a versatile text-completer that can shift styles based on the input prompt.

## Model Details

### Model Description
This model was created to explore the limits of "Knowledge Density" in small language models. By mixing high-fact density data (Wikipedia, News) with specialized technical data (Programming, Law) and historical texts, the model acts as a "Jack-of-all-trades" completion engine.
- **Developed by:** Fu01978
- **Model type:** Causal Language Model (Transformer Decoder)
- **Language(s) (NLP):** English (Primary), Spanish (Programming/Law)
- **License:** MIT
- **Finetuned from model:** openai-community/gpt2

## Uses
### Direct Use
The model is best used for **constrained text completion**. It excels when given a clear context or "trigger phrase" to help it navigate its diverse training data.

### Out-of-Scope Use
- **Fact-Checking:** This model should **not** be used as a primary source of historical or legal facts.
- **High-Stakes Advice:** Do not use for legal or medical decision-making.

## Bias, Risks, and Limitations
- **Temporal Hallucinations:** Because the model was trained on 14 vastly different time periods simultaneously, it frequently mixes historical facts (e.g., placing the 1944 Battle of the Bulge in the 1898 Spanish-American War).
- **Small Context Window:** As a GPT-2 base model, it has a limited context window and may lose coherence in long-form generation.

### Recommendations
Users should use Beam Search ($num\_beams \ge 3$) and a high Repetition Penalty ($1.5+$) to prevent the model from entering logic loops or mixing unrelated datasets.

## How to Get Started with the Model
Use the code below to get started with the model.
```py
from transformers import pipeline

generator = pipeline("text-generation", model="Fu01978/gpt2-mega-wiki-logic")
prompt = "In the Python programming language, a decorator is"
print(generator(prompt, max_new_tokens=50, repetition_penalty=1.5, num_beams=5)[0]['generated_text'])
```

## Training Details

### Training Data
The model was trained on a combined pool of 123233 rows.

### Training Metrics

| Step | Training Loss |
| :--- | :-----------: |
| 100  | 3.0273        |
| 200  | 2.8873        |
| 300  | 2.7780        |
| 400  | 2.8189        |
| 500  | 2.8553        |

### Training Procedure
#### Training Hyperparameters
- **Steps:** 500
- **Batch Size:** 4 (with 4 Gradient Accumulation steps)
- **Learning Rate:** 4e-5
- **Precision:** fp16 (Mixed Precision)
- **Optimizer:** AdamW with Weight Decay (0.01)

## Technical Specifications

### Model Architecture and Objective
- **Architecture:** Standard GPT-2 Transformer Decoder.
- **Objective:** Causal Language Modeling (Next-token prediction).
- **Parameters:** 124 Million.