Update README.md
Browse files
README.md
CHANGED
|
@@ -2,16 +2,18 @@
|
|
| 2 |
|
| 3 |
This repository contains the implementation of a lightweight, modified version of the GPT architecture **Leap-0** trained from scratch using FineWeb-Edu, an open-source dataset. The project demonstrates the design, training, and optimization of a custom natural language model on local hardware.
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
## Features
|
| 6 |
- **Custom GPT Architecture**: A miniaturized version of the GPT model tailored for efficient training on limited hardware.
|
| 7 |
- **Local Training**: Complete model training executed on local resources, enabling cost-effective development.
|
| 8 |
- **Open-Source Datasets**: Trained using publicly available FineWeb-Edu dataset to ensure accessibility and reproducibility.
|
| 9 |
- **Scalable Design**: Architecture optimized for experimentation and scalability while maintaining resource efficiency.
|
| 10 |
|
| 11 |
-
|
| 12 |
-
<img src="LLM.drawio.png" alt="Description of the image" width="300">
|
| 13 |
-
<p><strong>Figure 1: Architecture of Leap</p>
|
| 14 |
-
</div>
|
| 15 |
|
| 16 |
## Implementation Details
|
| 17 |
1. **Model Architecture**
|
|
|
|
| 2 |
|
| 3 |
This repository contains the implementation of a lightweight, modified version of the GPT architecture **Leap-0** trained from scratch using FineWeb-Edu, an open-source dataset. The project demonstrates the design, training, and optimization of a custom natural language model on local hardware.
|
| 4 |
|
| 5 |
+
<div align="center">
|
| 6 |
+
<img src="LLM.drawio.png" alt="Description of the image" width="300">
|
| 7 |
+
<p><strong>Figure 1: Architecture of Leap</p>
|
| 8 |
+
</div>
|
| 9 |
+
|
| 10 |
## Features
|
| 11 |
- **Custom GPT Architecture**: A miniaturized version of the GPT model tailored for efficient training on limited hardware.
|
| 12 |
- **Local Training**: Complete model training executed on local resources, enabling cost-effective development.
|
| 13 |
- **Open-Source Datasets**: Trained using publicly available FineWeb-Edu dataset to ensure accessibility and reproducibility.
|
| 14 |
- **Scalable Design**: Architecture optimized for experimentation and scalability while maintaining resource efficiency.
|
| 15 |
|
| 16 |
+
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
## Implementation Details
|
| 19 |
1. **Model Architecture**
|