Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
-
This repository contains the implementation of a lightweight, modified version of the GPT architecture **
|
| 4 |
|
| 5 |
## Features
|
| 6 |
- **Custom GPT Architecture**: A miniaturized version of the GPT model tailored for efficient training on limited hardware.
|
|
@@ -10,7 +10,7 @@ This repository contains the implementation of a lightweight, modified version o
|
|
| 10 |
|
| 11 |
<div align="center">
|
| 12 |
<img src="LLM.drawio.png" alt="Description of the image" width="300">
|
| 13 |
-
<p><strong>Figure 1: Architecture of
|
| 14 |
</div>
|
| 15 |
|
| 16 |
## Implementation Details
|
|
@@ -19,7 +19,7 @@ This repository contains the implementation of a lightweight, modified version o
|
|
| 19 |
- Incorporates modifications to parameter scaling to suit resource-constrained environments.
|
| 20 |
|
| 21 |
2. **Training**
|
| 22 |
-
- Training executed locally on NVIDIA GeForce RTX
|
| 23 |
|
| 24 |
3. **Testing**
|
| 25 |
- A simple Streamlit UI created for testing generation capability of the model.
|
|
@@ -66,26 +66,4 @@ This repository contains the implementation of a lightweight, modified version o
|
|
| 66 |
- Dependencies listed in `requirements.txt`
|
| 67 |
- **Note**: Different OS support different versions of PyTorch/Tensorflow to use CUDA (local GPU). Install only after verifying for your OS.
|
| 68 |
|
| 69 |
-
## Usage
|
| 70 |
-
1. Clone the repository:
|
| 71 |
-
```bash
|
| 72 |
-
git clone https://github.com/pulkundwar29/LangPWT
|
| 73 |
-
cd LangPWT
|
| 74 |
-
```
|
| 75 |
-
2. Create and activate a virtual environment:
|
| 76 |
-
```bash
|
| 77 |
-
venv env
|
| 78 |
-
env\scripts\activate
|
| 79 |
-
```
|
| 80 |
-
3. Install dependencies:
|
| 81 |
-
```bash
|
| 82 |
-
pip install -r requirements.txt
|
| 83 |
-
```
|
| 84 |
-
4. Run the training file **trainpwt.py**
|
| 85 |
-
5. Run the streamlit file: **trial_pwt.py**
|
| 86 |
-
6. Enter your prompt and hit the Generate button.
|
| 87 |
|
| 88 |
-
<div align="center">
|
| 89 |
-
<img src="ex1.png" alt="example text">
|
| 90 |
-
<p><strong>Figure 2: Example of Text Generated using LangPWT</p>
|
| 91 |
-
</div>
|
|
|
|
| 1 |
+
# Leap-0
|
| 2 |
|
| 3 |
+
This repository contains the implementation of a lightweight, modified version of the GPT architecture **Leap-0** trained from scratch using FineWeb-Edu, an open-source dataset. The project demonstrates the design, training, and optimization of a custom natural language model on local hardware.
|
| 4 |
|
| 5 |
## Features
|
| 6 |
- **Custom GPT Architecture**: A miniaturized version of the GPT model tailored for efficient training on limited hardware.
|
|
|
|
| 10 |
|
| 11 |
<div align="center">
|
| 12 |
<img src="LLM.drawio.png" alt="Description of the image" width="300">
|
| 13 |
+
<p><strong>Figure 1: Architecture of Leap</p>
|
| 14 |
</div>
|
| 15 |
|
| 16 |
## Implementation Details
|
|
|
|
| 19 |
- Incorporates modifications to parameter scaling to suit resource-constrained environments.
|
| 20 |
|
| 21 |
2. **Training**
|
| 22 |
+
- Training executed locally on NVIDIA GeForce RTX 4500 ada 24GB GPU, leveraging PyTorch.
|
| 23 |
|
| 24 |
3. **Testing**
|
| 25 |
- A simple Streamlit UI created for testing generation capability of the model.
|
|
|
|
| 66 |
- Dependencies listed in `requirements.txt`
|
| 67 |
- **Note**: Different OS support different versions of PyTorch/Tensorflow to use CUDA (local GPU). Install only after verifying for your OS.
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
|
|
|
|
|
|
|
|
|
|
|