Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+GPT-2_scratch.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,10 @@
----
-license: mit
----
+# GPT-2 From Scratch
+This repo contains the custom implementation of [GPT-2](https://openai.com/index/gpt-2-1-5b-release/). This implementation was created following the playlist ["LLMs from Scratch" by Vizuara](https://youtube.com/playlist?list=PLPTV0NXA_ZSgsLAr8YCgCwhPIJNNtexWu&si=eksVKcxWNTVzJRUa). The original course has all the code in an ```ipynb``` notebook.
+
+As I was following along the course, I have modularized the said implemenation and have also created an OpenAI weights compatible implementation (not covered in the course). In the course, the OpenAI's tensorflow weights were preprocessed and then injected into the custom implementation, whereas in my OpenAI compatible implementation (located in [compat folder](compat/)), I have used a larger (combined) K,Q,V matrix that aligns with the OpenAI weights, whereas the custom implementation (in the course) deals with this separately.
+
+The scripts used to map the OpenAI weights (PyTorch weights downloaded from hugging face) to my OpenAI compatible implementation is [mapper_v3.py](./mapper_v3.py).
+
+![GPT-2 Architecture](./GPT-2_scratch.jpg)
+
+Please check the [Github Repo](https://github.com/B4S1C-Coder/GPT-2-from-scratch) for usage.

gpt2_124m_compat_openai.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee616ca7ab582f584a74dbbed83803d87508f15fa2e88406a9dc75b28087cecb
+size 702539117

gpt2_124m_converted.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f88ad435e5d6c2b1750c96c9999d1e220e3df6d22a6a288d7b6256b5ded125e5
+size 497811599

gpt2_124m_openai_checkpoint.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86a0214a6b2badaab581b375589790b1eaf4dc2844f69f6cf622c29d5f6f5406
+size 497812255

gpt2_124m_reduced_ctx__checkpoint.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b2d5485523c262903be8dfb30eca8d86455fe37041b7a5f9e81d49fdb4810e5
+size 1952401499

gpt2_355m_compat_openai.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91fe1e26031f70a5e7513d65336d4a6fb6039436a91a62490bb8fd87ff68d45d
+size 1725924021

gpt2_355m_it_custom.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6b241be4ac14c160782e243c29e6178acba7e4e546577191f80c24df0e94422
+size 1725927081

gpt2_355m_openai_checkpoint.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3e042dc1011e5433b6ae98ea5bb0855038a84d43e5a32ae42914868c37f6b84
+size 1419396991

gpt2_spam_classif_weights.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9838aa44ee0879e206fcc6f11d1087fb6749d97020ef60ee04ee4019742f67bc
+size 548158667