Update README.md

README.md

@@ -42,8 +42,6 @@ This model recreates the GPT architecture scaled down just to 10.8M parameters.
 <!-- Provide the basic links for the model. -->
 
 - **Repository:** [(https://github.com/Deepmalya2506/From-Kernel-to-Silicon---Transformers)]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
 
 ## Uses
 
@@ -88,107 +86,64 @@ Use the code below to get started with the model.
 #### Training Hyperparameters
 
 - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- batch_size = 64 # how many independent sequences will we process in parallel?
+- block_size = 256 # what is the maximum context length for predictions?
+- max_iters = 5000
+- eval_interval = 500
+- learning_rate = 3e-4
+- device = 'cuda' if torch.cuda.is_available() else 'cpu' 
+- eval_iters = 200
+- n_embed=384
+- n_head=6    # therefore every head (head_size)= 384//6 = 64 dim
+- dropout=0.2
 
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
 
 ## Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
 
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+* step 0: train loss 4.3585, val loss 4.3557
+* step 500: train loss 2.0170, val loss 2.0917
+* step 1000: train loss 1.6125, val loss 1.7842
+* step 1500: train loss 1.4475, val loss 1.6449
+* step 2000: train loss 1.3575, val loss 1.5784
+* step 2500: train loss 1.2872, val loss 1.5347
+* step 3000: train loss 1.2367, val loss 1.5039
+* step 3500: train loss 1.1909, val loss 1.4929
+* step 4000: train loss 1.1529, val loss 1.4847
+* step 4500: train loss 1.1201, val loss 1.4798
+
+
+#### Overview & Purpose
+To prevent overfitting on the 1.1M character TinyShakespeare dataset, **Dropout (0.2)** is injected into:
+1. Attention softmax weights (dropping attention edges).
+2. Multi-Head projection layer outputs.
+3. FeedForward Network layer outputs.
+
+#### Complete System Architecture Summary
+
+```
+====================================================================================================
+Layer / Submodule               Input Shape             Output Shape            Parameters
+====================================================================================================
+Token Embedding Table           (B, T)                  (B, T, 384)             65 x 384 = 24,960
+Positional Embedding Table      (T,)                    (T, 384)                256 x 384 = 98,304
+----------------------------------------------------------------------------------------------------
+Block 1..6 (x6 Stacked):
+  ├── LayerNorm 1               (B, T, 384)             (B, T, 384)             2 x 384 = 768
+  ├── Multi-Head Attention:
+  │   ├── Key Linear            (B, T, 384)             (B, T, 384)             384 x 384 = 147,456
+  │   ├── Query Linear          (B, T, 384)             (B, T, 384)             384 x 384 = 147,456
+  │   ├── Value Linear          (B, T, 384)             (B, T, 384)             384 x 384 = 147,456
+  │   └── Out Projection        (B, T, 384)             (B, T, 384)             384 x 384 + 384 = 147,840
+  ├── LayerNorm 2               (B, T, 384)             (B, T, 384)             2 x 384 = 768
+  └── FeedForward Net:
+      ├── FC 1 (Expand)         (B, T, 384)             (B, T, 1536)            384 x 1536 + 1536 = 591,360
+      └── FC 2 (Contract)       (B, T, 1536)            (B, T, 384)             1536 x 384 + 384 = 590,208
+----------------------------------------------------------------------------------------------------
+Final LayerNorm                 (B, T, 384)             (B, T, 384)             2 x 384 = 768
+Language Model Head             (B, T, 384)             (B, T, 65)              384 x 65 + 65 = 25,025
+====================================================================================================
+Total Parameters: ~10,788,929 (~10.8 Million)
+====================================================================================================
+```