Hugging Face's logo

Spaces:
duoan
/
TorchCode
Running

App Files Community
github-actions[bot] commited on
Commit
1ccc68c
Β·
0 Parent(s):

Deploy prebuilt image 8a49ba150f76c92fc6e2fd30631e07b88c803988

Browse files
Files changed (2) hide show
  1. Dockerfile +1 -0
  2. README.md +370 -0
Dockerfile ADDED
@@ -0,0 +1 @@
 
 
1
+ FROM ghcr.io/duoan/torchcode:latest
README.md ADDED
@@ -0,0 +1,370 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: TorchCode
3
+ emoji: πŸ”₯
4
+ colorFrom: red
5
+ colorTo: yellow
6
+ sdk: docker
7
+ app_port: 7860
8
+ pinned: false
9
+ ---
10
+
11
+ <div align="center">
12
+
13
+ # πŸ”₯ TorchCode
14
+
15
+ **Crack the PyTorch interview.**
16
+
17
+ Practice implementing operators and architectures from scratch β€” the exact skills top ML teams test for.
18
+
19
+ *Like LeetCode, but for tensors. Self-hosted. Jupyter-based. Instant feedback.*
20
+
21
+ [![PyTorch](https://img.shields.io/badge/PyTorch-ee4c2c?style=for-the-badge&logo=pytorch&logoColor=white)](https://pytorch.org)
22
+ [![Jupyter](https://img.shields.io/badge/Jupyter-F37626?style=for-the-badge&logo=jupyter&logoColor=white)](https://jupyter.org)
23
+ [![Docker](https://img.shields.io/badge/Docker-2496ED?style=for-the-badge&logo=docker&logoColor=white)](https://www.docker.com)
24
+ [![Python](https://img.shields.io/badge/Python_3.11-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://python.org)
25
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](LICENSE)
26
+
27
+ [![GitHub stars](https://img.shields.io/github/stars/duoan/TorchCode?style=social)](https://github.com/duoan/TorchCode)
28
+ [![GitHub Container Registry](https://img.shields.io/badge/ghcr.io-TorchCode-blue?style=flat-square&logo=github)](https://ghcr.io/duoan/torchcode)
29
+ [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Spaces-TorchCode-blue?style=flat-square)](https://huggingface.co/spaces/duoan/TorchCode)
30
+ ![Problems](https://img.shields.io/badge/problems-40-orange?style=flat-square)
31
+ ![GPU](https://img.shields.io/badge/GPU-not%20required-brightgreen?style=flat-square)
32
+
33
+ [![Star History Chart](https://api.star-history.com/svg?repos=duoan/TorchCode&type=Date)](https://star-history.com/#duoan/TorchCode&Date)
34
+
35
+ </div>
36
+
37
+ ---
38
+
39
+ ## 🎯 Why TorchCode?
40
+
41
+ Top companies (Meta, Google DeepMind, OpenAI, etc.) expect ML engineers to implement core operations **from memory on a whiteboard**. Reading papers isn't enough β€” you need to write `softmax`, `LayerNorm`, `MultiHeadAttention`, and full Transformer blocks code.
42
+
43
+ TorchCode gives you a **structured practice environment** with:
44
+
45
+ | | Feature | |
46
+ |---|---|---|
47
+ | 🧩 | **40 curated problems** | The most frequently asked PyTorch interview topics |
48
+ | βš–οΈ | **Automated judge** | Correctness checks, gradient verification, and timing |
49
+ | 🎨 | **Instant feedback** | Colored pass/fail per test case, just like competitive programming |
50
+ | πŸ’‘ | **Hints when stuck** | Nudges without full spoilers |
51
+ | πŸ“– | **Reference solutions** | Study optimal implementations after your attempt |
52
+ | πŸ“Š | **Progress tracking** | What you've solved, best times, and attempt counts |
53
+ | πŸ”„ | **One-click reset** | Toolbar button to reset any notebook back to its blank template β€” practice the same problem as many times as you want |
54
+ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](#) | **Open in Colab** | Every notebook has an "Open in Colab" badge + toolbar button β€” run problems in Google Colab with zero setup |
55
+
56
+ No cloud. No signup. No GPU needed. Just `make run` β€” or try it instantly on Hugging Face.
57
+
58
+ ---
59
+
60
+ ## πŸš€ Quick Start
61
+
62
+ ### Option 0 β€” Try it online (zero install)
63
+
64
+ **[Launch on Hugging Face Spaces](https://huggingface.co/spaces/duoan/TorchCode)** β€” opens a full JupyterLab environment in your browser. Nothing to install.
65
+
66
+ Or open any problem directly in Google Colab β€” every notebook has an [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/01_relu.ipynb) badge.
67
+
68
+ ### Option 0b β€” Use the judge in Colab (pip)
69
+
70
+ In Google Colab, install the judge from PyPI so you can run `check(...)` without cloning the repo:
71
+
72
+ ```bash
73
+ !pip install torch-judge
74
+ ```
75
+
76
+ Then in a notebook cell:
77
+
78
+ ```python
79
+ from torch_judge import check, status, hint, reset_progress
80
+ status() # list all problems and your progress
81
+ check("relu") # run tests for the "relu" task
82
+ hint("relu") # show a hint
83
+ ```
84
+
85
+ ### Option 1 β€” Pull the pre-built image (fastest)
86
+
87
+ ```bash
88
+ docker run -p 8888:8888 -e PORT=8888 ghcr.io/duoan/torchcode:latest
89
+ ```
90
+
91
+ ### Option 2 β€” Build locally
92
+
93
+ ```bash
94
+ make run
95
+ ```
96
+
97
+ Open **<http://localhost:8888>** β€” that's it. Works with both Docker and Podman (auto-detected).
98
+
99
+ ---
100
+
101
+ ## πŸ“‹ Problem Set
102
+
103
+ > **Frequency**: πŸ”₯ = very likely in interviews, ⭐ = commonly asked, πŸ’‘ = emerging / differentiator
104
+
105
+ ### 🧱 Fundamentals β€” "Implement X from scratch"
106
+
107
+ The bread and butter of ML coding interviews. You'll be asked to write these without `torch.nn`.
108
+
109
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
110
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
111
+ | 1 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/01_relu.ipynb" target="_blank">ReLU</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/01_relu.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `relu(x)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Activation functions, element-wise ops |
112
+ | 2 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/02_softmax.ipynb" target="_blank">Softmax</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/02_softmax.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_softmax(x, dim)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Numerical stability, exp/log tricks |
113
+ | 16 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/16_cross_entropy.ipynb" target="_blank">Cross-Entropy Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/16_cross_entropy.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `cross_entropy_loss(logits, targets)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Log-softmax, logsumexp trick |
114
+ | 17 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/17_dropout.ipynb" target="_blank">Dropout</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/17_dropout.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MyDropout` (nn.Module) | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Train/eval mode, inverted scaling |
115
+ | 18 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/18_embedding.ipynb" target="_blank">Embedding</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/18_embedding.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MyEmbedding` (nn.Module) | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Lookup table, `weight[indices]` |
116
+ | 19 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/19_gelu.ipynb" target="_blank">GELU</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/19_gelu.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_gelu(x)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | ⭐ | Gaussian error linear unit, `torch.erf` |
117
+ | 20 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/20_weight_init.ipynb" target="_blank">Kaiming Init</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/20_weight_init.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `kaiming_init(weight)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | ⭐ | `std = sqrt(2/fan_in)`, variance scaling |
118
+ | 21 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/21_gradient_clipping.ipynb" target="_blank">Gradient Clipping</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/21_gradient_clipping.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `clip_grad_norm(params, max_norm)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | ⭐ | Norm-based clipping, direction preservation |
119
+ | 31 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/31_gradient_accumulation.ipynb" target="_blank">Gradient Accumulation</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/31_gradient_accumulation.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `accumulated_step(model, opt, ...)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ’‘ | Micro-batching, loss scaling |
120
+ | 40 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/40_linear_regression.ipynb" target="_blank">Linear Regression</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/40_linear_regression.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `LinearRegression` (3 methods) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Normal equation, GD from scratch, nn.Linear |
121
+ | 3 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/03_linear.ipynb" target="_blank">Linear Layer</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/03_linear.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `SimpleLinear` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | `y = xW^T + b`, Kaiming init, `nn.Parameter` |
122
+ | 4 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/04_layernorm.ipynb" target="_blank">LayerNorm</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/04_layernorm.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_layer_norm(x, Ξ³, Ξ²)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Normalization, running stats, affine transform |
123
+ | 7 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/07_batchnorm.ipynb" target="_blank">BatchNorm</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/07_batchnorm.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_batch_norm(x, γ, β)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Batch vs layer statistics, train/eval behavior |
124
+ | 8 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/08_rmsnorm.ipynb" target="_blank">RMSNorm</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/08_rmsnorm.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `rms_norm(x, weight)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | LLaMA-style norm, simpler than LayerNorm |
125
+ | 15 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/15_mlp.ipynb" target="_blank">SwiGLU MLP</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/15_mlp.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `SwiGLUMLP` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Gated FFN, `SiLU(gate) * up`, LLaMA/Mistral-style |
126
+ | 22 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/22_conv2d.ipynb" target="_blank">Conv2d</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/22_conv2d.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_conv2d(x, weight, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Convolution, unfold, stride/padding |
127
+
128
+ ### 🧠 Attention Mechanisms β€” The heart of modern ML interviews
129
+
130
+ If you're interviewing for any role touching LLMs or Transformers, expect at least one of these.
131
+
132
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
133
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
134
+ | 23 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/23_cross_attention.ipynb" target="_blank">Cross-Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/23_cross_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MultiHeadCrossAttention` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Encoder-decoder, Q from decoder, K/V from encoder |
135
+ | 5 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/05_attention.ipynb" target="_blank">Scaled Dot-Product Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/05_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `scaled_dot_product_attention(Q, K, V)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | `softmax(QK^T/√d_k)V`, the foundation of everything |
136
+ | 6 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/06_multihead_attention.ipynb" target="_blank">Multi-Head Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/06_multihead_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MultiHeadAttention` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Parallel heads, split/concat, projection matrices |
137
+ | 9 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/09_causal_attention.ipynb" target="_blank">Causal Self-Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/09_causal_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `causal_attention(Q, K, V)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Autoregressive masking with `-inf`, GPT-style |
138
+ | 10 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/10_gqa.ipynb" target="_blank">Grouped Query Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/10_gqa.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `GroupQueryAttention` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | GQA (LLaMA 2), KV sharing across heads |
139
+ | 11 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/11_sliding_window.ipynb" target="_blank">Sliding Window Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/11_sliding_window.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `sliding_window_attention(Q, K, V, w)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | Mistral-style local attention, O(n·w) complexity |
140
+ | 12 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/12_linear_attention.ipynb" target="_blank">Linear Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/12_linear_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `linear_attention(Q, K, V)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Kernel trick, `Ο†(Q)(Ο†(K)^TV)`, O(nΒ·dΒ²) |
141
+ | 14 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/14_kv_cache.ipynb" target="_blank">KV Cache Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/14_kv_cache.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `KVCacheAttention` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Incremental decoding, cache K/V, prefill vs decode |
142
+ | 24 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/24_rope.ipynb" target="_blank">RoPE</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/24_rope.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `apply_rope(q, k)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Rotary position embedding, relative position via rotation |
143
+ | 25 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/25_flash_attention.ipynb" target="_blank">Flash Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/25_flash_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `flash_attention(Q, K, V, block_size)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Tiled attention, online softmax, memory-efficient |
144
+
145
+ ### πŸ—οΈ Architecture & Adaptation β€” Put it all together
146
+
147
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
148
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
149
+ | 26 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/26_lora.ipynb" target="_blank">LoRA</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/26_lora.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `LoRALinear` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Low-rank adaptation, frozen base + `BA` update |
150
+ | 27 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/27_vit_patch.ipynb" target="_blank">ViT Patch Embedding</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/27_vit_patch.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `PatchEmbedding` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ’‘ | Image β†’ patches β†’ linear projection |
151
+ | 13 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/13_gpt2_block.ipynb" target="_blank">GPT-2 Block</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/13_gpt2_block.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `GPT2Block` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | Pre-norm, causal MHA + MLP (4x, GELU), residual connections |
152
+ | 28 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/28_moe.ipynb" target="_blank">Mixture of Experts</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/28_moe.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MixtureOfExperts` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | Mixtral-style, top-k routing, expert MLPs |
153
+
154
+ ### βš™οΈ Training & Optimization
155
+
156
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
157
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
158
+ | 29 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/29_adam.ipynb" target="_blank">Adam Optimizer</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/29_adam.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MyAdam` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Momentum + RMSProp, bias correction |
159
+ | 30 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/30_cosine_lr.ipynb" target="_blank">Cosine LR Scheduler</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/30_cosine_lr.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `cosine_lr_schedule(step, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Linear warmup + cosine annealing |
160
+
161
+ ### 🎯 Inference & Decoding
162
+
163
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
164
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
165
+ | 32 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/32_topk_sampling.ipynb" target="_blank">Top-k / Top-p Sampling</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/32_topk_sampling.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `sample_top_k_top_p(logits, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Nucleus sampling, temperature scaling |
166
+ | 33 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/33_beam_search.ipynb" target="_blank">Beam Search</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/33_beam_search.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `beam_search(log_prob_fn, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Hypothesis expansion, pruning, eos handling |
167
+ | 34 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/34_speculative_decoding.ipynb" target="_blank">Speculative Decoding</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/34_speculative_decoding.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `speculative_decode(target, draft, ...)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Accept/reject, draft model acceleration |
168
+
169
+ ### πŸ”¬ Advanced β€” Differentiators
170
+
171
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
172
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
173
+ | 35 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/35_bpe.ipynb" target="_blank">BPE Tokenizer</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/35_bpe.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `SimpleBPE` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Byte-pair encoding, merge rules, subword splits |
174
+ | 36 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/36_int8_quantization.ipynb" target="_blank">INT8 Quantization</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/36_int8_quantization.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `Int8Linear` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Per-channel quantize, scale/zero-point, buffer vs param |
175
+ | 37 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/37_dpo_loss.ipynb" target="_blank">DPO Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/37_dpo_loss.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `dpo_loss(chosen, rejected, ...)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Direct preference optimization, alignment training |
176
+ | 38 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/38_grpo_loss.ipynb" target="_blank">GRPO Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/38_grpo_loss.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `grpo_loss(logps, rewards, group_ids, eps)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Group relative policy optimization, RLAIF, within-group normalized advantages |
177
+ | 39 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/39_ppo_loss.ipynb" target="_blank">PPO Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/39_ppo_loss.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `ppo_loss(new_logps, old_logps, advantages, clip_ratio)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | PPO clipped surrogate loss, policy gradient, trust region |
178
+
179
+ ---
180
+
181
+ ## βš™οΈ How It Works
182
+
183
+ Each problem has **two** notebooks:
184
+
185
+ | File | Purpose |
186
+ |------|---------|
187
+ | `01_relu.ipynb` | ✏️ Blank template β€” write your code here |
188
+ | `01_relu_solution.ipynb` | πŸ“– Reference solution β€” check when stuck |
189
+
190
+ ### Workflow
191
+
192
+ ```text
193
+ 1. Open a blank notebook β†’ Read the problem description
194
+ 2. Implement your solution β†’ Use only basic PyTorch ops
195
+ 3. Debug freely β†’ print(x.shape), check gradients, etc.
196
+ 4. Run the judge cell β†’ check("relu")
197
+ 5. See instant colored feedback β†’ βœ… pass / ❌ fail per test case
198
+ 6. Stuck? Get a nudge β†’ hint("relu")
199
+ 7. Review the reference solution β†’ 01_relu_solution.ipynb
200
+ 8. Click πŸ”„ Reset in the toolbar β†’ Blank slate β€” practice again!
201
+ ```
202
+
203
+ ### In-Notebook API
204
+
205
+ ```python
206
+ from torch_judge import check, hint, status
207
+
208
+ check("relu") # Judge your implementation
209
+ hint("causal_attention") # Get a hint without full spoiler
210
+ status() # Progress dashboard β€” solved / attempted / todo
211
+ ```
212
+
213
+ ---
214
+
215
+ ## πŸ“… Suggested Study Plan
216
+
217
+ > **Total: ~12–16 hours spread across 3–4 weeks. Perfect for interview prep on a deadline.**
218
+
219
+ | Week | Focus | Problems | Time |
220
+ |:----:|-------|----------|:----:|
221
+ | **1** | 🧱 Foundations | ReLU β†’ Softmax β†’ CE Loss β†’ Dropout β†’ Embedding β†’ GELU β†’ Linear β†’ LayerNorm β†’ BatchNorm β†’ RMSNorm β†’ SwiGLU MLP β†’ Conv2d | 2–3 hrs |
222
+ | **2** | 🧠 Attention Deep Dive | SDPA β†’ MHA β†’ Cross-Attn β†’ Causal β†’ GQA β†’ KV Cache β†’ Sliding Window β†’ RoPE β†’ Linear Attn β†’ Flash Attn | 3–4 hrs |
223
+ | **3** | πŸ—οΈ Architecture + Training | GPT-2 Block β†’ LoRA β†’ MoE β†’ ViT Patch β†’ Adam β†’ Cosine LR β†’ Grad Clip β†’ Grad Accumulation β†’ Kaiming Init | 3–4 hrs |
224
+ | **4** | 🎯 Inference + Advanced | Top-k/p Sampling β†’ Beam Search β†’ Speculative Decoding β†’ BPE β†’ INT8 Quant β†’ DPO Loss β†’ GRPO Loss β†’ PPO Loss + speed run | 3–4 hrs |
225
+
226
+ ---
227
+
228
+ ## πŸ›οΈ Architecture
229
+
230
+ ```text
231
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
232
+ β”‚ Docker / Podman Container β”‚
233
+ β”‚ β”‚
234
+ β”‚ JupyterLab (:8888) β”‚
235
+ β”‚ β”œβ”€β”€ templates/ (reset on each run) β”‚
236
+ β”‚ β”œβ”€β”€ solutions/ (reference impl) β”‚
237
+ β”‚ β”œβ”€β”€ torch_judge/ (auto-grading) β”‚
238
+ β”‚ β”œβ”€β”€ torchcode-labext (JLab plugin) β”‚
239
+ β”‚ β”‚ πŸ”„ Reset β€” restore template β”‚
240
+ β”‚ β”‚ πŸ”— Colab β€” open in Colab β”‚
241
+ β”‚ └── PyTorch (CPU), NumPy β”‚
242
+ β”‚ β”‚
243
+ β”‚ Judge checks: β”‚
244
+ β”‚ βœ“ Output correctness (allclose) β”‚
245
+ β”‚ βœ“ Gradient flow (autograd) β”‚
246
+ β”‚ βœ“ Shape consistency β”‚
247
+ β”‚ βœ“ Edge cases & numerical stability β”‚
248
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
249
+ ```
250
+
251
+ Single container. Single port. No database. No frontend framework. No GPU.
252
+
253
+ ## πŸ› οΈ Commands
254
+
255
+ ```bash
256
+ make run # Build & start (http://localhost:8888)
257
+ make stop # Stop the container
258
+ make clean # Stop + remove volumes + reset all progress
259
+ ```
260
+
261
+ ## 🧩 Adding Your Own Problems
262
+
263
+ TorchCode uses auto-discovery β€” just drop a new file in `torch_judge/tasks/`:
264
+
265
+ ```python
266
+ TASK = {
267
+ "id": "my_task",
268
+ "title": "My Custom Problem",
269
+ "difficulty": "medium",
270
+ "function_name": "my_function",
271
+ "hint": "Think about broadcasting...",
272
+ "tests": [ ... ],
273
+ }
274
+ ```
275
+
276
+ No registration needed. The judge picks it up automatically.
277
+
278
+ ---
279
+
280
+ ## πŸ“¦ Publishing `torch-judge` to PyPI (maintainers)
281
+
282
+ The judge is published as a separate package so Colab/users can `pip install torch-judge` without cloning the repo.
283
+
284
+ ### Automatic (GitHub Action)
285
+
286
+ Pushing to `master` after changing the package version triggers [`.github/workflows/pypi-publish.yml`](.github/workflows/pypi-publish.yml), which builds and uploads to PyPI. No git tag is required.
287
+
288
+ 1. **Bump version** in `torch_judge/_version.py` (e.g. `__version__ = "0.1.1"`).
289
+ 2. **Configure PyPI Trusted Publisher** (one-time):
290
+ - PyPI β†’ Your project **torch-judge** β†’ **Publishing** β†’ **Add a new pending publisher**
291
+ - Owner: `duoan`, Repository: `TorchCode`, Workflow: `pypi-publish.yml`, Environment: (leave empty)
292
+ - Run the workflow once (push a version bump to `master` or **Actions β†’ Publish torch-judge to PyPI β†’ Run workflow**); PyPI will then link the publisher.
293
+ 3. **Release**: commit the version bump and `git push origin master`.
294
+
295
+ Alternatively, use an API token: add repository secret `PYPI_API_TOKEN` (value = `pypi-...` from PyPI) and set `TWINE_USERNAME=__token__` and `TWINE_PASSWORD` from that secret in the workflow if you prefer not to use Trusted Publishing.
296
+
297
+ ### Manual
298
+
299
+ ```bash
300
+ pip install build twine
301
+ python -m build
302
+ twine upload dist/*
303
+ ```
304
+
305
+ Version is in `torch_judge/_version.py`; bump it before each release.
306
+
307
+ ---
308
+
309
+ ## ❓ FAQ
310
+
311
+ <details>
312
+ <summary><b>Do I need a GPU?</b></summary>
313
+ <br>
314
+ No. Everything runs on CPU. The problems test correctness and understanding, not throughput.
315
+ </details>
316
+
317
+ <details>
318
+ <summary><b>Can I keep my solutions between runs?</b></summary>
319
+ <br>
320
+ Blank templates reset on every <code>make run</code> so you practice from scratch. Save your work under a different filename if you want to keep it. You can also click the <b>πŸ”„ Reset</b> button in the notebook toolbar at any time to restore the blank template without restarting.
321
+ </details>
322
+
323
+ <details>
324
+ <summary><b>Can I use Google Colab instead?</b></summary>
325
+ <br>
326
+ Yes! Every notebook has an <b>Open in Colab</b> badge at the top. Click it to open the problem directly in Google Colab β€” no Docker or local setup needed. You can also use the <b>Colab</b> toolbar button inside JupyterLab.
327
+ </details>
328
+
329
+ <details>
330
+ <summary><b>How are solutions graded?</b></summary>
331
+ <br>
332
+ The judge runs your function against multiple test cases using <code>torch.allclose</code> for numerical correctness, verifies gradients flow properly via autograd, and checks edge cases specific to each operation.
333
+ </details>
334
+
335
+ <details>
336
+ <summary><b>Who is this for?</b></summary>
337
+ <br>
338
+ Anyone preparing for ML/AI engineering interviews at top tech companies, or anyone who wants to deeply understand how PyTorch operations work under the hood.
339
+ </details>
340
+
341
+ ---
342
+
343
+ ## 🀝 Contributors
344
+
345
+ Thanks to everyone who has contributed to TorchCode.
346
+
347
+ <!-- readme: contributors -start -->
348
+ <!-- readme: contributors -end -->
349
+
350
+ Auto-generated from the [GitHub contributors graph](https://github.com/duoan/TorchCode/graphs/contributors) with avatars and GitHub usernames.
351
+
352
+ ---
353
+
354
+ <div align="center">
355
+
356
+ **Built for engineers who want to deeply understand what they build.**
357
+
358
+ If this helped your interview prep, consider giving it a ⭐
359
+
360
+ ---
361
+
362
+ ### β˜• Buy Me a Coffee
363
+
364
+ <a href="https://buymeacoffee.com/duoan" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
365
+
366
+ <img src="./bmc_qr.png" alt="BMC QR Code" width="150" height="150">
367
+
368
+ *Scan to support*
369
+
370
+ </div>