| === SYSTEM === |
| 2026-07-06T00:33:20+00:00 |
| NVIDIA RTX A4000, 16376 MiB, 550.144.03 |
| Build cuda_12.4.r12.4/compiler.34097967_0 |
| === CHAPTER CHECKS === |
| --- code/ch02 --- |
| nvcc -O3 -arch=native -lm -o hello hello.cu |
| nvcc -O3 -arch=native -lm -o vector_add vector_add.cu |
| code/ch02/hello |
| Hello from block 0, thread 0 |
| Hello from block 0, thread 1 |
| Hello from block 0, thread 2 |
| Hello from block 0, thread 3 |
| Hello from block 1, thread 0 |
| Hello from block 1, thread 1 |
| Hello from block 1, thread 2 |
| Hello from block 1, thread 3 |
| code/ch02/vector_add |
| OK vector_add |
| --- code/ch03 --- |
| nvcc -O3 -arch=native -lm -o indices indices.cu |
| nvcc -O3 -arch=native -lm -o matrix_add matrix_add.cu |
| nvcc -O3 -arch=native -lm -o grid_stride grid_stride.cu |
| code/ch03/grid_stride |
| OK grid_stride |
| code/ch03/indices |
| block (0,1) thread (0,0) -> row 2 col 0 idx 16 |
| block (0,1) thread (1,0) -> row 2 col 1 idx 17 |
| block (0,1) thread (2,0) -> row 2 col 2 idx 18 |
| block (0,1) thread (3,0) -> row 2 col 3 idx 19 |
| block (0,1) thread (0,1) -> row 3 col 0 idx 24 |
| block (0,1) thread (1,1) -> row 3 col 1 idx 25 |
| block (0,1) thread (2,1) -> row 3 col 2 idx 26 |
| block (0,1) thread (3,1) -> row 3 col 3 idx 27 |
| block (1,1) thread (0,0) -> row 2 col 4 idx 20 |
| block (1,1) thread (1,0) -> row 2 col 5 idx 21 |
| block (1,1) thread (2,0) -> row 2 col 6 idx 22 |
| block (1,1) thread (3,0) -> row 2 col 7 idx 23 |
| block (1,1) thread (0,1) -> row 3 col 4 idx 28 |
| block (1,1) thread (1,1) -> row 3 col 5 idx 29 |
| block (1,1) thread (2,1) -> row 3 col 6 idx 30 |
| block (1,1) thread (3,1) -> row 3 col 7 idx 31 |
| block (1,0) thread (0,0) -> row 0 col 4 idx 4 |
| block (1,0) thread (1,0) -> row 0 col 5 idx 5 |
| block (1,0) thread (2,0) -> row 0 col 6 idx 6 |
| block (1,0) thread (3,0) -> row 0 col 7 idx 7 |
| block (1,0) thread (0,1) -> row 1 col 4 idx 12 |
| block (1,0) thread (1,1) -> row 1 col 5 idx 13 |
| block (1,0) thread (2,1) -> row 1 col 6 idx 14 |
| block (1,0) thread (3,1) -> row 1 col 7 idx 15 |
| block (0,0) thread (0,0) -> row 0 col 0 idx 0 |
| block (0,0) thread (1,0) -> row 0 col 1 idx 1 |
| block (0,0) thread (2,0) -> row 0 col 2 idx 2 |
| block (0,0) thread (3,0) -> row 0 col 3 idx 3 |
| block (0,0) thread (0,1) -> row 1 col 0 idx 8 |
| block (0,0) thread (1,1) -> row 1 col 1 idx 9 |
| block (0,0) thread (2,1) -> row 1 col 2 idx 10 |
| block (0,0) thread (3,1) -> row 1 col 3 idx 11 |
| code/ch03/matrix_add |
| OK matrix_add |
| --- code/ch04 --- |
| nvcc -O3 -arch=native -lm -o bandwidth bandwidth.cu |
| nvcc -O3 -arch=native -lm -o coalesce coalesce.cu |
| code/ch04/bandwidth |
| buffer size: 134217728 floats (537 MB each) |
| traffic per launch (read + write): 1074 MB |
| average time over 10 launches: 2.725 ms |
| achieved bandwidth: 394.0 GB/s |
| OK bandwidth |
| code/ch04/coalesce |
| matrix: 8192 x 8192 floats (268 MB read + 268 MB written per pass) |
| row-order (coalesced): 1.366 ms (392.9 GB/s) |
| column-order (strided) : 5.375 ms (99.9 GB/s) |
| strided / coalesced slowdown: 3.9x |
| OK coalesce |
| --- code/ch05 --- |
| nvcc -O3 -arch=native -lm -o transpose transpose.cu |
| nvcc -O3 -arch=native -lm -o race race.cu |
| code/ch05/race |
| RACE (may vary): reverse_broken: 2046720 of 4194304 elements wrong |
| a data race is undefined behavior: the count above can change |
| from run to run, and 0 wrong is luck, not correctness |
| OK race_fixed |
| code/ch05/transpose |
| transpose: 4096 x 2048 floats (67.1 MB moved per run) |
| OK transpose_naive |
| OK transpose_tiled |
| OK transpose_padded |
|
|
| kernel ms GB/s speedup |
| transpose_naive 0.533 125.9 1.00x |
| transpose_tiled 0.277 242.1 1.92x |
| transpose_padded 0.194 346.3 2.75x |
| --- code/ch06 --- |
| nvcc -O3 -arch=native -lm histogram.cu -o histogram |
| nvcc -O3 -arch=native -lm reduce.cu -o reduce |
| code/ch06/histogram |
| histogram (random bytes): 3.298 ms |
| OK histogram |
| histogram (one hot bin): 6.743 ms (2.0x slower) |
| OK histogram_worst_case |
| code/ch06/reduce |
| CPU (double) sum: -151.5765 |
| reduce_atomic: sum -151.6804 32.582 ms |
| reduce_tree: sum -151.5632 0.350 ms |
| reduce_shuffle: sum -151.5666 0.218 ms |
| OK reduce_atomic |
| OK reduce_tree |
| OK reduce_shuffle |
| --- code/ch07 --- |
| nvcc -O3 -arch=native -lm silent_fail.cu -o silent_fail |
| nvcc -O3 -arch=native -lm timing.cu -o timing |
| code/ch07/silent_fail |
| --- part 1: illegal launch, no error checking --- |
| kernel "ran" in 0.1864 ms (says the host clock) |
| out[0] = 0.000000, expected 1.251290 <- the kernel never ran |
| --- part 2: same launch, but now we ask --- |
| cudaGetLastError says: invalid configuration argument |
| --- part 3: legal launch, checked, verified --- |
| OK silent_fail_fixed |
| code/ch07/timing |
| WRONG (host clock, no sync): 0.0034 ms |
| RIGHT (events, warmup, avg of 10): 0.3461 ms = 387.8 GB/s |
| OK timing |
| --- code/ch08 --- |
| nvcc -O3 -arch=native -lm -o matmul_naive matmul_naive.cu |
| code/ch08/matmul_naive |
| matmul: C(1024,1024) = A(1024,1024) @ B(1024,1024), 2.15 GFLOP per run |
| computing CPU reference (single core; this takes a few seconds)... |
| OK matmul_naive |
|
|
| CPU: 4628.6 ms 0.46 GFLOP/s |
| GPU: 1.851 ms 1160.4 GFLOP/s |
| speedup: 2501.2x |
| --- code/ch09 --- |
| nvcc -O3 -arch=native -lm -o matmul_ladder matmul_ladder.cu |
| nvcc -O3 -arch=native -lm -o matmul_forward matmul_forward.cu |
| code/ch09/matmul_forward |
| computing CPU references (a few seconds)... |
| matmul_forward: N=8192 C=384 OC=1152 bias=yes |
| OK matmul_forward |
| matmul_forward: 4.879 ms, 1485.4 GFLOP/s |
| matmul_forward_ragged: N=100 C=65 OC=37 bias=NULL |
| OK matmul_forward_ragged |
| code/ch09/matmul_ladder |
| matmul ladder: C(1024,1024) = A(1024,1024) @ B(1024,1024), 2.15 GFLOP per run |
| computing CPU reference (a few seconds)... |
| OK matmul_naive |
| OK matmul_tiled |
| OK matmul_regtile |
|
|
| kernel ms GFLOP/s speedup |
| matmul_naive 1.955 1098.6 1.00x |
| matmul_tiled 1.577 1361.7 1.24x |
| matmul_regtile 0.370 5807.6 5.29x |
| --- code/ch10 --- |
| nvcc -O3 -arch=native -lm softmax.cu -o softmax |
| nvcc -O3 -arch=native -lm reduce_row.cu -o reduce_row |
| code/ch10/reduce_row |
| OK rowmax |
| OK rowsum |
| code/ch10/softmax |
| one row of large scores through both kernels (a demonstration): |
| logits: 101.3 99.1 102.7 12.0 100.0 103.9 97.2 100.8 |
| naive: nan nan nan 0.000000 nan nan nan nan |
| safe: 0.051216 0.005675 0.207688 0.000000 0.013958 0.689551 0.000849 0.031064 |
| OK softmax_v65 |
| softmax_v65: N=8192 V=65 0.046 ms ~229 GB/s effective |
| OK softmax_v1000 |
| softmax_v1000: N=8192 V=1000 0.173 ms ~948 GB/s effective |
| OK softmax_rowsum |
| --- code/ch11 --- |
| nvcc -O3 -arch=native -lm layernorm.cu -o layernorm |
| nvcc -O3 -arch=native -lm gelu.cu -o gelu |
| code/ch11/gelu |
| x gelu(x) |
| -2 -0.0454 |
| -1 -0.1588 |
| 0 0.0000 |
| 1 0.8412 |
| 2 1.9546 |
| OK gelu |
| code/ch11/layernorm |
| N=8192 rows, C=384 channels |
| OK layernorm_out |
| OK layernorm_mean |
| OK layernorm_rstd |
| N=1000 rows, C=100 channels |
| OK layernorm_ragged |
| --- code/ch12 --- |
| nvcc -O3 -arch=native -lm encoder.cu -o encoder |
| nvcc -O3 -arch=native -lm crossentropy.cu -o crossentropy |
| code/ch12/crossentropy |
| OK softmax |
| OK crossentropy |
| uniform logits: mean loss 4.174387, expected ln(65) = 4.174387 |
| OK uniform_loss_4.17 |
| code/ch12/encoder |
| OK encoder |
| --- code/ch13 --- |
| nvcc -O3 -arch=native -lm attention.cu -o attention |
| code/ch13/attention |
| OK attention_out |
| OK attention_att |
| OK attention_causal |
| --- code/ch16 --- |
| cc -O2 -lm -o toy_backprop toy_backprop.c |
| code/ch16/toy_backprop |
| forward: z = 7.0, y = 49.0 |
| backward: dx = 42.0, dw = 28.0, db = 14.0 |
| finite difference dx ~= 42.002 |
| OK toy_backprop |
| === GPT BUILD === |
| rm -f prepare_data train_gpt generate_gpt grad_check |
| cc -O2 -lm -o prepare_data prepare_data.c |
| nvcc -O3 -arch=native -lm -o train_gpt train_gpt.cu |
| nvcc -O3 -arch=native -lm -o generate_gpt generate_gpt.cu |
| nvcc -O3 -arch=native -lm -o grad_check grad_check.cu |
| === DATA === |
| curl -L -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt |
| % Total % Received % Xferd Average Speed Time Time Time Current |
| Dload Upload Total Spent Left Speed |
|
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1089k 100 1089k 0 0 4798k 0 --:--:-- --:--:-- --:--:-- 4819k |
| tokens: 1115394 | vocab: 65 | train: 1003854 | val: 111540 |
| === GRAD CHECK === |
| analytic -0.074804 |
| numerical -0.074863 |
| diff 0.000060 |
| OK grad_check_wte0 |
| === TRAIN 5000 === |
| parameters: 10770816 |
| step 1 | loss 4.2383 | lr 0.000010 | 7300 tok/s |
| step 100 | loss 2.4751 | lr 0.001000 | 7039 tok/s |
| step 200 | loss 2.3741 | lr 0.000999 | 6967 tok/s |
| val 250 | loss 2.2469 |
| step 300 | loss 2.1620 | lr 0.000996 | 7033 tok/s |
| step 400 | loss 1.8983 | lr 0.000992 | 7017 tok/s |
| step 500 | loss 1.7598 | lr 0.000985 | 7035 tok/s |
| val 500 | loss 1.9397 |
| step 600 | loss 1.7097 | lr 0.000977 | 7024 tok/s |
| step 700 | loss 1.7504 | lr 0.000967 | 7036 tok/s |
| val 750 | loss 1.8425 |
| step 800 | loss 1.6235 | lr 0.000955 | 7022 tok/s |
| step 900 | loss 1.5730 | lr 0.000942 | 7052 tok/s |
| step 1000 | loss 1.6028 | lr 0.000927 | 7029 tok/s |
| val 1000 | loss 1.7558 |
| saved model.bin at step 1000 |
| step 1100 | loss 1.6541 | lr 0.000911 | 7029 tok/s |
| step 1200 | loss 1.6146 | lr 0.000893 | 7051 tok/s |
| val 1250 | loss 1.6679 |
| step 1300 | loss 1.4642 | lr 0.000873 | 7035 tok/s |
| step 1400 | loss 1.3288 | lr 0.000853 | 7034 tok/s |
| step 1500 | loss 1.3054 | lr 0.000831 | 7043 tok/s |
| val 1500 | loss 1.6317 |
| step 1600 | loss 1.3982 | lr 0.000807 | 7063 tok/s |
| step 1700 | loss 1.3385 | lr 0.000783 | 7052 tok/s |
| val 1750 | loss 1.6163 |
| step 1800 | loss 1.2542 | lr 0.000758 | 7047 tok/s |
| step 1900 | loss 1.3436 | lr 0.000732 | 7050 tok/s |
| step 2000 | loss 1.2770 | lr 0.000705 | 7062 tok/s |
| val 2000 | loss 1.5911 |
| saved model.bin at step 2000 |
| step 2100 | loss 1.2688 | lr 0.000678 | 7052 tok/s |
| step 2200 | loss 1.1882 | lr 0.000650 | 7042 tok/s |
| val 2250 | loss 1.6044 |
| step 2300 | loss 1.3317 | lr 0.000622 | 7046 tok/s |
| step 2400 | loss 1.1211 | lr 0.000593 | 7036 tok/s |
| step 2500 | loss 1.2006 | lr 0.000564 | 7059 tok/s |
| val 2500 | loss 1.6112 |
| step 2600 | loss 1.1568 | lr 0.000536 | 7065 tok/s |
| step 2700 | loss 1.2186 | lr 0.000507 | 7067 tok/s |
| val 2750 | loss 1.6321 |
| step 2800 | loss 1.1401 | lr 0.000478 | 7069 tok/s |
| step 2900 | loss 1.0421 | lr 0.000450 | 7064 tok/s |
| step 3000 | loss 1.1062 | lr 0.000422 | 7057 tok/s |
| val 3000 | loss 1.6381 |
| saved model.bin at step 3000 |
| step 3100 | loss 0.9624 | lr 0.000395 | 7069 tok/s |
| step 3200 | loss 1.0200 | lr 0.000368 | 7073 tok/s |
| val 3250 | loss 1.6753 |
| step 3300 | loss 1.0242 | lr 0.000342 | 7089 tok/s |
| step 3400 | loss 1.0527 | lr 0.000317 | 7055 tok/s |
| step 3500 | loss 0.9729 | lr 0.000293 | 7063 tok/s |
| val 3500 | loss 1.7316 |
| step 3600 | loss 0.9506 | lr 0.000269 | 7077 tok/s |
| step 3700 | loss 0.9488 | lr 0.000247 | 7067 tok/s |
| val 3750 | loss 1.7931 |
| step 3800 | loss 0.9607 | lr 0.000227 | 7055 tok/s |
| step 3900 | loss 0.9824 | lr 0.000207 | 7056 tok/s |
| step 4000 | loss 0.7658 | lr 0.000189 | 7031 tok/s |
| val 4000 | loss 1.8825 |
| saved model.bin at step 4000 |
| step 4100 | loss 0.8902 | lr 0.000173 | 7044 tok/s |
| step 4200 | loss 0.7832 | lr 0.000158 | 7047 tok/s |
| val 4250 | loss 1.9381 |
| step 4300 | loss 0.8469 | lr 0.000145 | 7036 tok/s |
| step 4400 | loss 0.8653 | lr 0.000133 | 7053 tok/s |
| step 4500 | loss 0.8695 | lr 0.000123 | 7035 tok/s |
| val 4500 | loss 1.9576 |
| step 4600 | loss 0.7353 | lr 0.000115 | 7048 tok/s |
| step 4700 | loss 0.7704 | lr 0.000108 | 7046 tok/s |
| val 4750 | loss 1.9924 |
| step 4800 | loss 0.7446 | lr 0.000104 | 7032 tok/s |
| step 4900 | loss 0.8131 | lr 0.000101 | 7042 tok/s |
| step 5000 | loss 0.7931 | lr 0.000100 | 7054 tok/s |
| val 5000 | loss 2.0483 |
| saved model.bin at step 5000 |
| === GENERATE ROMEO === |
| ROMEO: |
| Ay not sustom's master and by earor-woo's love; |
| She damned life, lest have had some removed, |
| Lest in neither slander man, and might have |
| Because the moon his commission. |
| |
| MARCIUS: |
| No, I'll consent the choler is prince: |
| I am gone to great the last of my master; |
| That's smules not put to my counsel. |
| |
| LARTIUS: |
| He'll down away; alas, my liege, may stand |
| And in the truth of one sons with that for night |
| But upon'sy. Let thou go; |
| How sometimes thee on the ground the vault? |
| Though thy wife will then to thee to thine, |
| And doth me, thy force should thee from thy brother |
| As thou didst uncontent? I think thee, that would ens |
| Of gracious that here were most under his pleasing |
| pay a wordfather, a mother; a boote ardone |
| And this one in so to be a child, and so meek |
| To lie in; and so, as the which I do ha |
| === GENERATE JULIET === |
| JULIET: |
| And fearn gather nothing by thy lance. |
| Is my daughter's most reason can strengthen's stand |
| My separance up and hereport of their children, |
| My mildness and constant seize in purge |
| With informal enrage, that the habit ours. |
| Say thou never have me once what I have: |
| Not hear then beat his; 'twere thy toother such |
| And see the gates that which he can say. |
| |
| KING EDWARD IV: |
| The gods be thereof there! O thee, climbs: |
| This is there: 'for thou re's more there's service: |
| 'Tis thou there? |
| Why, villain, if thou know'st this noble armed, |
| It must are the alive. But what is this? |
| |
| GLOUCESTER: |
| It should be, there's no consort or charge. |
| |
| YORK: |
| At what command? |
| |
| CLARENCE: |
| It is. For that docts but loss this thought of thee, |
| As doubt the truth of the whole head. |
| |
| GLOUCESTER: |
| Your brother is and Polixenes, |
| Ho |
| === ARTIFACTS === |
| -rw-r--r-- 1 root root 124M Jul 6 02:12 model.bin |
| -rw-r--r-- 1 root root 3.9M Jul 6 00:34 train.bin |
| -rw-r--r-- 1 root root 436K Jul 6 00:34 val.bin |
| -rw-r--r-- 1 root root 73 Jul 6 00:34 vocab.bin |
| 75eb8311e2bdec22b0c0e40084fb39fb1ed190662d3cd51f26e9f4b37ee06691 model.bin |
| 464eb51d66b5b643889064b82c75607e2d756df11ec960e5e6aaaf03c71734be vocab.bin |
| a3f591a5c0dfa7f54a2ec0ab39c4561d6475f1af28264f38076932ca425d5037 train.bin |
| 836220b36a012c19cd285816001700117edef4f8e1098130b234d4b8fa766d86 val.bin |
| === DONE === |
| 2026-07-06T02:12:17+00:00 |
| |
