cnn0 / RESULTS.md
shyo2's picture
Devin
Add results writeup, data provenance, and README
ee3b201
|
Raw History Blame Contribute Delete
3.5 kB

a0 โ€” ์ˆœ์ˆ˜ CNN vs Residual CNN ์‹คํ—˜ ์ •๋ฆฌ

MNIST ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ ์ˆœ์ˆ˜ CNN์˜ ๊นŠ์ด ํ™•์žฅ ํ•œ๊ณ„(degradation ๋ฌธ์ œ)๋ฅผ ํ™•์ธํ•˜๊ณ , residual connection์ด ์ด๋ฅผ ํ•ด๊ฒฐํ•˜๋Š”์ง€ ๊ฒ€์ฆํ•œ ์‹คํ—˜ ๊ธฐ๋ก.

์‹คํ—˜ ํ™˜๊ฒฝ

  • GPU: NVIDIA GeForce RTX 5070 Ti (16GB)
  • PyTorch 2.11.0+cu128, torchvision 0.26.0
  • ๋ฐ์ดํ„ฐ์…‹: MNIST (60k train / 10k test)
  • ๊ณตํ†ต ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ: Adam (lr=1e-3), CosineAnnealingLR, batch=128, epochs=15
  • depth = ๋ธ”๋ก ์ˆ˜. ๊ฐ ๋ธ”๋ก์€ Conv-BN-ReLU ร—2๋กœ ๊ตฌ์„ฑ๋˜์–ด ์‹ค์ œ conv ๋ ˆ์ด์–ด ์ˆ˜๋Š” 1 + 2ร—depth
  • ์ฑ„๋„ ์ˆ˜ 64 ๊ณ ์ •, stem์—์„œ stride 2๋กœ 28ร—28 โ†’ 14ร—14 ๋‹ค์šด์ƒ˜ํ”Œ ํ›„ ๋™์ผ ํ•ด์ƒ๋„ ์œ ์ง€
  • global average pooling + linear classifier

๋ชจ๋ธ ๊ตฌ์กฐ

PlainBlock:  Conv โ†’ BN โ†’ ReLU โ†’ Conv โ†’ BN โ†’ ReLU
ResBlock:    Conv โ†’ BN โ†’ ReLU โ†’ Conv โ†’ BN โ†’ ( + input ) โ†’ ReLU

--no-bn ์˜ต์…˜์œผ๋กœ BatchNorm์„ Identity๋กœ ๊ต์ฒด ๊ฐ€๋Šฅ (๊ณ ์ „์  degradation ์‹คํ—˜ ์žฌํ˜„์šฉ).

๊ฒฐ๊ณผ

BatchNorm ์žˆ์Œ โ€” best val accuracy

depth (blocks) plain CNN + residual
5 99.70% โ€”
10 99.62% 99.58%
15 99.60% 99.52%
20 99.58% 99.66%
40 98.89% (๋ถˆ์•ˆ์ •) 99.56%

BatchNorm ์—†์Œ โ€” best val accuracy

depth (blocks) plain CNN + residual
20 11.35% (๋ถ•๊ดด) 99.39%
40 11.35% (๋ถ•๊ดด) 99.40%

๊ด€์ฐฐ ๊ฒฐ๊ณผ

1. BN์ด ์žˆ์œผ๋ฉด MNIST์—์„œ๋Š” ํฌํ™”๋จ

๋ชจ๋“  ์„ค์ •์ด ~99.5% ๊ทผ์ฒ˜์— ๋„๋‹ฌํ•ด ์ตœ์ข… ์ •ํ™•๋„ ์ฐจ์ด๋Š” ๋…ธ์ด์ฆˆ ์ˆ˜์ค€. ๋‹ค๋งŒ depth๊ฐ€ ๊นŠ์–ด์งˆ์ˆ˜๋ก ์ดˆ๊ธฐ ์ˆ˜๋ ด์ด ๋А๋ ค์ง€๋Š” ๊ฒฝํ–ฅ์ด ์ผ๊ด€๋˜๊ฒŒ ๊ด€์ฐฐ๋จ:

  • depth5 epoch1: loss=0.23, train_acc=95.2%
  • depth20 epoch1: loss=0.61, train_acc=79.7%
  • depth40 epoch1: loss=1.60, train_acc=37.9%

๋™์ผ ์กฐ๊ฑด์˜ residual ๋ชจ๋ธ์€ ๊นŠ์ด์™€ ๋ฌด๊ด€ํ•˜๊ฒŒ epoch1๋ถ€ํ„ฐ ~96% train_acc๋กœ ๋น ๋ฅด๊ฒŒ ์ˆ˜๋ ด.

2. depth40 plain ๋ชจ๋ธ์˜ ํ•™์Šต ๋ถˆ์•ˆ์ •

epoch 11์—์„œ loss๊ฐ€ 0.031 โ†’ 0.478๋กœ ๊ธ‰๋“ฑํ•˜๋ฉฐ train_acc๊ฐ€ 99.2% โ†’ 84.1%๋กœ ๋ถ•๊ดด. ์ดํ›„ ์žฌ์ˆ˜๋ ดํ–ˆ์œผ๋‚˜ ์ตœ์ข… 98.1%๋กœ ๋‹ค๋ฅธ ๋ชจ๋ธ ๋Œ€๋น„ ๋ช…ํ™•ํžˆ ๋‚ฎ์Œ. residual ๋ชจ๋ธ์€ ๋™์ผ ๊นŠ์ด์—์„œ ๋๊นŒ์ง€ ์•ˆ์ •์ .

3. BN ์ œ๊ฑฐ ์‹œ degradation์ด ๊ทน๋ช…ํ•˜๊ฒŒ ๋“œ๋Ÿฌ๋‚จ

plain CNN (depth20/40, no-BN): 15 epoch ๋‚ด๋‚ด loss=2.30, val_acc=11.35%๋กœ ์™„์ „ ์ •์ง€. ๋žœ๋ค ์ˆ˜์ค€(10%)๊ณผ ๋™์ผ โ€” ๋„คํŠธ์›Œํฌ๊ฐ€ ์‚ฌ์‹ค์ƒ ํ•™์Šต๋˜์ง€ ์•Š์Œ.

residual CNN (depth20/40, no-BN): 99.4% ๋‹ฌ์„ฑ. BN ์—†์ด๋„ ๊นŠ์ด์— ๋ฌด๊ด€ํ•˜๊ฒŒ ์ •์ƒ ํ•™์Šต.

โ†’ residual connection์ด gradient์˜ identity path๋ฅผ ์ œ๊ณตํ•ด ๊นŠ์€ ๋„คํŠธ์›Œํฌ์—์„œ๋„ ์‹ ํ˜ธ ์ „ํŒŒ์™€ ์ตœ์ ํ™”๊ฐ€ ๊ฐ€๋Šฅํ•˜๊ฒŒ ํ•จ์„ ํ™•์ธ.

์žฌํ˜„ ๋ฐฉ๋ฒ•

# ์˜ˆ: depth20 plain, BN ์—†์Œ
./.venv/Scripts/python src/train.py --depth 20 --no-bn --epochs 15

# ์˜ˆ: depth20 residual, BN ์—†์Œ
./.venv/Scripts/python src/train.py --depth 20 --residual --no-bn --epochs 15

# ๊ทธ๋ž˜ํ”„ ์ƒ์„ฑ
./.venv/Scripts/python src/plot_results.py

๊ฒฐ๊ณผ๋Š” runs/depthN[_res][_nobn]/ ์— history.json(epoch๋ณ„ ์ง€ํ‘œ)๊ณผ model.pt๋กœ ์ €์žฅ. ์‹œ๊ฐํ™” ๊ฒฐ๊ณผ๋Š” results/ ์— PNG๋กœ ์ €์žฅ.

๊ฒฐ๋ก 

  • ์ˆœ์ˆ˜ CNN์€ ๊นŠ์ด๋ฅผ ๋Š˜๋ ค๋„ ์„ฑ๋Šฅ์ด ๋ณด์žฅ๋˜์ง€ ์•Š์œผ๋ฉฐ, BN ์œ ๋ฌด์— ๋”ฐ๋ผ ๋ถˆ์•ˆ์ •/๋ถ•๊ดด๊ฐ€ ๋ฐœ์ƒ
  • residual connection์€ ๋™์ผ ๊นŠ์ด์—์„œ ์•ˆ์ •์ ์ด๊ณ  ๋น ๋ฅธ ์ˆ˜๋ ด์„ ๊ฐ€๋Šฅํ•˜๊ฒŒ ํ•จ
  • "๊นŠ๊ฒŒ ์Œ“๋Š” ๊ฒƒ" ์ž์ฒด๋ณด๋‹ค "์‹ ํ˜ธ๊ฐ€ ์ž˜ ํ๋ฅด๋Š” ๊ตฌ์กฐ"๊ฐ€ ๋” ์ค‘์š”ํ•˜๋‹ค๋Š” ๊ฒƒ์„ ์‹คํ—˜์œผ๋กœ ํ™•์ธ