Aether-7B-5Attn / README_ko.md
SeaWolf-AI's picture
sync Aether family cross-links
7196c5d verified
|
Raw
History Blame Contribute Delete
20.4 kB
---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- aether
- mixture-of-experts
- heterogeneous-attention
- korean
- vidraft
- base-model
- fully-open
language:
- ko
- en
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
- open-web-math/open-web-math
- OpenCoder-LLM/opc-fineweb-code-corpus
- HAERAE-HUB/KOREAN-WEBTEXT
- HAERAE-HUB/KOREAN-SyntheticText-1.5B
---
# Aether-7B-5Attn โ€” Base
<!-- AETHER-FAMILY-LINKS -->
**Aether family** โ€” [![5Attn IT](https://img.shields.io/badge/%F0%9F%92%AC_Aether--7B--5Attn--it-Instruct-green)](https://huggingface.co/FINAL-Bench/Aether-7B-5Attn-it) [![7Attn Base](https://img.shields.io/badge/%F0%9F%A7%A9_Aether--7B--7Attn--base-Open_Weights-teal)](https://huggingface.co/FINAL-Bench/Aether-7B-7Attn-base) [![11Attn](https://img.shields.io/badge/%F0%9F%94%9F_Aether--6B--11Attn-Open_Weights-purple)](https://huggingface.co/FINAL-Bench/Aether-6B-11Attn) [![Checkpoints](https://img.shields.io/badge/%F0%9F%97%82%EF%B8%8F_Checkpoints-Intermediate-lightgrey)](https://huggingface.co/datasets/FINAL-Bench/Aether-7B-5Attn-checkpoints) [![Demo](https://img.shields.io/badge/%F0%9F%9A%80_Demo-Aether_Sovereign_AI-orange)](https://huggingface.co/spaces/FINAL-Bench/Aether-Sovereign-AI) [![Blog](https://img.shields.io/badge/%F0%9F%93%9D_Blog-Open_Source_LLM-yellow)](https://huggingface.co/blog/FINAL-Bench/opensource-llm) [![Collection](https://img.shields.io/badge/%F0%9F%93%9A_Collection-Aether_Foundation-blue)](https://huggingface.co/collections/FINAL-Bench/aether-foundation-model)
<!-- /AETHER-FAMILY-LINKS -->
![Aether-7B-5Attn](aether_hero.png)
[![Blog](https://img.shields.io/badge/%F0%9F%A4%97%20Blog-Aether%20์ด์•ผ๊ธฐ%20์ฝ๊ธฐ-FFD21E)](https://huggingface.co/blog/FINAL-Bench/opensource-llm)
> ๐Ÿ“š **[Aether Foundation Model ์ปฌ๋ ‰์…˜](https://huggingface.co/collections/FINAL-Bench/aether-foundation-model) ์˜ ์ผ๋ถ€** โ€” ๋ฒ ์ด์Šคยทinstructยท์ฒดํฌํฌ์ธํŠธ๋ฅผ ํ•œ๊ณณ์—์„œ.
> ๐Ÿงฉ **์ค‘๊ฐ„ ์ฒดํฌํฌ์ธํŠธ** (110k ยท 115k ยท 162k)๋Š” ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ ๊ณต๊ฐœ: [Aether-7B-5Attn-checkpoints](https://huggingface.co/datasets/FINAL-Bench/Aether-7B-5Attn-checkpoints).
## ๐Ÿ‡ฐ๐Ÿ‡ท ํ•™์Šต ๋ฐ์ดํ„ฐ์™€ ํ•™์Šต ์ฝ”๋“œ๊นŒ์ง€ ์ „๋ถ€ ๊ณต๊ฐœํ•˜๋Š” ํ•œ๊ตญ ์ตœ์ดˆ์˜ ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ
**49์ธต์— ์–ดํ…์…˜์„ 7ร—7 ๋ผํ‹ด ๋ฐฉ์ง„์œผ๋กœ ๋ฐฐ์น˜ํ•œ 6.59B MoE ์–ธ์–ด๋ชจ๋ธ.**
๊ฐ€์ค‘์น˜ ยท **ํ•™์Šต ๋ฐ์ดํ„ฐ ๋ ˆ์‹œํ”ผ** ยท **ํ•™์Šต ์ฝ”๋“œ** ยท **ํ•™์Šต ๋กœ๊ทธ ์ „๋ฌธ** ยท **์•„ํ‚คํ…์ฒ˜ ์†Œ์Šค ์ „์ฒด** ๋ฅผ ๋‚จ๊น€์—†์ด ๊ณต๊ฐœํ•ฉ๋‹ˆ๋‹ค.
NVIDIA **B200 16์žฅ**(2-node FSDP)์œผ๋กœ **์•ฝ 46์ผ**, ์ด **144.2B ํ† ํฐ**์„ ์ฒ˜์Œ๋ถ€ํ„ฐ ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
**Apache-2.0**
---
## 1. ์™œ ์ „๋ถ€ ๊ณต๊ฐœํ•˜๋Š”๊ฐ€
์˜ค๋Š˜๋‚  "์˜คํ”ˆ LLM" ๋Œ€๋ถ€๋ถ„์€ **๊ฐ€์ค‘์น˜๋งŒ** ์ค๋‹ˆ๋‹ค. ๊ฐ€์ค‘์น˜๋งŒ์œผ๋กœ๋Š” ๋ชจ๋ธ์„ ์ดํ•ดํ•  ์ˆ˜๋„, ๊ฒ€์ฆํ•  ์ˆ˜๋„, ์ฒ˜์Œ๋ถ€ํ„ฐ ๋‹ค์‹œ ๋งŒ๋“ค ์ˆ˜๋„ ์—†์Šต๋‹ˆ๋‹ค. ๋ฌด์—‡์„ ๋จน์˜€๋Š”์ง€ ๋ชจ๋ฅด๋Š” ๋ชจ๋ธ์€ ๋ธ”๋ž™๋ฐ•์Šค์ผ ๋ฟ์ด๊ณ , ๊ทธ ์œ„์— ๊ตญ๊ฐ€์™€ ๊ธฐ์—…์˜ AI ์ฃผ๊ถŒ์„ ์„ธ์šธ ์ˆ˜๋Š” ์—†์Šต๋‹ˆ๋‹ค.
**Allen Institute์˜ OLMo๋Š” "์ง„์งœ ์˜คํ”ˆ"์˜ ๊ธฐ์ค€์„ ๋‹ค์‹œ ์„ธ์› ์Šต๋‹ˆ๋‹ค** โ€” ๊ฐ€์ค‘์น˜๋ฟ ์•„๋‹ˆ๋ผ ํ•™์Šต ๋ฐ์ดํ„ฐยทํ•™์Šต ์ฝ”๋“œยท๋กœ๊ทธ๊นŒ์ง€ ๊ณต๊ฐœํ•ด์•ผ ๋น„๋กœ์†Œ ๊ณผํ•™์ด ๋œ๋‹ค๋Š” ๊ธฐ์ค€์ž…๋‹ˆ๋‹ค. Aether๋Š” ๊ทธ ๊ธฐ์ค€์„ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
### ์šฐ๋ฆฌ๊ฐ€ ๊ณต๊ฐœํ•˜๋Š” ๊ฒƒ
| | ํ•ญ๋ชฉ | Aether |
|---|------|--------|
| 1 | ๊ฐ€์ค‘์น˜ (annealing ์™„๋ฃŒ ๋ฒ ์ด์Šค) | โœ… |
| 2 | **์•„ํ‚คํ…์ฒ˜ ์†Œ์Šค ์ „์ฒด** โ€” 5์ข… ์–ดํ…์…˜ยทMoEยท๋ผํ‹ด ๋ฐฉ์ง„ ๋ฐฐ์น˜ ๊ตฌํ˜„ | โœ… `aether_pkg/` |
| 3 | **ํ•™์Šต ๋ฐ์ดํ„ฐ ๋ ˆ์‹œํ”ผ** โ€” ์†Œ์Šค ์ €์žฅ์†Œยทconfigยทํ† ํฐ ์ˆ˜ยท๋ฐฐํ•ฉ ๊ฐ€์ค‘์น˜ยทํ† ํฌ๋‚˜์ด์ €ยทEOS | โœ… ยง5 |
| 4 | **ํ† ํฐํ™” ์Šคํฌ๋ฆฝํŠธ** โ€” ์šฐ๋ฆฌ๊ฐ€ ์“ด ๊ทธ ํŒŒ์ผ ๊ทธ๋Œ€๋กœ | โœ… `tokenize_one.py` |
| 5 | **ํ•™์Šต ์ฝ”๋“œ** โ€” FSDP ๋Ÿฐ์ฒ˜ยทํ•™์Šต ๋ฃจํ”„ยท๋…ธ๋“œ ์‹คํ–‰ ์Šคํฌ๋ฆฝํŠธ | โœ… `launcher_v2b_multi.py` ์™ธ |
| 6 | **ํ•™์Šต ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ ์ „์ฒด** | โœ… ยง5 |
| 7 | **ํ•™์Šต ๋กœ๊ทธ ์ „๋ฌธ** โ€” 162,000 ์Šคํ… 30์ผ ๊ธฐ๋ก | โœ… `logs/` |
| 8 | **ํ‰๊ฐ€ ์ฝ”๋“œ** | โœ… `eval/lmeval_run.py` |
| 9 | ์ค‘๊ฐ„ ์ฒดํฌํฌ์ธํŠธ (110k ยท 115k ยท 162k) | โœ… |
| 10 | ๋ผ์ด์„ ์Šค | โœ… **Apache-2.0** |
**ยง5์˜ ๋ ˆ์‹œํ”ผ๋งŒ์œผ๋กœ ์ œ3์ž๊ฐ€ ์šฐ๋ฆฌ ํ•™์Šต ๋ฐ์ดํ„ฐ๋ฅผ ๋ฐ”์ดํŠธ ๋‹จ์œ„๋กœ ๋™์ผํ•˜๊ฒŒ ์žฌ๊ตฌ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.** ์†Œ์Šค๊ฐ€ ์ „๋ถ€ ๊ณต๊ฐœ ์ €์žฅ์†Œ์ด๋ฏ€๋กœ, ํ† ํฌ๋‚˜์ด์ €์™€ EOS์™€ ๋ฐฐํ•ฉ ๊ฐ€์ค‘์น˜๊ฐ€ ๊ฐ™์œผ๋ฉด ๊ฒฐ๊ณผ ๋ฐ”์ด๋„ˆ๋ฆฌ๋„ ๊ฐ™์Šต๋‹ˆ๋‹ค. ๊ฒ€์ฆํ•˜์‹ญ์‹œ์˜ค. ๊ทธ๋Ÿฌ๋ผ๊ณ  ๊ณต๊ฐœํ•ฉ๋‹ˆ๋‹ค.
### ํ•œ๊ตญ ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ ์ตœ์ดˆ์ž…๋‹ˆ๋‹ค
ํ•œ๊ตญ์—์„œ ์ฒ˜์Œ๋ถ€ํ„ฐ ํ•™์Šต๋œ ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ์€ ์ด์ „์—๋„ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ **ํ•™์Šต ๋ฐ์ดํ„ฐ๋ฅผ ๊ณต๊ฐœํ•œ ๋ชจ๋ธ์€ ์—†์—ˆ์Šต๋‹ˆ๋‹ค.** ์šฐ๋ฆฌ๊ฐ€ ์•„๋Š” ํ•œ Aether๊ฐ€ ์ฒ˜์Œ์ž…๋‹ˆ๋‹ค.
์ด๊ฒƒ์ด **์†Œ๋ฒ„๋ฆฐ AI**์˜ ์‹ค์งˆ์ž…๋‹ˆ๋‹ค. ์–ด๋А ๋‚˜๋ผ, ์–ด๋А ๊ธฐ์—…, ์–ด๋А ์—ฐ๊ตฌ์‹ค์ด๋“  ์ด ์ €์žฅ์†Œ๋งŒ์œผ๋กœ ์ž์‹ ์˜ ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ์„ **์ฒ˜์Œ๋ถ€ํ„ฐ ๋‹ค์‹œ** ๋งŒ๋“ค ์ˆ˜ ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ€์ค‘์น˜๋งŒ ๋ฐ›์•„ ์“ฐ๋Š” ๊ฒƒ์€ ์ฃผ๊ถŒ์ด ์•„๋‹™๋‹ˆ๋‹ค.
---
### ์˜คํ”ˆ์†Œ์Šค ์™„์ „๊ณต๊ฐœ LLM ๋น„๊ต (6๊ฐœ๊ตญ)
![Open-source LLM comparison](opensource_llm_comparison.png)
*์†Œ๋ฒ„๋ฆฐ ์™„์ „๊ณต๊ฐœ LLM 6๊ฐœ๊ตญ ๋Œ€๋น„. ๊ฐœ๋ฐฉ ํ•ญ๋ชฉ์€ ๋ชจ๋‘ ๋Œ€๋“ฑํ•˜๋ฉฐ, ๋น„๋“œ๋ž˜ํ”„ํŠธ์˜ ์ฐจ๋ณ„์ ์€ ์œ ์ผํ•œ ํ•œ๊ตญ AI ์Šคํƒ€ํŠธ์—…์ด๋ผ๋Š” ์ ๊ณผ 5์ข…(์ตœ๋‹ค) ์–ดํ…์…˜ยท๋ผํ‹ด๋ฐฉ์ง„ ๊ตฌ์กฐ์ž…๋‹ˆ๋‹ค.*
## 2. ์„ค๊ณ„ โ€” ์™œ ๋ชจ๋“  ์ธต์ด ๊ฐ™์€ ์–ดํ…์…˜์ด์–ด์•ผ ํ•˜๋Š”๊ฐ€
### 2.1 ํ•˜๋‚˜์˜ ์–ดํ…์…˜์ด ๋ชจ๋“  ์ธต์„ ๋‹ด๋‹นํ•ด์•ผ ํ•  ์ด์œ ๋Š” ์—†์Šต๋‹ˆ๋‹ค
GPT ์ดํ›„ ๊ฑฐ์˜ ๋ชจ๋“  ํŠธ๋žœ์Šคํฌ๋จธ๋Š” **๋ชจ๋“  ์ธต์—์„œ ๋˜‘๊ฐ™์€ ์–ดํ…์…˜์„ ์”๋‹ˆ๋‹ค.** 49์ธต์งœ๋ฆฌ ๋ชจ๋ธ์ด๋ผ๋ฉด 49๋ฒˆ ๊ฐ™์€ ์—ฐ์‚ฐ์„ ๋ฐ˜๋ณตํ•ฉ๋‹ˆ๋‹ค.
๊ทธ๋Ÿฐ๋ฐ ์ธต๋งˆ๋‹ค ํ•˜๋Š” ์ผ์€ ๋‹ค๋ฆ…๋‹ˆ๋‹ค. ์ดˆ๊ธฐ ์ธต์€ ๊ตญ์†Œ์ ์ธ ๋ฌธ๋ฒ•ยทํ˜•ํƒœ์†Œ ํŒจํ„ด์„ ์žก๊ณ , ์ค‘๊ฐ„ ์ธต์€ ๊ตฌ๋ฌธ ๊ตฌ์กฐ๋ฅผ, ํ›„๊ธฐ ์ธต์€ ๋ฌธ์„œ ์ „์ฒด๋ฅผ ๊ฐ€๋กœ์ง€๋ฅด๋Š” ์˜๋ฏธ ๊ด€๊ณ„๋ฅผ ๋‹ค๋ฃน๋‹ˆ๋‹ค. **ํ•˜๋Š” ์ผ์ด ๋‹ค๋ฅธ๋ฐ ์™œ ๊ฐ™์€ ์—ฐ์‚ฐ์„ ๊ฐ•์š”๋ฐ›์•„์•ผ ํ•ฉ๋‹ˆ๊นŒ.**
์ด๊ฑด ์„ฑ๋Šฅ ๋ฌธ์ œ ์ด์ „์— **์„ค๊ณ„ ๊ฐ€์ •์˜ ๋ฌธ์ œ**์ž…๋‹ˆ๋‹ค. "๋ชจ๋“  ์ธต = ๊ฐ™์€ ์–ดํ…์…˜"์€ ๊ฒ€์ฆ๋œ ๊ฒฐ๋ก ์ด ์•„๋‹ˆ๋ผ **ํ•œ ๋ฒˆ๋„ ์ง„์ง€ํ•˜๊ฒŒ ์˜์‹ฌ๋ฐ›์ง€ ์•Š์€ ๊ด€ํ–‰**์ž…๋‹ˆ๋‹ค. Aether๋Š” ๊ทธ ๊ฐ€์ •์„ ํ’€์–ด๋ด…๋‹ˆ๋‹ค.
### 2.2 ์–ดํ…์…˜์€ ์ €๋งˆ๋‹ค ๋‹ค๋ฅธ ๊ท€๋‚ฉ ํŽธํ–ฅ์„ ๊ฐ€์ง‘๋‹ˆ๋‹ค
| ์ข…๋ฅ˜ | ์ž˜ํ•˜๋Š” ๊ฒƒ | ๋Œ€๊ฐ€ |
|------|-----------|------|
| `full` | ๋ชจ๋“  ํ† ํฐ ์Œ์„ ์ •ํ™•ํžˆ ๋ด…๋‹ˆ๋‹ค. ์ •๋ณด ์†์‹ค 0 | ๊ธธ์ด์— **์ œ๊ณฑ**์œผ๋กœ ๋น„์Œ‰๋‹ˆ๋‹ค |
| `sliding` | ๊ตญ์†Œ์„ฑ์„ ์•„์ฃผ ์‹ธ๊ฒŒ ๋ด…๋‹ˆ๋‹ค (๊ธธ์ด์— ์„ ํ˜•) | ์œˆ๋„์šฐ ๋ฐ–์€ ๋ชป ๋ด…๋‹ˆ๋‹ค |
| `differential` | ๋‘ ์–ดํ…์…˜ ๋งต์˜ **์ฐจ์ด**๋ฅผ ์ทจํ•ด ๊ณตํ†ต ๋…ธ์ด์ฆˆ๋ฅผ ์ƒ์‡„ํ•ฉ๋‹ˆ๋‹ค | ํ—ค๋“œ ์ฐจ์›์„ ๋ฐ˜์œผ๋กœ ๋‚˜๋ˆ  ์”๋‹ˆ๋‹ค |
| `nsa` | ์••์ถ•ยท์„ ํƒยท์Šฌ๋ผ์ด๋”ฉ์„ ๊ฒŒ์ดํŠธ๋กœ ๋ฌถ์–ด ์žฅ๊ฑฐ๋ฆฌ๋ฅผ ์‹ธ๊ฒŒ ๋ด…๋‹ˆ๋‹ค | ๊ตฌ์กฐ๊ฐ€ ๋ณต์žกํ•ฉ๋‹ˆ๋‹ค |
| `hybrid` | `nsa`์˜ ์žฅ๊ฑฐ๋ฆฌ์™€ `differential`์˜ ๋…ธ์ด์ฆˆ ์–ต์ œ๋ฅผ ํ•จ๊ป˜ ์”๋‹ˆ๋‹ค | ๊ฐ€์žฅ ๋น„์Œ‰๋‹ˆ๋‹ค |
**์–ด๋А ํ•˜๋‚˜๊ฐ€ ๋‹ค๋ฅธ ๊ฒƒ๋“ค์„ ์ง€๋ฐฐํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.** ๊ฐ์ž ์ž˜ํ•˜๋Š” ๊ฒŒ ๋‹ค๋ฅด๊ณ  ๋Œ€๊ฐ€๋„ ๋‹ค๋ฆ…๋‹ˆ๋‹ค. ๊ทธ๋ ‡๋‹ค๋ฉด **ํ•œ ์ข…๋ฅ˜๋งŒ ๊ณจ๋ผ 49๋ฒˆ ๋ฐ˜๋ณตํ•˜๋Š” ๊ฒƒ์€, ๊ทธ ์ข…๋ฅ˜์˜ ์•ฝ์ ๋„ 49๋ฒˆ ๋ฐ˜๋ณตํ•˜๋Š” ๊ฒƒ**์ž…๋‹ˆ๋‹ค.
### 2.3 ๋ผํ‹ด ๋ฐฉ์ง„ โ€” ๊ฐ€์ •์„ ๊ณต์ •ํ•˜๊ฒŒ ์‹œํ—˜ํ•˜๋Š” ์žฅ์น˜
"์„ž์œผ๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™๋‹ค"๋Š” ์ง๊ด€์€ ์‰ฝ์ง€๋งŒ, ์•„๋ฌด๋ ‡๊ฒŒ๋‚˜ ์„ž์œผ๋ฉด **๊ฒฐ๊ณผ๋ฅผ ๋ฐฐ์น˜ ํƒ“์ธ์ง€ ์ข…๋ฅ˜ ํƒ“์ธ์ง€ ๊ตฌ๋ถ„ํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.** `full`์ด ์šฐ์—ฐํžˆ ํ›„๊ธฐ ์ธต์— ๋ชฐ๋ฆฌ๋ฉด ๊ทธ ๋ชจ๋ธ์€ "์ด์ข…์ด๋ผ์„œ" ์ž˜ํ•œ ๊ฒŒ ์•„๋‹ˆ๋ผ "ํ›„๊ธฐ ์ธต์ด full์ด๋ผ์„œ" ์ž˜ํ•œ ๊ฒƒ์ด ๋ฉ๋‹ˆ๋‹ค.
49์ธต = **7 ร— 7 ๋ผํ‹ด ๋ฐฉ์ง„**์œผ๋กœ ๊ตฌ์„ฑํ•œ ์ด์œ ๊ฐ€ ์ด๊ฒƒ์ž…๋‹ˆ๋‹ค. ๋ผํ‹ด ๋ฐฉ์ง„์€ **๊ฐ ์ข…๋ฅ˜๊ฐ€ ๊ฐ ์œ„์น˜์— ์ •ํ™•ํžˆ ํ•œ ๋ฒˆ์”ฉ** ๋‚˜ํƒ€๋‚˜๋„๋ก ๋ณด์žฅํ•˜๋Š” ์กฐํ•ฉ๋ก ์  ๋ฐฐ์น˜์ž…๋‹ˆ๋‹ค. ํŠน์ • ์ข…๋ฅ˜๊ฐ€ ํŠน์ • ๊นŠ์ด์— ํŽธ์ค‘๋˜๋Š” ๊ฒƒ์„ **๊ตฌ์กฐ์ ์œผ๋กœ** ๋ง‰์Šต๋‹ˆ๋‹ค.
์ฆ‰ ๋ผํ‹ด ๋ฐฉ์ง„์€ ์žฅ์‹์ด ์•„๋‹ˆ๋ผ **ํ†ต์ œ ์žฅ์น˜**์ž…๋‹ˆ๋‹ค. ์ด ๋ฐฐ์น˜ ๋•๋ถ„์— "์ด์ข… ๋ฐฐ์น˜์˜ ํšจ๊ณผ"๋ผ๋Š” ์งˆ๋ฌธ์„ ๋ฐฐ์น˜ ํŽธํ–ฅ ์—†์ด ๋˜์งˆ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
**์šฐ๋ฆฌ๊ฐ€ ์•„๋Š” ํ•œ, ์ด์ข… ์–ดํ…์…˜์„ ๋ผํ‹ด ๋ฐฉ์ง„์œผ๋กœ ๋ฐฐ์น˜ํ•œ ๊ณต๊ฐœ ๋ชจ๋ธ์€ ์—†์Šต๋‹ˆ๋‹ค.**
---
## 3. ์–ดํ…์…˜ ๊ตฌ์„ฑ โ€” ๋ผํ‹ด ๋ฐฉ์ง„ 7ร—7, 49์ธต
49์ธต์— 7๊ฐœ์˜ ์–ดํ…์…˜ ๋ผ๋ฒจ์ด ๋ผํ‹ด ๋ฐฉ์ง„์„ ๋”ฐ๋ผ, **๊ฐ๊ฐ ์ •ํ™•ํžˆ 7๊ฐœ ์ธต์”ฉ** ๋ฐฐ์น˜๋ฉ๋‹ˆ๋‹ค (7 ร— 7 = 49).
| ๋ผ๋ฒจ | ์ธต ์ˆ˜ | ์„ค๋ช… |
|------|------|------|
| `full` | 7 | ํ‘œ์ค€ ์ธ๊ณผ ์–ดํ…์…˜ (SDPA) |
| `differential` | 7 | QยทK๋ฅผ ๋ฐ˜์œผ๋กœ ๋‚˜๋ˆ  ๋‘ ์–ดํ…์…˜ ๋งต์„ ๋งŒ๋“ค๊ณ  ฮป ๊ฐ€์ค‘์œผ๋กœ ๋นผ์„œ ๊ณตํ†ต ๋…ธ์ด์ฆˆ๋ฅผ ์ƒ์‡„ |
| `sliding` | 7 | ์Šฌ๋ผ์ด๋”ฉ ์œˆ๋„์šฐ (window = 512) โ€” full ๊ฒฝ๋กœ ์œ„์˜ ๋งˆ์Šคํ‚น |
| `nsa` | 7 | ์••์ถ•ยท์„ ํƒยท์Šฌ๋ผ์ด๋”ฉ ๋ธŒ๋žœ์น˜๋ฅผ ํ•™์Šต๋œ ๊ฒŒ์ดํŠธ๋กœ ๊ฒฐํ•ฉ |
| `hybrid` | 7 | `nsa` + `differential` ๊ฒฐํ•ฉ |
| `compress` | 7 | NSA ๋ธ”๋กํ‰๊ท  ๊ฒฝ๋กœ โ€” ๋ณธ ์ฒดํฌํฌ์ธํŠธ์—์„œ๋Š” full ๊ฒฝ๋กœ๋กœ ๋™์ž‘ |
| `linear` | 7 | SDPA + ํ•™์Šต ๊ฒŒ์ดํŠธ |
| **ํ•ฉ๊ณ„** | **49** | |
**๋ชจ๋ธ๋ช… `5Attn`์˜ ์œ ๋ž˜.** ์œ„ 7๊ฐœ ๋ผ๋ฒจ์€ **5์ข…์˜ ์„œ๋กœ ๋‹ค๋ฅธ ๋ฉ”์ปค๋‹ˆ์ฆ˜**์œผ๋กœ ํ™˜์›๋ฉ๋‹ˆ๋‹ค โ€” `compress`๋Š” full ๊ฒฝ๋กœ๋กœ, `linear`๋Š” SDPA + ๊ฒŒ์ดํŠธ๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค. ๋ชจ๋ธ๋ช…์€ ๋ผ๋ฒจ ์ˆ˜(7)๊ฐ€ ์•„๋‹ˆ๋ผ **๋ฉ”์ปค๋‹ˆ์ฆ˜ ์ˆ˜(5)**๋ฅผ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
๋ฐฐ์น˜ ๊ทœ์น™๊ณผ ๊ฐ ๋ผ๋ฒจ์˜ ๊ตฌํ˜„ ์ „์ฒด๋Š” `aether_pkg/` ์— ๊ณต๊ฐœ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. `LATIN_SQUARE_7x7` ์—์„œ ์ธตโ†”๋ผ๋ฒจ ๋Œ€์‘์„ ์ง์ ‘ ํ™•์ธํ•˜์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
### 3.1 ๋ฉ”์ปค๋‹ˆ์ฆ˜๋ณ„ ๋น„์šฉ ํ”„๋กœํŒŒ์ผ (์‹ค์ธก)
5์ข… ๋ฉ”์ปค๋‹ˆ์ฆ˜ ๊ฐ๊ฐ์„ ๋‹จ๋… 1๊ฐœ ์ธต์œผ๋กœ ๋†“๊ณ  prefill ์ง€์—ฐ์‹œ๊ฐ„๊ณผ ์ตœ๋Œ€ ๋ฉ”๋ชจ๋ฆฌ๋ฅผ ์ธก์ •ํ–ˆ์Šต๋‹ˆ๋‹ค. ์–ดํ…์…˜์˜ ์—ฐ์‚ฐยท๋ฉ”๋ชจ๋ฆฌ ๋น„์šฉ์€ **ํ•™์Šต๋œ ๊ฐ€์ค‘์น˜ ๊ฐ’๊ณผ ๋ฌด๊ด€ํ•œ ์•„ํ‚คํ…์ฒ˜ ๊ณ ์œ  ์†์„ฑ**์ด๋ฏ€๋กœ, ์ด ํ‘œ๋Š” ๋ˆ„๊ตฌ๋‚˜ ์žฌํ˜„ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
| ์ข…๋ฅ˜ | 2K (ms / GB) | 8K (ms / GB) | 32K (ms / GB) |
|------|--------------|--------------|---------------|
| `full` | 0.4 / 0.0 | 1.5 / 0.2 | 13.6 / 0.7 |
| `differential` | 0.5 / 0.1 | 3.7 / 0.3 | 46.5 / 1.1 |
| **`sliding`** | 0.6 / 0.1 | 1.9 / 0.2 | **7.6 / 0.8** |
| `nsa` | 1.0 / 0.1 | 3.6 / 0.3 | 26.8 / 3.5 |
| `hybrid` | 1.5 / 0.1 | 7.7 / 0.3 | 74.7 / 3.5 |
**ยง2.2์—์„œ ๋งํ•œ "์ข…๋ฅ˜๋งˆ๋‹ค ๋‹ค๋ฅธ ๋Œ€๊ฐ€"๊ฐ€ ์ˆซ์ž๋กœ ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค.**
- **32K์—์„œ `sliding`(7.6ms)์ด `full`(13.6ms)๋ณด๋‹ค 1.8๋ฐฐ ๋น ๋ฆ…๋‹ˆ๋‹ค.** ๊ตญ์†Œ์„ฑ๋งŒ ๋ณด๋„๋ก ์„ค๊ณ„๋œ ๋Œ€๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค. ๋ฌธ๋งฅ์ด ๊ธธ์–ด์งˆ์ˆ˜๋ก ๊ฒฉ์ฐจ๊ฐ€ ๋ฒŒ์–ด์ง‘๋‹ˆ๋‹ค (2K์—์„œ๋Š” ์˜คํžˆ๋ ค `full`์ด ๋น ๋ฆ…๋‹ˆ๋‹ค).
- `hybrid`๋Š” `nsa`์™€ `differential`์„ **๋‘˜ ๋‹ค** ์‹คํ–‰ํ•˜๋ฏ€๋กœ ๋น„์šฉ๋„ ๋‘˜์˜ ํ•ฉ์— ์ˆ˜๋ ดํ•ฉ๋‹ˆ๋‹ค (26.8 + 46.5 โ‰ˆ 74.7).
- ๋ชจ๋ธ ์ „์ฒด๋Š” 32K ๋ฌธ๋งฅ์„ **3.46 GB**๋กœ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
์šฐ๋ฆฌ๊ฐ€ ์•„๋Š” ํ•œ **์ด์ข… ์–ดํ…์…˜ ์ข…๋ฅ˜๋ณ„ ๋น„์šฉ ํ”„๋กœํŒŒ์ผ์„ ์‹ค์ธกํ•ด ๊ณต๊ฐœํ•œ ์‚ฌ๋ก€๋Š” ์—†์Šต๋‹ˆ๋‹ค.** ์ด ํ‘œ ์ž์ฒด๊ฐ€ ํ›„์† ์—ฐ๊ตฌ๋ฅผ ์œ„ํ•œ ์ž๋ฃŒ์ž…๋‹ˆ๋‹ค.
> ์ด ํ‘œ๋Š” **๋น„์šฉ**๋งŒ ๋งํ•ฉ๋‹ˆ๋‹ค. ์ถœ๋ ฅ ํ’ˆ์งˆ์— ๋Œ€ํ•ด์„œ๋Š” ์•„๋ฌด๊ฒƒ๋„ ์ฃผ์žฅํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ํ’ˆ์งˆ ๋น„๊ต์—๋Š” ์–ดํ…์…˜ ๊ตฌ์„ฑ๋งŒ ๋‹ค๋ฅด๊ณ  ๋‚˜๋จธ์ง€๊ฐ€ ์ „๋ถ€ ๋™์ผํ•œ ํ†ต์ œ ์‹คํ—˜์ด ํ•„์š”ํ•˜๋ฉฐ, ๊ทธ๊ฒƒ์€ ๋ณ„๋„๋กœ ์ˆ˜ํ–‰๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.
---
## 3.2 ๊ตฌ์กฐ ์‹ฌํ™” โ€” 7ร—7 ๋ผํ‹ด ์ง๊ต ๋ฐฐ์น˜
![7x7 Latin-Square Attention Placement](latin_square_7x7.png)
49๊ฐœ ์ธต์€ **7 ร— 7 ๋ผํ‹ด ๋ฐฉ์ง„(Latin square)** ์„ ์ด๋ฃน๋‹ˆ๋‹ค. ๊ฐ ์ธต์˜ ์–ดํ…์…˜ ์ข…๋ฅ˜๋Š” ๋‹ค์Œ ๊ทœ์น™์œผ๋กœ ์ •ํ•ด์ง‘๋‹ˆ๋‹ค:
```
attention_type(layer) = ATTN_TYPES[(row + col) mod 7]
๋‹จ, row = layer // 7, col = layer % 7
```
### ๋ผํ‹ด ๋ฐฉ์ง„์˜ ์„ฑ์งˆ (๊ตฌ์กฐ์  ํŠน์ด์„ฑ)
- **๊ฐ ์ข…๋ฅ˜๊ฐ€ ๋ชจ๋“  ํ–‰์— ์ •ํ™•ํžˆ ํ•œ ๋ฒˆ, ๋ชจ๋“  ์—ด์—๋„ ์ •ํ™•ํžˆ ํ•œ ๋ฒˆ** ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค (7์ข… ร— 7์ธต = 49). ์ด๊ฒƒ์ด "๋ผํ‹ด ๋ฐฉ์ง„"์˜ ์ •์˜์ž…๋‹ˆ๋‹ค.
- ๊ทธ ๊ฒฐ๊ณผ **์–ด๋–ค ์–ดํ…์…˜ ์ข…๋ฅ˜๋„ ํŠน์ • ๊นŠ์ด(์ดˆ๊ธฐยท์ค‘๊ฐ„ยทํ›„๊ธฐ ์ธต)์— ํŽธ์ค‘๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.** ์ˆœํ™˜ ์ด๋™(+1)์œผ๋กœ ๋ฐฐ์น˜๋˜์–ด ๊นŠ์ด ๋ฐฉํ–ฅ์œผ๋กœ ๊ณ ๋ฅด๊ฒŒ ๋ถ„์‚ฐ๋ฉ๋‹ˆ๋‹ค.
- **์™œ 7์ธ๊ฐ€**: 7์€ ์†Œ์ˆ˜(prime)์ด๋ฏ€๋กœ ์ˆœํ™˜ ์ด๋™ `(row+col) mod 7` ์ด ์™„์ „ํ•œ ๋ผํ‹ด ๋ฐฉ์ง„(๊ฐ ํ–‰ยท์—ด์ด 7์ข…์˜ ์ˆœ์—ด)์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค. 7ยฒ=49์ธต์œผ๋กœ ๋”ฑ ๋–จ์–ด์ง‘๋‹ˆ๋‹ค.
### ์™œ ์ด๋ ‡๊ฒŒ ๋ฐฐ์น˜ํ•˜๋Š”๊ฐ€ โ€” ํ†ต์ œ ์žฅ์น˜
"์„ž์œผ๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™๋‹ค"๋Š” ์ง๊ด€์€ ์‰ฝ์ง€๋งŒ, ์•„๋ฌด๋ ‡๊ฒŒ๋‚˜ ์„ž์œผ๋ฉด **๊ฒฐ๊ณผ๊ฐ€ ๋ฐฐ์น˜ ํŽธํ–ฅ ํƒ“์ธ์ง€ ์ข…๋ฅ˜ ํƒ“์ธ์ง€ ๊ตฌ๋ถ„ํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.** `full` ์ด ์šฐ์—ฐํžˆ ํ›„๊ธฐ ์ธต์— ๋ชฐ๋ฆฌ๋ฉด "์ด์ข…์ด๋ผ์„œ" ์ž˜ํ•œ ๊ฒŒ ์•„๋‹ˆ๋ผ "ํ›„๊ธฐ ์ธต์ด full ์ด๋ผ์„œ" ์ž˜ํ•œ ๊ฒƒ์ด ๋ฉ๋‹ˆ๋‹ค.
**๋ผํ‹ด ๋ฐฉ์ง„์€ ์ด ํŽธํ–ฅ์„ ๊ตฌ์กฐ์ ์œผ๋กœ ์ œ๊ฑฐํ•ฉ๋‹ˆ๋‹ค.** ๊ฐ ์ข…๋ฅ˜๊ฐ€ ๋ชจ๋“  ๊นŠ์ด๋ฅผ ์ •ํ™•ํžˆ ํ•œ ๋ฒˆ์”ฉ ๊ฑฐ์น˜๋ฏ€๋กœ, "์ด์ข… ๋ฐฐ์น˜์˜ ํšจ๊ณผ"๋ผ๋Š” ์งˆ๋ฌธ์„ **๊นŠ์ด ํŽธํ–ฅ ์—†์ด** ๋˜์งˆ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฆ‰ ๋ผํ‹ด ๋ฐฉ์ง„์€ ์„ฑ๋Šฅ์„ ์œ„ํ•œ ์žฅ์‹์ด ์•„๋‹ˆ๋ผ **์žฌํ˜„ ๊ฐ€๋Šฅํ•œ ๊ณผํ•™์„ ์œ„ํ•œ ํ†ต์ œ ์žฅ์น˜**์ž…๋‹ˆ๋‹ค.
### 7๊ฐœ ๋ผ๋ฒจ โ†’ 5๊ฐœ ๋ฉ”์ปค๋‹ˆ์ฆ˜
๋ฐฉ์ง„์— ๋ฐฐ์น˜๋˜๋Š” ๋ผ๋ฒจ์€ 7๊ฐœ(`nsa`ยท`differential`ยท`full`ยท`linear`ยท`sliding`ยท`compress`ยท`hybrid`)์ด์ง€๋งŒ, **์„œ๋กœ ๋‹ค๋ฅธ ๋ฉ”์ปค๋‹ˆ์ฆ˜์€ 5๊ฐœ**์ž…๋‹ˆ๋‹ค โ€” `compress` ๋Š” full ๊ฒฝ๋กœ๋กœ, `linear` ๋Š” SDPA+๊ฒŒ์ดํŠธ๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค(๊ทธ๋ฆผ์—์„œ ๋น—๊ธˆ). ๋ชจ๋ธ๋ช… `5Attn` ์€ ๋ผ๋ฒจ ์ˆ˜(7)๊ฐ€ ์•„๋‹ˆ๋ผ **๋ฉ”์ปค๋‹ˆ์ฆ˜ ์ˆ˜(5)** ๋ฅผ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
*์šฐ๋ฆฌ๊ฐ€ ์•„๋Š” ํ•œ, ์ด์ข… ์–ดํ…์…˜์„ ๋ผํ‹ด ๋ฐฉ์ง„์œผ๋กœ ๋ฐฐ์น˜ํ•œ ๊ณต๊ฐœ ๋ชจ๋ธ์€ ์—†์Šต๋‹ˆ๋‹ค.*
## 4. ์‚ฌ์–‘
| ํ•ญ๋ชฉ | ๊ฐ’ |
|------|-----|
| ์ด ํŒŒ๋ผ๋ฏธํ„ฐ | **6.59B** |
| ํ™œ์„ฑ ํŒŒ๋ผ๋ฏธํ„ฐ | **~2.98B** (ํ† ํฐ๋‹น) |
| ์ธต | 49 (7ร—7 ๋ผํ‹ด ๋ฐฉ์ง„) |
| ์ „๋ฌธ๊ฐ€ | 25๊ฐœ, top-7 ๋ผ์šฐํŒ…, ๊ณต์œ  ์ „๋ฌธ๊ฐ€ 1๊ฐœ |
| expert intermediate | 640 |
| hidden / intermediate | 2048 / 6144 |
| heads / KV heads / head_dim | 16 / 4 / 128 |
| ์–ดํœ˜ | 151,936 (Qwen ํ˜ธํ™˜ ํ† ํฌ๋‚˜์ด์ €) |
| ํ•™์Šต ์ปจํ…์ŠคํŠธ | 4096 |
| dtype | bfloat16 |
| ํฌ๊ธฐ | 13.2 GB |
| ํด๋ž˜์Šค | `AETHERV27wayForCausalLM` (`trust_remote_code=True`) |
---
## 5. ํ•™์Šต ๋ฐ์ดํ„ฐ โ€” ์ „์ฒด ๋ ˆ์‹œํ”ผ
**์•„๋ž˜ ์ •๋ณด๋งŒ์œผ๋กœ ์ œ3์ž๊ฐ€ ํ•™์Šต ๋ฐ์ดํ„ฐ๋ฅผ ๋ฐ”์ดํŠธ ๋‹จ์œ„๋กœ ๋™์ผํ•˜๊ฒŒ ์žฌ๊ตฌ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.**
### 5.1 ์†Œ์Šค
| ํŒŒ์ผ | ์ €์žฅ์†Œ | config | ํ† ํฐ | ๋ผ์ด์„ ์Šค |
|------|--------|--------|------|----------|
| `eng_fineweb_edu.bin` | `HuggingFaceFW/fineweb-edu` | `sample-100BT` | 15.000B | ODC-By |
| `syn_cosmopedia.bin` | `HuggingFaceTB/smollm-corpus` | `cosmopedia-v2` | 8.000B | ODC-By |
| `math_finemath.bin` | `HuggingFaceTB/finemath` | `finemath-3plus` | 6.002B | ODC-By |
| `code_opc.bin` | `OpenCoder-LLM/opc-fineweb-code-corpus` | default | 5.001B | MIT |
| `math_owm.bin` | `open-web-math/open-web-math` | default | 4.004B | ์› ์ €์žฅ์†Œ ์ฐธ์กฐ |
| `kor_webtext.bin` | `HAERAE-HUB/KOREAN-WEBTEXT` | default | 2.492B | ์› ์ €์žฅ์†Œ ์ฐธ์กฐ |
| `kor_synth.bin` | `HAERAE-HUB/KOREAN-SyntheticText-1.5B` | default | 1.650B | ์› ์ €์žฅ์†Œ ์ฐธ์กฐ |
| `phase15_mix_90b.bin` | ์œ„ ์†Œ์Šค๋“ค์˜ ์„ ํ–‰ ํ˜ผํ•ฉ๋ณธ | โ€” | 90B | โ€” |
์ „์ฒ˜๋ฆฌ: `AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")`, ๋ฌธ์„œ ๊ฒฝ๊ณ„์— **EOS 151645** ์‚ฝ์ž…, `uint32` ๋ฐฐ์—ด๋กœ ๊ธฐ๋ก.
์žฌํ˜„ ์Šคํฌ๋ฆฝํŠธ `tokenize_one.py` ๋ฅผ ํ•จ๊ป˜ ๊ณต๊ฐœํ•ฉ๋‹ˆ๋‹ค.
```bash
python tokenize_one.py HuggingFaceFW/fineweb-edu sample-100BT text 15 out/eng_fineweb_edu.bin
```
### 5.2 ๋ฐฐํ•ฉ๋น„
```python
DATA_POOL = [ # (ํŒŒ์ผ, ์ƒ˜ํ”Œ๋ง ๊ฐ€์ค‘์น˜)
("phase15_mix_90b.bin", 1.0),
("eng_fineweb_edu.bin", 2.0),
("syn_cosmopedia.bin", 2.0),
("math_finemath.bin", 3.5),
("math_owm.bin", 3.5),
("code_opc.bin", 2.5),
("kor_webtext.bin", 2.0),
("kor_synth.bin", 2.0),
]
```
๊ฐ€์ค‘์น˜ ํ•ฉ 18.5 ๊ธฐ์ค€ ์‹คํšจ ๋น„์ค‘:
| ์˜์—ญ | ๋น„์ค‘ |
|------|------|
| ์ˆ˜ํ•™ (finemath + open-web-math) | 37.8% |
| ํ•œ๊ตญ์–ด (webtext + synth) | 21.6% |
| ์˜์–ด ์›นยทํ•ฉ์„ฑ (fineweb-edu + cosmopedia) | 21.6% |
| ์ฝ”๋“œ (opc) | 13.5% |
| phase15 ํ˜ผํ•ฉ | 5.4% |
---
## 6. ํ•™์Šต
| ํ•ญ๋ชฉ | ๊ฐ’ |
|------|-----|
| **ํ•˜๋“œ์›จ์–ด** | **NVIDIA B200 ร— 16** (2-node FSDP) |
| **์ „์ฒด ํ•™์Šต ๊ธฐ๊ฐ„** | **2026-05-30 โ†’ 2026-07-16, ์•ฝ 46์ผ** |
| **์ตœ์ข… ๋‹จ๊ณ„** | 2026-06-15 โ†’ 07-16, **30์ผ 11์‹œ๊ฐ„**, B200 16์žฅ, **์•ฝ 11,700 B200-์‹œ๊ฐ„** |
| **์ฒ˜๋ฆฌ๋Ÿ‰** (์ตœ์ข… ๋‹จ๊ณ„) | ์•ฝ 32,000 tok/s |
| ์ด ์Šคํ… | 162,000 |
| ์ด ํ† ํฐ | **144.2B** |
| ์˜ตํ‹ฐ๋งˆ์ด์ € | AdamW, ฮฒ=(0.9, 0.95), eps=1e-8 |
| LR | cosine, peak 5e-5 โ†’ min 5e-6 |
| weight decay | 0.1 (ํŠธ๋žœ์Šคํฌ๋จธ ์ธต) / 0.0 (์ž„๋ฒ ๋”ฉยทnorm) |
| ์ž„๋ฒ ๋”ฉ LR ๋ฐฐ์ˆ˜ | 0.1 |
| Layer-wise decay | 0.97 |
| grad clip | 1.0 |
| ์‹œํ€€์Šค | 4096 |
| ํ›„์† | annealing (๋ณธ ์ฒดํฌํฌ์ธํŠธ๋Š” annealing ์™„๋ฃŒ๋ณธ) |
ํ•™์Šต ์ฝ”๋“œ(`launcher_v2b_multi.py`, `launcher_v2b_multi.py`, `run_s1.sh`, `run_s2.sh`)์™€ **์™„์ฃผ ๋กœ๊ทธ ์ „๋ฌธ**์„ ํ•จ๊ป˜ ๊ณต๊ฐœํ•ฉ๋‹ˆ๋‹ค.
---
## 7. ํ‰๊ฐ€
**lm-evaluation-harness 0.4.11**, 0-shot, n=600, `batch_size=1`.
### 7.1 ์˜์–ด
| ๊ณผ์ œ | acc | acc_norm |
|------|-----|----------|
| **SciQ** | **73.7** | 63.0 |
| **PIQA** | **66.3** | 65.8 |
| BoolQ | 54.3 | โ€” |
| **ARC-Easy** | **52.5** | 48.7 |
| WinoGrande | 51.8 | โ€” |
| HellaSwag | 37.2 | 41.0 |
| OpenBookQA | 20.0 | 32.6 |
| ARC-Challenge | 22.2 | 25.8 |
### 7.2 ํ•œ๊ตญ์–ด (KoBEST)
| ๊ณผ์ œ | ์ ์ˆ˜ |
|------|------|
| **HellaSwag** (acc_norm) | **44.6 ยฑ2.2** |
| **COPA** | **57.2 ยฑ2.0** |
| **SentiNeg** | **55.7 ยฑ2.5** |
| WiC | 48.8 ยฑ2.0 |
| BoolQ | 47.8 ยฑ2.0 |
### 7.3 ์–ธ์–ด ๋ชจ๋ธ๋ง
| ์ž…๋ ฅ | perplexity |
|------|-----------|
| ์ž์—ฐ์Šค๋Ÿฌ์šด ํ•œ๊ตญ์–ด ๋ฌธ์žฅ | **5.4** |
| ๊ฐ™์€ ๋ฌธ์žฅ์˜ ์–ด์ˆœ์„ ๋’ค์„ž์€ ๊ฒƒ | 41.3 |
| ๋ฌด์ž‘์œ„ ํ† ํฐ | 2233.4 |
ํ•œ๊ตญ์–ด ๋ฌธ๋ฒ•ยท์˜๋ฏธ ๊ตฌ์กฐ๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ํ•™์Šต๋˜์—ˆ์Œ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.
---
## 8. ์‚ฌ์šฉ๋ฒ•
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL = "FINAL-Bench/Aether-7B-5Attn"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
MODEL, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
)
ids = tok("์ธ๊ณต์ง€๋Šฅ์ด๋ž€", return_tensors="pt").to("cuda")
out = model.generate(**ids, max_new_tokens=128,
do_sample=True, temperature=0.8, top_p=0.9,
repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))
```
### ์‚ฌ์šฉ
> โš ๏ธ **batch_size=1 ๋กœ ์‚ฌ์šฉํ•˜์„ธ์š”.** ๋ณธ ๋ชจ๋ธ์˜ ์ผ๋ถ€ ์–ดํ…์…˜(NSA ๊ณ„์—ด)์€ padding mask ๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š์œผ๋ฏ€๋กœ, ์—ฌ๋Ÿฌ ์‹œํ€€์Šค๋ฅผ padding ํ•˜์—ฌ ๋ฐฐ์น˜๋กœ ๋„ฃ์œผ๋ฉด ๊ฒฐ๊ณผ๊ฐ€ ์กฐ์šฉํžˆ ์˜ค์—ผ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ•œ ๋ฒˆ์— ํ•˜๋‚˜์”ฉ(batch_size=1) ์ถ”๋ก ํ•˜์‹ญ์‹œ์˜ค.
์‹œ ์ฐธ๊ณ 
- **๋ฒ ์ด์Šค ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.** ์ง€์‹œ(instruction) ํŠœ๋‹์ด ๋˜์–ด ์žˆ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. instruct ๋ฒ„์ „์€ ์ค€๋น„ ์ค‘์ž…๋‹ˆ๋‹ค.
- **`do_sample=True` ๋ฅผ ์‚ฌ์šฉํ•˜์‹ญ์‹œ์˜ค.** ๋ฒ ์ด์Šค ๋ชจ๋ธ ํŠน์„ฑ์ƒ greedy ๋””์ฝ”๋”ฉ์€ ๋ฐ˜๋ณต์— ๋น ์ง‘๋‹ˆ๋‹ค.
- **`batch_size=1` ๋˜๋Š” ํŒจ๋”ฉ ์—†๋Š” ์ž…๋ ฅ์„ ์‚ฌ์šฉํ•˜์‹ญ์‹œ์˜ค.** ์ปค์Šคํ…€ ์–ดํ…์…˜ ๊ตฌํ˜„ ํŠน์„ฑ์ƒ ์ขŒํŒจ๋”ฉ ๋ฐฐ์น˜์—์„œ๋Š” ์ถœ๋ ฅ์ด ๋‹ฌ๋ผ์งˆ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
- **์„œ๋น™์€ HF `transformers`** ๋กœ ํ•˜์‹ญ์‹œ์˜ค. ์ปค์Šคํ…€ ์•„ํ‚คํ…์ฒ˜์ด๋ฏ€๋กœ vLLM์€ ์•„์ง ์ง€์›ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
- **์•ˆ์ „์„ฑ ์ •๋ ฌ์„ ํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค.** RLHFยท์•ˆ์ „์„ฑ ํŠœ๋‹์ด ์ ์šฉ๋˜์ง€ ์•Š์€ ์—ฐ๊ตฌ์šฉ ๋ฒ ์ด์Šค ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค. ์„œ๋น„์Šค์— ๊ทธ๋Œ€๋กœ ์‚ฌ์šฉํ•˜์ง€ ๋งˆ์‹ญ์‹œ์˜ค.
---
## 9. ๊ณต๊ฐœ ๋ฒ”์œ„
| ํ•ญ๋ชฉ | ์ƒํƒœ |
|------|------|
| ๊ฐ€์ค‘์น˜ (annealing ์™„๋ฃŒ ๋ฒ ์ด์Šค) | โœ… ๋ณธ ์ €์žฅ์†Œ |
| ์•„ํ‚คํ…์ฒ˜ ์†Œ์Šค ์ „์ฒด | โœ… `aether_pkg/` |
| ํ•™์Šต ๋ฐ์ดํ„ฐ ๋ ˆ์‹œํ”ผ (์†Œ์Šคยทconfigยทํ† ํฐ์ˆ˜ยท๋ฐฐํ•ฉ๋น„ยทํ† ํฌ๋‚˜์ด์ €ยทEOS) | โœ… ยง5 |
| ํ† ํฐํ™” ์Šคํฌ๋ฆฝํŠธ | โœ… `tokenize_one.py` |
| ํ•™์Šต ์ฝ”๋“œ | โœ… `launcher_v2b_multi.py`, `launcher_v2b_multi.py`, `run_s1.sh`, `run_s2.sh` |
| ํ•™์Šต ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ ์ „์ฒด | โœ… ยง6 |
| ํ•™์Šต ๋กœ๊ทธ ์ „๋ฌธ | โœ… `logs/` |
| ํ‰๊ฐ€ ์ฝ”๋“œ | โœ… `eval/lmeval_run.py` |
| ์ค‘๊ฐ„ ์ฒดํฌํฌ์ธํŠธ | โœ… step 110k / 115k / 162k |
| ๋…ผ๋ฌธ | ์ค€๋น„ ์ค‘ |
**๋ผ์ด์„ ์Šค: Apache-2.0** (๊ฐ€์ค‘์น˜ยท์ฝ”๋“œ). ํ•™์Šต ๋ฐ์ดํ„ฐ๋Š” ๊ฐ ์› ์ €์žฅ์†Œ์˜ ๋ผ์ด์„ ์Šค๋ฅผ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค (ยง5.1).
---
## 10. ์ธ์šฉ
```bibtex
@misc{aether7b5attn2026,
title = {Aether-7B-5Attn: A Heterogeneous-Attention Mixture-of-Experts Language Model},
author = {VIDRAFT},
year = {2026},
url = {https://huggingface.co/FINAL-Bench/Aether-7B-5Attn}
}
```
---
## 11. ๋ฌธ์˜
**์ฃผ์‹ํšŒ์‚ฌ ๋น„๋“œ๋ž˜ํ”„ํŠธ (VIDRAFT)** ยท arxivgpt@gmail.com