werty1248 commited on
Commit
0f0dd23
ยท
verified ยท
1 Parent(s): 09ba802

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -82,7 +82,7 @@ Janet์€ ๋งค์ผ ๋†์‚ฐ๋ฌผ ์‹œ์žฅ์—์„œ 18๋‹ฌ๋Ÿฌ๋ฅผ ๋ฒŒ์–ด๋“ค์ž…๋‹ˆ๋‹ค. ์ด๋Š”
82
  2. [HyperCLOVAX-1.5B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B)์— CoT ํ”„๋กฌํ”„ํŠธ๋ฅผ ์ถ”๊ฐ€(user ์ž…๋ ฅ์—)ํ•œ ํ›„, **Pass@8** ์—์„œ ํ†ต๊ณผํ•œ ์งˆ๋ฌธ๋งŒ ์ถ”์ถœ (10,433/54,832๊ฐœ)
83
  3. [HyperCLOVAX-1.5B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B)(+CoT)๋กœ **32ํšŒ ์ถ”๊ฐ€ ์ถ”๋ก ** ํ›„ ์ •๋‹ต์„ ๋งžํžŒ ๋‹ต๋ณ€๋งŒ ์ถ”์ถœ + [Kanana-nano-2.1b](https://huggingface.co/kakaocorp/kanana-nano-2.1b-instruct)(+CoT)๋กœ **32ํšŒ ์ถ”๊ฐ€ ์ถ”๋ก ** ํ›„ ์ •๋‹ต์„ ๋งžํžŒ ๋‹ต๋ณ€๋งŒ ์ถ”์ถœ
84
  - HyperCLOVAX-1.5B CoT Prompt: ```"์œ ์ €๊ฐ€ ์ตœ์ข…์ ์œผ๋กœ ๊ตฌํ•˜๊ณ ์ž ํ•˜๋Š” ๊ฐ’์ด ๋ฌด์—‡์ธ์ง€ ๋‹ค์‹œ ํ•œ ๋ฒˆ ์ •์˜ํ•˜๊ณ , ๋ฌธ์ œ์—์„œ ์ œ์‹œ๋œ ์กฐ๊ฑด๋„ ๊น”๋”ํ•˜๊ฒŒ ์ •๋ฆฌํ•˜์—ฌ ์žฌ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๋‚˜์„œ ๋‹ต์„ ๊ตฌํ•˜๊ธฐ ์œ„ํ•ด ๊ตฌ์ฒด์ ์œผ๋กœ ๋…ผ๋ฆฌ ๋ฐ ์ˆ˜์‹์„ ์ „๊ฐœํ•˜๋ฉฐ ๋ฌธ์ œ๋ฅผ ํ‘ผ ํ›„, ์ตœ์ข… ๋‹ต๋ณ€์„ \\boxed{} ์•ˆ์— ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.\n\n"```
85
- - Kanana-2.1b CoT Prompt (๋งˆ์ง€๋ง‰์— "์ œ๋ชฉ ์—†์ด ๋‚ด์šฉ๋งŒ ์ถœ๋ ฅํ•ฉ๋‹ˆ๋‹ค." ์ถ”๊ฐ€): ```""์œ ์ €๊ฐ€ ์ตœ์ข…์ ์œผ๋กœ ๊ตฌํ•˜๊ณ ์ž ํ•˜๋Š” ๊ฐ’์ด ๋ฌด์—‡์ธ์ง€ ๋‹ค์‹œ ํ•œ ๋ฒˆ ์ •์˜ํ•˜๊ณ , ๋ฌธ์ œ์—์„œ ์ œ์‹œ๋œ ์กฐ๊ฑด๋„ ๊น”๋”ํ•˜๊ฒŒ ์ •๋ฆฌํ•˜์—ฌ ์žฌ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๋‚˜์„œ ๋‹ต์„ ๊ตฌํ•˜๊ธฐ ์œ„ํ•ด ๊ตฌ์ฒด์ ์œผ๋กœ ๋…ผ๋ฆฌ ๋ฐ ์ˆ˜์‹์„ ์ „๊ฐœํ•˜๋ฉฐ ๋ฌธ์ œ๋ฅผ ํ‘ผ ํ›„, ์ตœ์ข… ๋‹ต๋ณ€์„ \\boxed{} ์•ˆ์— ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ์ œ๋ชฉ ์—†์ด ๋‚ด์šฉ๋งŒ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.\n\n"```
86
 
87
  #### ์ •๋‹ต์„ ๋งžํžŒ ๋‹ต๋ณ€ ์ˆ˜ ๋ถ„ํฌ
88
 
@@ -94,7 +94,7 @@ Janet์€ ๋งค์ผ ๋†์‚ฐ๋ฌผ ์‹œ์žฅ์—์„œ 18๋‹ฌ๋Ÿฌ๋ฅผ ๋ฒŒ์–ด๋“ค์ž…๋‹ˆ๋‹ค. ์ด๋Š”
94
 
95
  - 10,433๊ฐœ ์งˆ๋ฌธ์— ๋Œ€ํ•œ 38,037๊ฐœ ๋‹ต๋ณ€ ๋ฐ์ดํ„ฐ ์…‹
96
  - Axolotl๋กœ ํ•™์Šต (2x A40์—์„œ 2์‹œ๊ฐ„ = 4 GPU Hours)
97
- - CoT ํ”„๋กฌํ”„ํŠธ๋ฅผ ์ถ”๊ฐ€ํ•˜๊ธด ํ–ˆ์œผ๋‚˜, ํ‰๊ท  ํ† ํฐ ์ˆ˜๋Š” ์งง์Œ(ํ‰๊ท  300ํ† ํฐ ๋ฏธ๋งŒ)
98
 
99
  <details><summary>axolotl config</summary>
100
 
@@ -187,6 +187,7 @@ deepspeed: ./deepspeed_configs/zero3_bf16.json
187
 
188
  #### Least Similarity Sampling
189
  - **ํ’€์ด๊ฐ€ ํ‹€๋ ธ๋Š”๋ฐ ๋‹ต๋งŒ ๋งž์ถ˜ ๊ฒฝ์šฐ**๊ฐ€ ์ƒ๊ธฐ๋ฉด, ๋‹ค๋ฅธ ์˜ฌ๋ฐ”๋ฅธ ํ’€์ด๋“ค๊ณผ ๋‚ฎ์€ ์œ ์‚ฌ๋„๋ฅผ ๊ฐ€์ ธ ๊ฑฐ์˜ ํ™•์ •์ ์œผ๋กœ ์ƒ˜ํ”Œ๋ง๋˜๋Š” ๋ฌธ์ œ ์žˆ์Œ. ๋ณ„๋„ ํ•„ํ„ฐ๋งํ•˜์ง€ ์•Š์Œ.
 
190
 
191
  ### ์ถœ๋ ฅ
192
  - **ํ‰๊ท  ์‘๋‹ต ๊ธธ์ด๊ฐ€ 753 token**(GSM8K ๊ธฐ์ค€) ์ •๋„๋กœ, reasoning ๋ชจ๋ธ๋ณด๋‹ค ์งง์€ ํŽธ (์•„์ง Reasoning ํ•™์Šต์„ ํ•˜์ง€ ์•Š์•˜๊ธฐ ๋•Œ๋ฌธ)
 
82
  2. [HyperCLOVAX-1.5B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B)์— CoT ํ”„๋กฌํ”„ํŠธ๋ฅผ ์ถ”๊ฐ€(user ์ž…๋ ฅ์—)ํ•œ ํ›„, **Pass@8** ์—์„œ ํ†ต๊ณผํ•œ ์งˆ๋ฌธ๋งŒ ์ถ”์ถœ (10,433/54,832๊ฐœ)
83
  3. [HyperCLOVAX-1.5B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B)(+CoT)๋กœ **32ํšŒ ์ถ”๊ฐ€ ์ถ”๋ก ** ํ›„ ์ •๋‹ต์„ ๋งžํžŒ ๋‹ต๋ณ€๋งŒ ์ถ”์ถœ + [Kanana-nano-2.1b](https://huggingface.co/kakaocorp/kanana-nano-2.1b-instruct)(+CoT)๋กœ **32ํšŒ ์ถ”๊ฐ€ ์ถ”๋ก ** ํ›„ ์ •๋‹ต์„ ๋งžํžŒ ๋‹ต๋ณ€๋งŒ ์ถ”์ถœ
84
  - HyperCLOVAX-1.5B CoT Prompt: ```"์œ ์ €๊ฐ€ ์ตœ์ข…์ ์œผ๋กœ ๊ตฌํ•˜๊ณ ์ž ํ•˜๋Š” ๊ฐ’์ด ๋ฌด์—‡์ธ์ง€ ๋‹ค์‹œ ํ•œ ๋ฒˆ ์ •์˜ํ•˜๊ณ , ๋ฌธ์ œ์—์„œ ์ œ์‹œ๋œ ์กฐ๊ฑด๋„ ๊น”๋”ํ•˜๊ฒŒ ์ •๋ฆฌํ•˜์—ฌ ์žฌ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๋‚˜์„œ ๋‹ต์„ ๊ตฌํ•˜๊ธฐ ์œ„ํ•ด ๊ตฌ์ฒด์ ์œผ๋กœ ๋…ผ๋ฆฌ ๋ฐ ์ˆ˜์‹์„ ์ „๊ฐœํ•˜๋ฉฐ ๋ฌธ์ œ๋ฅผ ํ‘ผ ํ›„, ์ตœ์ข… ๋‹ต๋ณ€์„ \\boxed{} ์•ˆ์— ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.\n\n"```
85
+ - Kanana-2.1b CoT Prompt (๋งˆ์ง€๋ง‰์— "์ œ๋ชฉ ์—†์ด ๋‚ด์šฉ๋งŒ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค." ์ถ”๊ฐ€): ```""์œ ์ €๊ฐ€ ์ตœ์ข…์ ์œผ๋กœ ๊ตฌํ•˜๊ณ ์ž ํ•˜๋Š” ๊ฐ’์ด ๋ฌด์—‡์ธ์ง€ ๋‹ค์‹œ ํ•œ ๋ฒˆ ์ •์˜ํ•˜๊ณ , ๋ฌธ์ œ์—์„œ ์ œ์‹œ๋œ ์กฐ๊ฑด๋„ ๊น”๋”ํ•˜๊ฒŒ ์ •๋ฆฌํ•˜์—ฌ ์žฌ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๋‚˜์„œ ๋‹ต์„ ๊ตฌํ•˜๊ธฐ ์œ„ํ•ด ๊ตฌ์ฒด์ ์œผ๋กœ ๋…ผ๋ฆฌ ๋ฐ ์ˆ˜์‹์„ ์ „๊ฐœํ•˜๋ฉฐ ๋ฌธ์ œ๋ฅผ ํ‘ผ ํ›„, ์ตœ์ข… ๋‹ต๋ณ€์„ \\boxed{} ์•ˆ์— ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค. ์ œ๋ชฉ ์—†์ด ๋‚ด์šฉ๋งŒ ์ž‘์„ฑํ•ฉ๋‹ˆ๋‹ค.\n\n"```
86
 
87
  #### ์ •๋‹ต์„ ๋งžํžŒ ๋‹ต๋ณ€ ์ˆ˜ ๋ถ„ํฌ
88
 
 
94
 
95
  - 10,433๊ฐœ ์งˆ๋ฌธ์— ๋Œ€ํ•œ 38,037๊ฐœ ๋‹ต๋ณ€ ๋ฐ์ดํ„ฐ ์…‹
96
  - Axolotl๋กœ ํ•™์Šต (2x A40์—์„œ 2์‹œ๊ฐ„ = 4 GPU Hours)
97
+ - CoT ํ”„๋กฌํ”„ํŠธ๋ฅผ ์ถ”๊ฐ€ํ•˜๊ธด ํ–ˆ์œผ๋‚˜, ํ‰๊ท  ๋‹ต๋ณ€ ํ† ํฐ ์ˆ˜๋Š” ์งง์Œ(ํ‰๊ท  300ํ† ํฐ ๋ฏธ๋งŒ)
98
 
99
  <details><summary>axolotl config</summary>
100
 
 
187
 
188
  #### Least Similarity Sampling
189
  - **ํ’€์ด๊ฐ€ ํ‹€๋ ธ๋Š”๋ฐ ๋‹ต๋งŒ ๋งž์ถ˜ ๊ฒฝ์šฐ**๊ฐ€ ์ƒ๊ธฐ๋ฉด, ๋‹ค๋ฅธ ์˜ฌ๋ฐ”๋ฅธ ํ’€์ด๋“ค๊ณผ ๋‚ฎ์€ ์œ ์‚ฌ๋„๋ฅผ ๊ฐ€์ ธ ๊ฑฐ์˜ ํ™•์ •์ ์œผ๋กœ ์ƒ˜ํ”Œ๋ง๋˜๋Š” ๋ฌธ์ œ ์žˆ์Œ. ๋ณ„๋„ ํ•„ํ„ฐ๋งํ•˜์ง€ ์•Š์Œ.
190
+ - ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ, **ํ’€์ด ์—†์ด ๋‹ต๋งŒ ์ถœ๋ ฅํ•œ ๊ฒฝ์šฐ**๊ฐ€ ๋ฐœ์ƒํ• ๋•Œ๋„, ํ’€์ด๊ฐ€ ์žˆ๋Š” ๋‹ค๋ฅธ ๋‹ต๋ณ€๊ณผ ์œ ์‚ฌ๋„๊ฐ€ ๋‚ฎ์•„ ์ ์–ด๋„ ํ•˜๋‚˜๋Š” ํ™•์ •์ ์œผ๋กœ ์ƒ˜ํ”Œ๋ง๋˜๋Š” ๋ฌธ์ œ๊ฐ€ ์žˆ์Œ. ๋ณ„๋„ ํ•„ํ„ฐ๋งํ•˜์ง€ ์•Š์Œ.
191
 
192
  ### ์ถœ๋ ฅ
193
  - **ํ‰๊ท  ์‘๋‹ต ๊ธธ์ด๊ฐ€ 753 token**(GSM8K ๊ธฐ์ค€) ์ •๋„๋กœ, reasoning ๋ชจ๋ธ๋ณด๋‹ค ์งง์€ ํŽธ (์•„์ง Reasoning ํ•™์Šต์„ ํ•˜์ง€ ์•Š์•˜๊ธฐ ๋•Œ๋ฌธ)