ryanu/EEVE-10.8-BOOK-v0.1

파라미터	값
Task	Book (사회과학, 기술과학, 철학, 법학, 예술 등)
데이터 크기	5000개
모델	qlora
max_seq_length	1024
num_train_epochs	3
per_device_train_batch_size	8
gradient_accumulation_steps	32
evaluation_strategy	"steps"
eval_steps	2000
logging_steps	25
optim	"paged_adamw_8bit"
learning_rate	2e-4
lr_scheduler_type	"cosine"
warmup_steps	10
warmup_ratio	0.05
report_to	"tensorboard"
weight_decay	0.01
max_steps	-1

Summary

Book

모델 이름	Rouge-1	Rouge-2	Rouge-L
*ryanu/EEVE-10.8-BOOK-v0.1	0.2454	0.1158	0.2404
meta-llama/llama-3-70b-instruct	0.2269	0.0925	0.2186
meta-llama/llama-3-8b-instruct	0.2137	0.0883	0.2020
yanolja/EEVE-Korean-Instruct-2.8B-v1.0	0.2095	0.0866	0.1985
mistralai/mixtral-8x7b-instruct-v0-1	0.1735	0.0516	0.1668
ibm-mistralai/mixtral-8x7b-instruct-v01-q	0.1724	0.0534	0.1630

Paper

모델 이름	Rouge-1	Rouge-2	Rouge-L
*meta-llama/llama-3-8b-instruct	0.2044	0.0868	0.1895
ryanu/EEVE-10.8-BOOK-v0.1	0.2004	0.0860	0.1938
meta-llama/llama-3-70b-instruct	0.1935	0.0783	0.1836
yanolja/EEVE-Korean-Instruct-2.8B-v1.0	0.1934	0.0829	0.1832
mistralai/mixtral-8x7b-instruct-v0-1	0.1774	0.0601	0.1684
ibm-mistralai/mixtral-8x7b-instruct-v01-q	0.1702	0.0561	0.1605

RAG Q&A

모델 이름	Rouge-1	Rouge-2	Rouge-L
*meta-llama/llama-3-70b-instruct	0.4418	0.2986	0.4297
*meta-llama/llama-3-8b-instruct	0.4391	0.3100	0.4273
mistralai/mixtral-8x7b-instruct-v0-1	0.4022	0.2653	0.3916
ibm-mistralai/mixtral-8x7b-instruct-v01-q	0.3105	0.1763	0.2960
yanolja/EEVE-Korean-Instruct-10.8B-v1.0	0.3191	0.2069	0.3136
ryanu/EEVE-10.8-BOOK-v0.1	0.2185	0.1347	0.2139

prompt template

다음 문장을 3~5문장으로 반복되는 구문없이 텍스트에 제시된 주요 논거를 간략하게 요약해줘.

문장: {context}

요약: {summary}