Hugging Face's logo

pathcosmos
/
frankenstallm

Text Generation
Transformers
Safetensors
GGUF
Korean
English
llama
3b
korean
from-scratch
orpo
instruction-tuned
preference-aligned
fp8
b200
Eval Results (legacy)
text-generation-inference
Model card Files
xet
Community
51

FRANKENSTALLM 3B

한국어 3B LLM을 처음부터 직접 만들었습니다 — 토크나이저 학습부터 사전학습, SFT, ORPO까지, 8× NVIDIA B200 GPU 위에서.

개발자 pathcosmos
파라미터 ~24억 (weight tying 적용, 3B급)
언어 한국어 (주), 영어 (부)
라이선스 Apache 2.0
학습 3단계: 사전학습 → SFT → ORPO
하드웨어 8× NVIDIA B200 (FP8), 총 ~86시간

빠른 시작

Transformers 

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "pathcosmos/frankenstallm"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

inputs = tokenizer(
    "한국의 전통 음식 중 김치에 대해 설명해주세요.",
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.2,  # 권장
        top_p=0.9,
        max_new_tokens=512,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Ollama (GGUF) 

# GGUF + Modelfile 다운로드
huggingface-cli download pathcosmos/frankenstallm \
  gguf/frankenstallm-3b-v2-Q4_K_M.gguf \
  gguf/Modelfile.3b-v2-Q4_K_M \
  --local-dir ./frankenstallm

# Modelfile 내 FROM 경로 수정 후 생성
ollama create frankenstallm -f ./frankenstallm/gguf/Modelfile.3b-v2-Q4_K_M

# 실행
ollama run frankenstallm

모델 특징

  • 처음부터 만든 한국어 토크나이저: SentencePiece Unigram, 64K 어휘, 한국어 문자 커버리지 99.95%
  • 3단계 학습 파이프라인: 사전학습 (57K 스텝, ~600억 토큰) → SFT (25.5K 스텝, 240만 샘플) → ORPO (10K 스텝, 63만 선호도 쌍)
  • B200 FP8 네이티브 학습: TransformerEngine MXFP8 — BF16 대비 이론적 2배 처리량
  • GGUF 배포 지원: Q4_K_M (757MB), Q8_0 (1.2GB), F16 (2.3GB) + Ollama Modelfile 제공

아키텍처

구성 요소 값
구조 Decoder-only Transformer (LLaMA 스타일)
Hidden size 3,072
레이어 수 28
어텐션 헤드 24
KV 헤드 8 (GQA 3:1)
FFN 차원 8,192 (SwiGLU)
어휘 크기 64,000
컨텍스트 길이 4,096 (학습 시 2,048)
위치 인코딩 RoPE (θ=500,000)
정규화 Pre-norm RMSNorm
어텐션 구현 FlashAttention-2
정밀도 FP8 (TransformerEngine MXFP8)
Weight tying 적용 (embedding ↔ lm_head)

학습 파이프라인

Phase 1: 사전학습

항목 값
스텝 수 57,000
최종 loss 1.466
학습 토큰 ~600억 (385억 고유 × ~1.5 에폭)
소요 시간 ~63시간
데이터 CC-100 KO, HPLT KO, C4 KO, 나무위키, 위키피디아 KO, Cosmopedia (EN)
배치 크기 5 × 8 GPU × 8 accum × 2,048 seq = ~65만 토큰/스텝

Phase 2: SFT (지도 미세조정)

항목 값
스텝 수 25,500 (77.3% 지점에서 조기 종료)
최적 val_loss 1.8851 (step 23,000)
소요 시간 ~15.5시간
데이터 24개 소스, 243만 9,397 샘플 (7.48 GB)
구성 SFT 70% + 사전학습 리플레이 30% (치명적 망각 방지)
지식 망각률 0.9% (19개 데이터셋 기준)

Phase 3: ORPO (선호도 최적화)

항목 값
스텝 수 9,997 (조기 수렴)
최적 eval_loss 1.625
선호도 정확도 76.02%
보상 마진 0.6100
소요 시간 ~7시간
데이터 한국어 HF 데이터셋 7종, ~63만 선호도 쌍
하이퍼파라미터 beta=0.25, lr=1.2e-5, eff_batch=128

총 학습 시간: 8× B200에서 약 86시간

벤치마크

학습 단계별 성능 변화 (Base → SFT → ORPO)

벤치마크 Base SFT ORPO 변화 (Base→ORPO)
KoBEST 평균 (0-shot) 43.7% 43.3% 52.8% +9.1pp
KoBEST COPA 49.3% 48.6% 63.9% +14.6pp
KoBEST HellaSwag-KO 21.6% 19.8% 38.0% +16.4pp
KoBEST SentiNeg 48.6% 49.1% 62.5% +13.9pp
KoBEST BoolQ 50.3% 50.1% 50.6% +0.3pp
PIQA 52.5% 52.6% 59.9% +7.3pp
ARC-Easy 25.6% 25.9% 36.0% +10.4pp
HAE-RAE 19.7% 19.9% 21.8% +2.1pp
HellaSwag EN 26.2% 26.1% 29.2% +3.0pp
Greedy 3-gram 반복률 61.0% 73.0% 30.9% -30.1pp
EOS 종료율 0% 60% 67% +67pp
PPL 망각률 — 0.9% 4.1% 15% 이내 ✅

3B급 모델 비교 (Ollama, 35개 테스트)

모델 파라미터 한국어 NLU 지식 지시 수행 추론 평균 점수
Qwen 2.5 3B 3B 100.0 20.8 55.6 62.5 63.4
Phi-4 Mini 3.8B 66.7 29.2 33.3 87.5 60.6
FRANKENSTALLM 3B 3B 100.0 75.0 66.7 50.0 46.7

FRANKENSTALLM은 한국어 NLU (Qwen과 동률), 한국어 지식 (75.0 vs 20.8/29.2), 지시 수행 (66.7 vs 55.6/33.3)에서 앞섭니다.

추론 속도 (Ollama, Q4_K_M)

모델 평균 TTFT TPS 비고
FRANKENSTALLM 3B 16.7ms 142.5 가장 빠름
Phi-4 Mini 3.8B 25.6ms 100.4
Qwen 2.5 3B 28.2ms 93.8

Perplexity 보존율 (ORPO 지식 유지)

데이터셋 Base PPL ORPO PPL 망각률
Korean C4 5.72 5.87 +2.7%
Korean Wiki 11.84 12.21 +3.2%
최대 망각률 — — 4.1% ✅

학습 데이터

사전학습 (~385억 토큰)

분류 소스 추정 토큰 수
한국어 웹 크롤 C4 KO, CC-100 KO, HPLT KO ~172억
한국어 백과사전 위키피디아 KO, 나무위키 (2개 버전) ~28억
영어 교육 Cosmopedia (Stories, Web, Stanford, WikiHow, OpenStax, Khan) ~57억
영어 수학·과학 AutoMathText, OpenWebMath, Proof-Pile-2 ~85억
코드 StarCoder (필터링) ~43억

SFT (240만 샘플, 24개 소스)

영역 비율 주요 데이터셋
추론/CoT 38% reasoning_r1_1.4m, magpie_reasoning
한국어 지시문 23% korean_instruction_mix, open_korean_instructions, kullm_v2
영어 일반 16% openhermes_2.5, ultrachat_200k
수학 12% NuminaMath-CoT, orca-math-ko
대화/코드/기타 11% smol-koreantalk, Evol-Instruct-Code-80k-ko

ORPO (~63만 선호도 쌍, 7개 소스)

데이터셋 용량 영역
nayohan/preference-collection-ko-full 4.9GB 일반 선호도
heegyu/orca-math-korean-preference-cleaned 1.6GB 수학 추론
kuotient/orca-math-korean-dpo-pairs 750MB 수학 DPO
maywell/ko_Ultrafeedback_binarized 394MB 피드백 정렬
tellang/yeji-preference-ko-v1 171MB 일반 선호도
jojo0217/korean_rlhf_dataset 137MB RLHF 쌍
lemon-mint/korean-realqa-reasoning-v01-preference 58MB QA 추론

GGUF & Ollama

제공 양자화 파일

파일 크기 설명
gguf/frankenstallm-3b-v2-Q4_K_M.gguf 757MB 권장 — 크기 대비 최적 품질
gguf/frankenstallm-3b-v2-Q8_0.gguf 1.2GB 높은 품질
gguf/frankenstallm-3b-v2-f16.gguf 2.3GB 전체 정밀도
model.safetensors 4.76GB Transformers 네이티브 (ORPO best, byte-fallback 수정 완료)

권장 샘플링 파라미터

파라미터 값 비고
temperature 0.7 한국어 생성 품질 최적
repeat_penalty 1.2 필수 — 미적용 시 greedy 반복률 30.9%
top_p 0.9 Nucleus 샘플링
top_k 50 Top-k 후보 수
max_tokens 512 최대 생성 길이
num_ctx 4096 컨텍스트 윈도우 (초과 금지)

⚠️ 반드시 repeat_penalty >= 1.2를 사용하세요. 적용하면 반복률이 0% 로 떨어집니다. 미적용 시 greedy 디코딩에서 ~31% 3-gram 반복이 발생합니다.

제한 사항

  • 영어 성능 제한: MMLU-EN ~23%, HellaSwag-EN ~29% — 한국어 특화 모델입니다
  • 코드 생성: 거의 불가능 (학습 데이터에 코드 비중이 낮음)
  • Greedy 반복: repeat_penalty 미사용 시 30.9% 3-gram 반복 — 반드시 repeat_penalty >= 1.2 사용
  • 안전성: 안전 정렬(safety alignment) 데이터가 학습에 포함되지 않았으므로 적절한 가드레일과 함께 사용하세요
  • 규모 차이: 수조 토큰으로 학습된 상용 3B 모델 대비 ~600억 토큰으로 학습 — 전반적 벤치마크 점수는 낮을 수 있습니다

하드웨어 및 학습 환경

구성 요소 사양
GPU 8× NVIDIA B200 (183GB HBM3e × 8, 총 ~1.47TB)
FP8 연산 2,250 TFLOPS/GPU (총 18,000 TFLOPS)
인터커넥트 NVLink 5.0, NVSwitch all-to-all mesh
CPU 2× AMD EPYC 9365 (72코어, Zen 5)
RAM 2.21 TB DDR5
PyTorch 2.10.0a0+b4e4ee81d3.nv25.12 (NVIDIA 커스텀)
TransformerEngine 2.10.0
FlashAttention 2.7.4
NCCL 2.28.9
CUDA 13.1
총 학습 시간 ~86시간 (사전학습 63h + SFT 15.5h + ORPO 7h)

인용 

@misc{frankenstallm2026,
  title={FRANKENSTALLM: A Korean 3B LLM Built From Scratch on B200 GPUs},
  author={pathcosmos},
  year={2026},
  url={https://huggingface.co/pathcosmos/frankenstallm},
  note={3-phase training (Pretrain, SFT, ORPO) with FP8 on 8x NVIDIA B200}
}

링크 및 연락처

🇺🇸 English version below

FRANKENSTALLM 3B

A Korean 3B LLM built entirely from scratch — tokenizer, pretraining, SFT, and ORPO — on 8× NVIDIA B200 GPUs.

Developer pathcosmos
Parameters ~2.4B (3B-class with weight tying)
Languages Korean (primary), English (secondary)
License Apache 2.0
Training 3-phase: Pretrain → SFT → ORPO
Hardware 8× NVIDIA B200 (FP8), ~86 hours total

Quick Start

Transformers 

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "pathcosmos/frankenstallm"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

inputs = tokenizer(
    "한국의 전통 음식 중 김치에 대해 설명해주세요.",
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.2,  # recommended
        top_p=0.9,
        max_new_tokens=512,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Ollama (GGUF) 

# Download GGUF + Modelfile
huggingface-cli download pathcosmos/frankenstallm \
  gguf/frankenstallm-3b-v2-Q4_K_M.gguf \
  gguf/Modelfile.3b-v2-Q4_K_M \
  --local-dir ./frankenstallm

# Fix FROM path in Modelfile, then create
ollama create frankenstallm -f ./frankenstallm/gguf/Modelfile.3b-v2-Q4_K_M

# Run
ollama run frankenstallm

Model Highlights

  • From-scratch Korean tokenizer: SentencePiece Unigram, 64K vocab, 99.95% Korean character coverage
  • 3-phase training pipeline: Pretrain (57K steps, ~60B tokens) → SFT (25.5K steps, 2.4M samples) → ORPO (10K steps, 630K preference pairs)
  • B200 FP8 native training: TransformerEngine MXFP8 on NVIDIA B200 — 2× theoretical throughput vs BF16
  • GGUF deployment ready: Q4_K_M (757MB), Q8_0 (1.2GB), F16 (2.3GB) with optimized Ollama Modelfiles

Architecture

Component Value
Type Decoder-only Transformer (LLaMA-style)
Hidden size 3,072
Layers 28
Attention heads 24
KV heads 8 (GQA 3:1)
FFN dim 8,192 (SwiGLU)
Vocab size 64,000
Context length 4,096 (trained at 2,048)
Position encoding RoPE (θ=500,000)
Normalization Pre-norm RMSNorm
Attention impl FlashAttention-2
Precision FP8 (MXFP8 via TransformerEngine)
Weight tying Yes (embedding ↔ lm_head)

Training Pipeline

Phase 1: Pretraining

Detail Value
Steps 57,000
Final loss 1.466
Tokens seen ~60B (38.5B unique × ~1.5 epochs)
Duration ~63 hours
Data CC-100 KO, HPLT KO, C4 KO, NamuWiki, Wikipedia KO, Cosmopedia (EN)
Batch size 5 × 8 GPU × 8 accum × 2,048 seq = ~655K tok/step

Phase 2: Supervised Fine-Tuning (SFT)

Detail Value
Steps 25,500 (early stop at 77.3%)
Best val_loss 1.8851 (step 23,000)
Duration ~15.5 hours
Data 2,439,397 samples from 24 sources (7.48 GB)
Mix 70% SFT + 30% pretrain replay (catastrophic forgetting prevention)
Knowledge forgetting 0.9% (19 datasets)

Phase 3: ORPO (Odds Ratio Preference Optimization)

Detail Value
Steps 9,997 (early convergence)
Best eval_loss 1.625
Preference accuracy 76.02%
Reward margin 0.6100
Duration ~7 hours
Data ~630K preference pairs from 7 Korean HF datasets
Hyperparams beta=0.25, lr=1.2e-5, eff_batch=128

Total training time: ~86 hours on 8× B200

Benchmarks

Training Phase Progression (Base → SFT → ORPO)

Benchmark Base SFT ORPO Δ (Base→ORPO)
KoBEST Avg (0-shot) 43.7% 43.3% 52.8% +9.1pp
KoBEST COPA 49.3% 48.6% 63.9% +14.6pp
KoBEST HellaSwag-KO 21.6% 19.8% 38.0% +16.4pp
KoBEST SentiNeg 48.6% 49.1% 62.5% +13.9pp
KoBEST BoolQ 50.3% 50.1% 50.6% +0.3pp
PIQA 52.5% 52.6% 59.9% +7.3pp
ARC-Easy 25.6% 25.9% 36.0% +10.4pp
HAE-RAE 19.7% 19.9% 21.8% +2.1pp
HellaSwag EN 26.2% 26.1% 29.2% +3.0pp
Greedy 3-gram repetition 61.0% 73.0% 30.9% -30.1pp
EOS termination rate 0% 60% 67% +67pp
PPL forgetting — 0.9% 4.1% within 15% ✅

3B-class Model Comparison (Ollama, 35 tests)

Model Params Korean NLU Knowledge Instruction Reasoning Avg Score
Qwen 2.5 3B 3B 100.0 20.8 55.6 62.5 63.4
Phi-4 Mini 3.8B 66.7 29.2 33.3 87.5 60.6
FRANKENSTALLM 3B 3B 100.0 75.0 66.7 50.0 46.7

FRANKENSTALLM leads in Korean NLU (tied with Qwen), Korean Knowledge (75 vs 20.8/29.2), and Instruction Following (66.7 vs 55.6/33.3).

Inference Speed (Ollama, Q4_K_M)

Model Avg TTFT TPS Note
FRANKENSTALLM 3B 16.7ms 142.5 Fastest
Phi-4 Mini 3.8B 25.6ms 100.4
Qwen 2.5 3B 28.2ms 93.8

Perplexity Preservation (ORPO Knowledge Retention)

Dataset Base PPL ORPO PPL Forgetting
Korean C4 5.72 5.87 +2.7%
Korean Wiki 11.84 12.21 +3.2%
Max forgetting — — 4.1% ✅

Training Data

Pretraining (~38.5B tokens)

Category Sources Est. Tokens
Korean Web Crawl C4 KO, CC-100 KO, HPLT KO ~17.2B
Korean Encyclopedia Wikipedia KO, NamuWiki (2 versions) ~2.8B
English Educational Cosmopedia (Stories, Web, Stanford, WikiHow, OpenStax, Khan) ~5.7B
English Math/Science AutoMathText, OpenWebMath, Proof-Pile-2 ~8.5B
Code StarCoder (filtered) ~4.3B

SFT (2.4M samples, 24 sources)

Domain Share Key Datasets
Reasoning/CoT 38% reasoning_r1_1.4m, magpie_reasoning
Korean Instructions 23% korean_instruction_mix, open_korean_instructions, kullm_v2
English General 16% openhermes_2.5, ultrachat_200k
Math 12% NuminaMath-CoT, orca-math-ko
Dialog/Code/Other 11% smol-koreantalk, Evol-Instruct-Code-80k-ko

ORPO (~630K preference pairs, 7 sources)

Dataset Size Domain
nayohan/preference-collection-ko-full 4.9GB General preference
heegyu/orca-math-korean-preference-cleaned 1.6GB Math reasoning
kuotient/orca-math-korean-dpo-pairs 750MB Math DPO
maywell/ko_Ultrafeedback_binarized 394MB Feedback alignment
tellang/yeji-preference-ko-v1 171MB General preference
jojo0217/korean_rlhf_dataset 137MB RLHF pairs
lemon-mint/korean-realqa-reasoning-v01-preference 58MB QA reasoning

GGUF & Ollama

Available Quantizations

File Size Description
gguf/frankenstallm-3b-v2-Q4_K_M.gguf 757MB Recommended — best size/quality balance
gguf/frankenstallm-3b-v2-Q8_0.gguf 1.2GB Higher quality
gguf/frankenstallm-3b-v2-f16.gguf 2.3GB Full precision
model.safetensors 4.76GB Transformers native (ORPO best, byte-fallback fixed)

Recommended Sampling Parameters

Parameter Value Notes
temperature 0.7 Optimal for Korean generation quality
repeat_penalty 1.2 Required — without it, greedy repetition is 30.9%
top_p 0.9 Nucleus sampling
top_k 50 Top-k candidates
max_tokens 512 Max generation length
num_ctx 4096 Context window (do not exceed)

⚠️ Always use repeat_penalty >= 1.2. With it, repetition drops to 0%. Without it, greedy decoding produces ~31% 3-gram repetition.

Limitations

  • English performance is limited: MMLU-EN ~23%, HellaSwag-EN ~29% — this is a Korean-focused model
  • Code generation: Near zero capability (limited code in training data)
  • Greedy repetition: 30.9% 3-gram repetition without repeat_penalty — always use sampling with repeat_penalty >= 1.2
  • Safety: Safety alignment data was not included in training; use with appropriate guardrails
  • Scale gap: Compared to commercial 3B models trained on trillions of tokens, this model was trained on ~60B tokens — expect lower overall benchmark scores

Hardware & Training Environment

Component Specification
GPU 8× NVIDIA B200 (183GB HBM3e each, ~1.47TB total)
FP8 Compute 2,250 TFLOPS/GPU (18,000 TFLOPS total)
Interconnect NVLink 5.0, NVSwitch all-to-all mesh
CPU 2× AMD EPYC 9365 (72 cores, Zen 5)
RAM 2.21 TB DDR5
PyTorch 2.10.0a0+b4e4ee81d3.nv25.12 (NVIDIA custom)
TransformerEngine 2.10.0
FlashAttention 2.7.4
NCCL 2.28.9
CUDA 13.1
Total training ~86 hours (Pretrain 63h + SFT 15.5h + ORPO 7h)

Citation 

@misc{frankenstallm2026,
  title={FRANKENSTALLM: A Korean 3B LLM Built From Scratch on B200 GPUs},
  author={pathcosmos},
  year={2026},
  url={https://huggingface.co/pathcosmos/frankenstallm},
  note={3-phase training (Pretrain, SFT, ORPO) with FP8 on 8x NVIDIA B200}
}

Links & Contact

Downloads last month
670
GGUF
Model size
3B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

4-bit

8-bit

16-bit

Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train pathcosmos/frankenstallm

Evaluation results