Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -9,6 +9,7 @@ tags:
   - from-scratch
   - korean
   - gpt
+pipeline_tag: text-generation
 model-index:
   - name: SOVYN-85M
     results:
@@ -23,88 +24,106 @@ model-index:
 
 # SOVYN-85M
 
-**85.4M 파라미터 한국어 추론 특화 GPT 모델**
+처음부터 학습한 85M 파라미터 한국어 추론 모델.
 
-완전히 처음부터(from scratch) 학습된 한국어 추론 AI입니다.
-수학, 코딩, 논리, 과학 등 다양한 추론 문제를 단계별로 풀이합니다.
+수학, 코드 트레이싱, 논리, 물리, 화학, 생물, 지구과학, 한국사, 미적분 등 119개 카테고리의 문제를 단계별로 풀이한다.
 
-## 모델 구조
+## 스펙
 
-| 항목 | 값 |
-|------|-----|
-| Architecture | GPT (Decoder-only Transformer) |
-| Parameters | 85.4M |
-| Layers | 12 |
-| Heads | 12 |
-| Embed Dim | 768 |
-| Context Length | 512 |
-| Vocab Size | 16,384 (BPE) |
-| Attention | Flash Attention (SDPA) |
+| | |
+|---|---|
+| 파라미터 | 85.4M |
+| 아키텍처 | GPT (Decoder-only) |
+| 레이어 | 12 |
+| 어텐션 헤드 | 12 |
+| 임베딩 차원 | 768 |
+| 컨텍스트 길이 | 512 |
+| 어휘 크기 | 16,384 (BPE) |
+| 어텐션 | Flash Attention (SDPA) |
+| 정밀도 | float16 |
 
-## 학습 데이터
+## 학습
 
-- **591,261개** 합성 추론 문제 (119 카테고리)
-- **27.97M 토큰** (BPE, vocab 16,384)
-- 카테고리: 수학, 대수, 미적분, 물리, 화학, 생물, 지구과학, 한국사, 코딩, 논리, 영어, 한국어, 함수 등
+- 데이터: 591,261개 합성 추론 문제 (119 카테고리), 27.97M 토큰
+- 옵티마이저: AdamW (lr=3e-4, weight_decay=0.1)
+- 스케줄: Cosine decay + warmup 500 steps
+- 배치: 16 x 4 grad_accum = effective 64
+- 스텝: 20,000
+- GPU: RTX 5080 16GB
+- 학습 시간: ~4시간
 
-## 학습 설정
+## 벤치마크
 
-- Optimizer: AdamW (lr=3e-4, weight_decay=0.1)
-- Schedule: Cosine decay with warmup (500 steps)
-- Batch: 16 × 4 grad_accum = effective 64
-- Steps: 20,000
-- Mixed Precision: AMP + GradScaler
-- Hardware: NVIDIA RTX 5080 (16GB)
-
-## 벤치마크 결과
+자체 벤치마크 52문제, 10개 카테고리.
 
 | 카테고리 | 정확도 |
 |---------|--------|
-| 산술_기본 | 100% |
-| 코드_트레이싱 | 100% |
-| 숫자_성질 | 100% |
+| 산술 | 100% |
+| 코드 트레이싱 | 100% |
+| 숫자 성질 | 100% |
 | 서술형 | 100% |
-| 연산_우선순위 | 88% |
-| 리스트_연산 | 83% |
-| 괄호_연산 | 80% |
+| 연산 우선순위 | 88% |
+| 리스트 연산 | 83% |
+| 괄호 연산 | 80% |
 | 방정식 | 80% |
 | 논리 | 80% |
 | 수열 | 33% |
-| **전체** | **86.5% (A등급)** |
+| **전체** | **86.5%** |
 
 ## 사용법
 
+```bash
+pip install torch safetensors tokenizers huggingface_hub
+```
+
 ```python
 import torch
+from safetensors.torch import load_file
 from tokenizers import Tokenizer
-
-# 모델 로드 (커스텀 아키텍처 필요)
-from train_125m import GPT125M, ModelConfig
-
-cfg = ModelConfig()
-model = GPT125M(cfg)
-state_dict = torch.load("pytorch_model.bin", map_location="cpu")
+from huggingface_hub import hf_hub_download
+
+# 다운로드
+model_path = hf_hub_download("SOVYN/SOVYN-85M", "model.safetensors")
+tok_path = hf_hub_download("SOVYN/SOVYN-85M", "tokenizer.json")
+code_path = hf_hub_download("SOVYN/SOVYN-85M", "model.py")
+
+# 아키텍처 로드
+import importlib.util
+spec = importlib.util.spec_from_file_location("model", code_path)
+mod = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(mod)
+
+# 모델 로드
+model = mod.SOVYN85M()
+state_dict = load_file(model_path)
+state_dict = {k: v.float() for k, v in state_dict.items()}
 model.load_state_dict(state_dict)
 model.eval()
 
-# 토크나이저
-tokenizer = Tokenizer.from_file("tokenizer.json")
+tokenizer = Tokenizer.from_file(tok_path)
 
 # 추론
 prompt = "문제: 3x + 7 = 22일 때, x의 값을 구하시오.\n풀이:\n"
-input_ids = tokenizer.encode(prompt).ids
-input_tensor = torch.tensor([input_ids])
+ids = torch.tensor([tokenizer.encode(prompt).ids])
+out = model.generate(ids, max_new_tokens=200, temperature=0.3)
+print(tokenizer.decode(out[0].tolist()))
+```
+
+## 프롬프트 형식
 
-with torch.no_grad():
-    output = model.generate(input_tensor, max_new_tokens=200)
-    result = tokenizer.decode(output[0].tolist())
-    print(result)
+```
+문제: {내용}
+풀이:
 ```
 
-## 라이선스
+"풀이:" 이후를 생성. 단계별 풀이 + "답: {정답}" 형태로 출력.
 
-Apache-2.0
+## 제한사항
 
-## 만든 이
+- 합성 데이터로만 학습. 자유 대화 불가.
+- 수열(등비/피보나치) 약함.
+- 컨텍스트 512 토큰 제한.
 
-SOVYN
+## 라이선스
+
+Apache-2.0

config.json

@@ -1,16 +1,17 @@
 {
-  "model_type": "sovyn-gpt",
   "architectures": [
-    "GPT125M"
+    "SOVYN85M"
   ],
+  "model_type": "sovyn-gpt",
   "vocab_size": 16384,
-  "context_length": 512,
-  "embed_dim": 768,
-  "num_heads": 12,
-  "num_layers": 12,
-  "dropout": 0.1,
-  "bias": false,
-  "parameters": "85.4M",
-  "training_steps": 10000,
-  "best_val_loss": 0.46251316606998444
+  "max_position_embeddings": 512,
+  "hidden_size": 768,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "intermediate_size": 3072,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0.1,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float16"
 }

model.py

@@ -0,0 +1,118 @@
+"""
+SOVYN-85M 모델 아키텍처
+https://huggingface.co/SOVYN/SOVYN-85M
+"""
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import math
+
+
+class ModelConfig:
+    vocab_size: int = 16384
+    context_length: int = 512
+    embed_dim: int = 768
+    num_heads: int = 12
+    num_layers: int = 12
+    dropout: float = 0.1
+    bias: bool = False
+
+
+class CausalSelfAttention(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.num_heads = cfg.num_heads
+        self.head_dim = cfg.embed_dim // cfg.num_heads
+        self.embed_dim = cfg.embed_dim
+        self.qkv = nn.Linear(cfg.embed_dim, 3 * cfg.embed_dim, bias=cfg.bias)
+        self.proj = nn.Linear(cfg.embed_dim, cfg.embed_dim, bias=cfg.bias)
+        self.resid_drop = nn.Dropout(cfg.dropout)
+        self.dropout_p = cfg.dropout
+
+    def forward(self, x):
+        B, T, C = x.shape
+        qkv = self.qkv(x).reshape(B, T, 3, self.num_heads, self.head_dim)
+        qkv = qkv.permute(2, 0, 3, 1, 4)
+        q, k, v = qkv.unbind(0)
+        out = F.scaled_dot_product_attention(
+            q, k, v, is_causal=True,
+            dropout_p=self.dropout_p if self.training else 0.0,
+        )
+        out = out.transpose(1, 2).reshape(B, T, C)
+        return self.resid_drop(self.proj(out))
+
+
+class FeedForward(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        hidden = 4 * cfg.embed_dim
+        self.fc1 = nn.Linear(cfg.embed_dim, hidden, bias=cfg.bias)
+        self.fc2 = nn.Linear(hidden, cfg.embed_dim, bias=cfg.bias)
+        self.drop = nn.Dropout(cfg.dropout)
+
+    def forward(self, x):
+        return self.drop(self.fc2(F.gelu(self.fc1(x))))
+
+
+class Block(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.ln1 = nn.LayerNorm(cfg.embed_dim)
+        self.attn = CausalSelfAttention(cfg)
+        self.ln2 = nn.LayerNorm(cfg.embed_dim)
+        self.ffn = FeedForward(cfg)
+
+    def forward(self, x):
+        x = x + self.attn(self.ln1(x))
+        x = x + self.ffn(self.ln2(x))
+        return x
+
+
+class SOVYN85M(nn.Module):
+    def __init__(self, cfg=None):
+        super().__init__()
+        if cfg is None:
+            cfg = ModelConfig()
+        self.cfg = cfg
+        self.tok_emb = nn.Embedding(cfg.vocab_size, cfg.embed_dim)
+        self.pos_emb = nn.Embedding(cfg.context_length, cfg.embed_dim)
+        self.drop = nn.Dropout(cfg.dropout)
+        self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg.num_layers)])
+        self.ln_f = nn.LayerNorm(cfg.embed_dim)
+        self.head = nn.Linear(cfg.embed_dim, cfg.vocab_size, bias=False)
+        self.head.weight = self.tok_emb.weight
+
+    @property
+    def num_params(self):
+        return sum(p.numel() for p in self.parameters()) - self.tok_emb.weight.numel()
+
+    def forward(self, idx, targets=None):
+        B, T = idx.shape
+        pos = torch.arange(T, device=idx.device)
+        x = self.drop(self.tok_emb(idx) + self.pos_emb(pos))
+        for block in self.blocks:
+            x = block(x)
+        x = self.ln_f(x)
+        logits = self.head(x)
+        loss = None
+        if targets is not None:
+            loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1),
+                                   ignore_index=0)
+        return logits, loss
+
+    @torch.no_grad()
+    def generate(self, idx, max_new_tokens=200, temperature=0.8, top_k=50):
+        self.eval()
+        for _ in range(max_new_tokens):
+            ctx = idx[:, -self.cfg.context_length:]
+            logits, _ = self(ctx)
+            logits = logits[:, -1, :] / max(temperature, 1e-8)
+            if top_k > 0:
+                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
+                logits[logits < v[:, -1:]] = float('-inf')
+            probs = F.softmax(logits, dim=-1)
+            nxt = torch.multinomial(probs, 1)
+            idx = torch.cat([idx, nxt], dim=1)
+            if nxt.item() == 2:  # EOS
+                break
+        return idx

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0dce9dbb79b774716a7b2bf3962834dcc62295f653fd45c468b5539c9a3b0ee
+size 221073472

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 512,
+  "bos_token": "<BOS>",
+  "eos_token": "<EOS>",
+  "pad_token": "<PAD>",
+  "unk_token": "<UNK>"
+}