Upload folder using huggingface_hub

README.md

@@ -0,0 +1,162 @@
+---
+language:
+- ko
+license: apache-2.0
+library_name: transformers
+tags:
+- korean
+- causal-lm
+- chat
+- conversational
+- knowledge-distillation
+- small-language-model
+pipeline_tag: text-generation
+base_model: drlee1/hanforge-base
+---
+
+# HanForge 47M SFT — Korean Conversational Model
+
+A Korean chat model fine-tuned from [`drlee1/hanforge-base`](https://huggingface.co/drlee1/hanforge-base) with **knowledge distillation** on **24,693 Korean question-answer pairs** spanning five everyday domains.
+
+The model produces longer, more naturally phrased Korean responses than a templated baseline, but comes with reduced reliability under greedy decoding — **sampled decoding is recommended**.
+
+## Highlights
+
+- **Longer, more natural Korean responses** — averaging 130 characters (2–3 sentences)
+- **Five everyday domains**: greetings & conversation, food & cooking, Korean culture & geography, health & habits, emotional support
+- **Pure Korean output** — 100% Hangul ratio, zero foreign-script leakage
+- **Compact** — 47M parameters
+
+## Intended Use
+
+Suitable for:
+
+- **Korean chat applications** within everyday-conversation domains, where natural-sounding replies matter
+- **Resource-constrained deployments** needing a small Korean model
+- **Research** into small-LM knowledge distillation and instruction tuning
+
+Not suitable for:
+
+- Factual question answering requiring high accuracy (the synthetic data is not fact-checked)
+- Multi-step reasoning, coding, or technical tasks
+- Open-domain conversation outside the five training domains
+- Any safety-critical application
+
+## How to Use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "drlee1/hanforge-47M-SFT"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()
+
+USER, ASSISTANT = "<|user|>", "<|assistant|>"
+
+def chat(prompt: str, max_new_tokens: int = 200, seed: int = 42) -> str:
+    torch.manual_seed(seed)
+    text = f"{USER}\n{prompt}\n{ASSISTANT}\n"
+    inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)
+
+    # Add BOS manually
+    bos = inputs["input_ids"].new_full((1, 1), tokenizer.bos_token_id)
+    inputs["input_ids"] = torch.cat([bos, inputs["input_ids"]], dim=1)
+    inputs["attention_mask"] = torch.cat(
+        [inputs["attention_mask"].new_ones((1, 1)), inputs["attention_mask"]], dim=1
+    )
+
+    out = model.generate(
+        **inputs,
+        max_new_tokens=max_new_tokens,
+        do_sample=True,           # Sampled decoding is recommended
+        temperature=0.8,
+        top_p=0.9,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+    return tokenizer.decode(out[0, inputs["input_ids"].size(1):], skip_special_tokens=True).strip()
+
+print(chat("한국에서 가 볼 만한 여행지를 추천해 주세요."))
+```
+
+### Decoding tips
+
+- **Use sampling, not greedy.** Greedy decoding is prone to repetition with this model. Recommended settings: `temperature=0.8`, `top_p=0.9`.
+- **Try multiple seeds.** Some prompts produce a noticeably better answer on the second or third sampled generation.
+- **Cap output length.** 150–200 new tokens is usually enough; longer generations rarely improve quality.
+
+## Training Data
+
+Fine-tuned on **24,693 Korean question-answer pairs** prepared through a **knowledge-distillation** approach. The dataset spans 200 (domain, topic) pairs across five everyday domains, with each pair contributing roughly 100 diverse user-style questions paired with concise polite Korean answers.
+
+The five training domains are:
+
+| Domain | Topics covered |
+|---|---|
+| Daily greetings & conversation | greetings, thanks, apologies, introductions, mood, comfort, requests |
+| Food & cooking basics | Korean dishes, ingredients, simple recipes, recommendations |
+| Korean culture & geography | cities, mountains, traditional clothing, holidays, traditions |
+| Health & lifestyle habits | exercise, sleep, nutrition, stress, daily routines |
+| Emotions & empathy | sadness, loneliness, anxiety, joy, gratitude, comfort |
+
+After filtering for polite-ending and language-purity constraints (about 8.5% drop rate), the final training set carries 100% Hangul ratio, a consistent polite voice, and an average response length of ~134 characters.
+
+## Training Procedure
+
+Fine-tuned on top of [`drlee1/hanforge-base`](https://huggingface.co/drlee1/hanforge-base) using full-parameter SFT with response-only loss masking.
+
+| | |
+|---|---|
+| **Training samples** | 24,693 |
+| **Epochs** | 5 |
+| **Effective batch size** | 16 |
+| **Learning rate** | 5e-5 (cosine, 3% warmup) |
+| **Sequence length** | 384 |
+| **Precision** | bf16 mixed |
+| **Final training loss** | 10.4 |
+| **Validation perplexity** | ~25 |
+| **Wall-clock time** | ~19 minutes (Mac MPS) |
+
+## Evaluation
+
+Evaluated on 20 prompts (14 in-distribution, 6 out-of-distribution) under both greedy and sampled decoding.
+
+| Metric (sampled, t=0.8) | Result |
+|---|---|
+| Korean character ratio | 100% |
+| Foreign-script leakage | 0% |
+| End-of-sequence within 128 tokens | 90% |
+| Average response length | ~120 chars |
+
+| Metric (greedy) | Result |
+|---|---|
+| Korean character ratio | 100% |
+| Foreign-script leakage | 0% |
+| End-of-sequence within 128 tokens | 55% |
+| Maximum repeated-token run | up to ~200 (collapse risk) |
+
+The model is reliable on in-distribution Korean conversation but **not on out-of-distribution topics**. For abstract or domain-specific questions, responses are often well-formed Korean but semantically off.
+
+## Limitations and Bias
+
+- **Distilled-data origin**: Training answers were prepared via knowledge distillation. Facts, recommendations, and explanations may be incorrect, stale, or biased — do not rely on the model for accurate information.
+- **Domain restriction**: The five training domains define the model's reliable scope. Out-of-domain prompts produce responses that may look fluent but are often off-topic.
+- **Greedy decoding instability**: Small-scale models trained on longer responses tend to fall into repetition under greedy decoding. This model is no exception — always use sampling.
+- **No alignment / safety tuning**: Not RLHF'd, no harmful-content filtering. Inputs designed to elicit unsafe content may produce unsafe Korean text.
+- **Distillation bias**: Any biases present in the distillation source are inherited by the model.
+
+## License
+
+Released under the **Apache License 2.0**.
+
+## Citation
+
+```bibtex
+@misc{hanforge_47m_sft_2026,
+  author = {DongRyeol Lee},
+  title  = {HanForge 47M SFT: A Korean Conversational Model Trained via Knowledge Distillation},
+  year   = {2026},
+  note   = {Fine-tuned from drlee1/hanforge-base on 24.7k Korean Q\&A pairs across five everyday domains}
+}
+```

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "HanForgeForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1408,
+  "max_position_embeddings": 4096,
+  "model_type": "hanforge",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 2,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 50000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.5.1",
+  "unk_token_id": 3,
+  "use_cache": false,
+  "vocab_size": 24000,
+  "auto_map": {
+    "AutoConfig": "configuration_hanforge.HanForgeConfig",
+    "AutoModelForCausalLM": "modeling_hanforge.HanForgeForCausalLM"
+  },
+  "torch_dtype": "float32"
+}

configuration_hanforge.py

@@ -0,0 +1,87 @@
+from __future__ import annotations
+
+from transformers import PretrainedConfig
+
+
+class HanForgeConfig(PretrainedConfig):
+    model_type = "hanforge"
+
+    # <<< disabled (refactor 20260423, §4.1 hybrid local/global attention 미사용)
+    # 보존된 설계 자산: sliding_window / global_layer_interval / is_global_layer.
+    # 본 refactor에서는 full causal attention만 사용한다.
+    # sliding_window: int = 256
+    # global_layer_interval: int = 4
+    # def is_global_layer(self, layer_idx: int) -> bool:
+    #     return layer_idx % self.global_layer_interval == 0
+    # >>> end disabled
+
+    # <<< disabled (refactor 20260423, §4.2 YaRN 미사용)
+    # rope_scaling / original_max_position_embeddings 는 YaRN 확장 전제 필드였다.
+    # from-scratch 4k context 학습에서는 단순 RoPE 로 충분하다.
+    # original_max_position_embeddings: int = 4096
+    # rope_scaling: dict | None = None
+    # >>> end disabled
+
+    def __init__(
+        self,
+        vocab_size: int = 32000,
+        hidden_size: int = 384,
+        intermediate_size: int = 1024,
+        num_hidden_layers: int = 8,
+        num_attention_heads: int = 6,
+        num_key_value_heads: int = 2,
+        max_position_embeddings: int = 4096,
+        rope_theta: float = 50_000.0,
+        rms_norm_eps: float = 1e-6,
+        hidden_dropout_prob: float = 0.0,
+        attention_dropout: float = 0.0,
+        initializer_range: float = 0.02,
+        pad_token_id: int = 0,
+        bos_token_id: int = 1,
+        eos_token_id: int = 2,
+        unk_token_id: int = 3,
+        use_cache: bool = False,
+        **kwargs,
+    ):
+        # Back-compat: 과거 스크립트/체크포인트가 비활성화된 필드를 넘기더라도 무시한다.
+        kwargs.pop("sliding_window", None)
+        kwargs.pop("global_layer_interval", None)
+        kwargs.pop("original_max_position_embeddings", None)
+        kwargs.pop("rope_scaling", None)
+
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        self.hidden_dropout_prob = hidden_dropout_prob
+        self.attention_dropout = attention_dropout
+        self.initializer_range = initializer_range
+        self.use_cache = use_cache
+        tie_word_embeddings = kwargs.pop("tie_word_embeddings", True)
+
+        if hidden_size % num_attention_heads != 0:
+            raise ValueError("hidden_size must be divisible by num_attention_heads")
+        if num_attention_heads % num_key_value_heads != 0:
+            raise ValueError("num_attention_heads must be divisible by num_key_value_heads")
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            unk_token_id=unk_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+    @property
+    def head_dim(self) -> int:
+        return self.hidden_size // self.num_attention_heads
+
+    @property
+    def num_key_value_groups(self) -> int:
+        return self.num_attention_heads // self.num_key_value_heads

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1a1332ef89814c58a2fbfa3a76dff9667f8844bd118f0f8b27c34a28dbcf5e2
+size 188524624

modeling_hanforge.py

@@ -0,0 +1,338 @@
+from __future__ import annotations
+
+import math
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+
+try:
+    from configuration_hanforge import HanForgeConfig
+except ImportError:
+    from .configuration_hanforge import HanForgeConfig
+
+
+def rotate_half(x: torch.Tensor) -> torch.Tensor:
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor):
+    cos = cos.unsqueeze(1)
+    sin = sin.unsqueeze(1)
+    q = (q * cos) + (rotate_half(q) * sin)
+    k = (k * cos) + (rotate_half(k) * sin)
+    return q, k
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    if n_rep == 1:
+        return hidden_states
+    batch, num_key_value_heads, seq_len, head_dim = hidden_states.shape
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, seq_len, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, seq_len, head_dim)
+
+
+# DISABLED (refactor 20260423, §4.2): YaRN 본문 비활성화. from-scratch 4k context에서는 불필요.
+# 후일 context 확장 시 참조할 수 있도록 시그니처는 남기고 본문만 주석 처리한다.
+def _compute_yarn_parameters(config: HanForgeConfig, device=None):
+    raise NotImplementedError(
+        "YaRN is disabled in this refactor (see research/refactor_plan_20260423.md §4.2)."
+    )
+    # <<< disabled (refactor 20260423, §4.2)
+    # rope_params = dict(config.rope_scaling or {})
+    # dim = config.head_dim
+    # base = config.rope_theta
+    # if not rope_params or rope_params.get("rope_type", "default") == "default":
+    #     inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=device, dtype=torch.float32) / dim))
+    #     return inv_freq, 1.0
+    #
+    # factor = float(rope_params["factor"])
+    # beta_fast = float(rope_params.get("beta_fast", 32.0))
+    # beta_slow = float(rope_params.get("beta_slow", 1.0))
+    # mscale = rope_params.get("mscale")
+    # mscale_all_dim = rope_params.get("mscale_all_dim")
+    # original_max = int(rope_params["original_max_position_embeddings"])
+    #
+    # def get_mscale(scale, scale_factor=1.0):
+    #     if scale <= 1:
+    #         return 1.0
+    #     return 0.1 * scale_factor * math.log(scale) + 1.0
+    #
+    # if mscale is not None and mscale_all_dim is not None:
+    #     attention_factor = float(get_mscale(factor, mscale) / get_mscale(factor, mscale_all_dim))
+    # else:
+    #     attention_factor = float(get_mscale(factor))
+    #
+    # def find_correction_dim(num_rotations, local_dim, local_base, max_position_embeddings):
+    #     return (local_dim * math.log(max_position_embeddings / (num_rotations * 2 * math.pi))) / (
+    #         2 * math.log(local_base)
+    #     )
+    #
+    # def find_correction_range(low_rot, high_rot, local_dim, local_base, max_position_embeddings):
+    #     low = math.floor(find_correction_dim(low_rot, local_dim, local_base, max_position_embeddings))
+    #     high = math.ceil(find_correction_dim(high_rot, local_dim, local_base, max_position_embeddings))
+    #     return max(low, 0), min(high, local_dim - 1)
+    #
+    # def linear_ramp_factor(min_idx, max_idx, local_dim):
+    #     if min_idx == max_idx:
+    #         max_idx += 0.001
+    #     linear_func = (torch.arange(local_dim, dtype=torch.float32, device=device) - min_idx) / (max_idx - min_idx)
+    #     return torch.clamp(linear_func, 0, 1)
+    #
+    # pos_freqs = base ** (torch.arange(0, dim, 2, device=device, dtype=torch.float32) / dim)
+    # inv_freq_extrapolation = 1.0 / pos_freqs
+    # inv_freq_interpolation = 1.0 / (factor * pos_freqs)
+    # low, high = find_correction_range(beta_fast, beta_slow, dim, base, original_max)
+    # ramp = 1.0 - linear_ramp_factor(low, high, dim // 2)
+    # inv_freq = (inv_freq_interpolation * (1.0 - ramp)) + (inv_freq_extrapolation * ramp)
+    # return inv_freq, attention_factor
+    # >>> end disabled
+
+
+def _compute_rope_parameters(config: HanForgeConfig, device=None):
+    dim = config.head_dim
+    base = config.rope_theta
+    inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=device, dtype=torch.float32) / dim))
+    return inv_freq
+
+
+class HanForgeRMSNorm(nn.Module):
+    def __init__(self, hidden_size: int, eps: float = 1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.eps = eps
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(dim=-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.eps)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+class HanForgeRotaryEmbedding(nn.Module):
+    def __init__(self, config: HanForgeConfig):
+        super().__init__()
+        inv_freq = _compute_rope_parameters(config)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+    def forward(self, x: torch.Tensor, position_ids: torch.Tensor):
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
+        position_ids_expanded = position_ids[:, None, :].float()
+        freqs = (inv_freq_expanded @ position_ids_expanded).transpose(1, 2)
+        emb = torch.cat((freqs, freqs), dim=-1)
+        cos = emb.cos()
+        sin = emb.sin()
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+class HanForgeAttention(nn.Module):
+    def __init__(self, config: HanForgeConfig, layer_idx: int):
+        super().__init__()
+        self.layer_idx = layer_idx
+        self.num_heads = config.num_attention_heads
+        self.num_key_value_heads = config.num_key_value_heads
+        self.num_key_value_groups = config.num_key_value_groups
+        self.head_dim = config.head_dim
+        # DISABLED (refactor 20260423, §4.1): hybrid local/global attention 비활성화
+        # self.is_global = config.is_global_layer(layer_idx)
+        # self.sliding_window = config.sliding_window
+        self.q_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
+        kv_hidden = config.num_key_value_heads * self.head_dim
+        self.k_proj = nn.Linear(config.hidden_size, kv_hidden, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, kv_hidden, bias=False)
+        self.o_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
+        self.dropout = nn.Dropout(config.attention_dropout)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cos: torch.Tensor,
+        sin: torch.Tensor,
+        attention_mask: Optional[torch.Tensor],
+    ) -> torch.Tensor:
+        batch_size, seq_len, hidden_size = hidden_states.shape
+        q = self.q_proj(hidden_states).view(batch_size, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(hidden_states).view(batch_size, seq_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(hidden_states).view(batch_size, seq_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        q, k = apply_rotary_pos_emb(q, k, cos, sin)
+        k = repeat_kv(k, self.num_key_value_groups)
+        v = repeat_kv(v, self.num_key_value_groups)
+        scores = (q @ k.transpose(-2, -1)) / math.sqrt(self.head_dim)
+        if attention_mask is not None:
+            scores = scores.masked_fill(~attention_mask, torch.finfo(scores.dtype).min)
+        attn = F.softmax(scores, dim=-1)
+        attn = self.dropout(attn)
+        out = attn @ v
+        out = out.transpose(1, 2).contiguous().view(batch_size, seq_len, hidden_size)
+        return self.o_proj(out)
+
+
+class HanForgeMLP(nn.Module):
+    def __init__(self, config: HanForgeConfig):
+        super().__init__()
+        self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        return self.down_proj(F.silu(self.gate_proj(hidden_states)) * self.up_proj(hidden_states))
+
+
+class HanForgeDecoderLayer(nn.Module):
+    def __init__(self, config: HanForgeConfig, layer_idx: int):
+        super().__init__()
+        # DISABLED (refactor 20260423, §4.1): hybrid local/global 레이어 분기 비활성화.
+        # 모든 레이어가 causal full attention 경로로 동작한다.
+        # self.is_global = config.is_global_layer(layer_idx)
+        self.input_layernorm = HanForgeRMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.self_attn = HanForgeAttention(config, layer_idx)
+        self.post_attention_layernorm = HanForgeRMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.mlp = HanForgeMLP(config)
+
+    def forward(self, hidden_states: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, attention_mask: torch.Tensor):
+        hidden_states = hidden_states + self.self_attn(self.input_layernorm(hidden_states), cos, sin, attention_mask)
+        hidden_states = hidden_states + self.mlp(self.post_attention_layernorm(hidden_states))
+        return hidden_states
+
+
+class HanForgePreTrainedModel(PreTrainedModel):
+    config_class = HanForgeConfig
+    base_model_prefix = "model"
+    _no_split_modules = ["HanForgeDecoderLayer"]
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+
+
+class HanForgeModel(HanForgePreTrainedModel):
+    def __init__(self, config: HanForgeConfig):
+        super().__init__(config)
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList([HanForgeDecoderLayer(config, idx) for idx in range(config.num_hidden_layers)])
+        self.norm = HanForgeRMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.rotary_emb = HanForgeRotaryEmbedding(config)
+        self.post_init()
+
+    def _build_causal_mask(self, batch_size: int, seq_len: int, device: torch.device) -> torch.Tensor:
+        base = torch.tril(torch.ones(seq_len, seq_len, device=device, dtype=torch.bool))
+        return base.unsqueeze(0).unsqueeze(0).expand(batch_size, 1, seq_len, seq_len)
+
+    # DISABLED (refactor 20260423, §4.1): sliding window local mask 비활성화.
+    # def _build_local_mask(self, batch_size: int, seq_len: int, device: torch.device) -> torch.Tensor:
+    #     row = torch.arange(seq_len, device=device)[:, None]
+    #     col = torch.arange(seq_len, device=device)[None, :]
+    #     causal = col <= row
+    #     window = col >= (row - self.config.sliding_window + 1)
+    #     mask = (causal & window).to(torch.bool)
+    #     return mask.unsqueeze(0).unsqueeze(0).expand(batch_size, 1, seq_len, seq_len)
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        return_dict: bool = True,
+        **_: dict,
+    ):
+        batch_size, seq_len = input_ids.shape
+        hidden_states = self.embed_tokens(input_ids)
+        if position_ids is None:
+            position_ids = torch.arange(seq_len, device=input_ids.device).unsqueeze(0).expand(batch_size, -1)
+        cos, sin = self.rotary_emb(hidden_states, position_ids)
+        full_mask = self._build_causal_mask(batch_size, seq_len, hidden_states.device)
+        if attention_mask is not None:
+            key_mask = attention_mask[:, None, None, :].to(torch.bool)
+            full_mask = full_mask & key_mask
+
+        # DISABLED (refactor 20260423, §4.1): 모든 layer가 full causal mask 사용.
+        # local_mask 분기는 hybrid attention 재도입 시에만 사용한다.
+        for layer in self.layers:
+            hidden_states = layer(hidden_states, cos, sin, full_mask)
+
+        hidden_states = self.norm(hidden_states)
+        if not return_dict:
+            return (hidden_states,)
+        return BaseModelOutputWithPast(last_hidden_state=hidden_states)
+
+
+class HanForgeForCausalLM(HanForgePreTrainedModel, GenerationMixin):
+    # refactor 20260507 (§format/EOS): _tied_weights_keys 완전 제거.
+    # transformers 5.x의 _tied_weights_keys 메커니즘이 Phase 1 디버깅에서 from_pretrained 시
+    # .bin 파일의 학습된 weight를 silent하게 무시하고 random init 그대로 사용하는 버그를
+    # 일으킴. config tie_word_embeddings=False와 결합해서 두 weight를 별개로 명시 처리.
+    # (가능하면 학습 모델은 tie_word_embeddings=False로 저장. base 모델은 일시적으로 위험.)
+    _tied_weights_keys = None
+
+    def __init__(self, config: HanForgeConfig):
+        super().__init__(config)
+        self.model = HanForgeModel(config)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        self.post_init()
+        # refactor 20260423 (§9): tie lm_head.weight to embed_tokens.weight
+        # post_init 안에서 PreTrainedModel.tie_weights()가 동일 작업을 시도하지만,
+        # 작은 모델 + 32k vocab에서 파라미터 절약을 보장하기 위해 명시적으로 한다.
+        if getattr(config, "tie_word_embeddings", True):
+            self.lm_head.weight = self.model.embed_tokens.weight
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **kwargs):
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids, dtype=torch.long)
+        position_ids = attention_mask.long().cumsum(-1) - 1
+        position_ids = position_ids.clamp_min(0)
+        return {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "position_ids": position_ids,
+        }
+
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        return_dict: bool = True,
+        **kwargs,
+    ):
+        outputs = self.model(input_ids=input_ids, attention_mask=attention_mask, return_dict=True, **kwargs)
+        hidden_states = outputs.last_hidden_state
+        logits = self.lm_head(hidden_states)
+        loss = None
+        if labels is not None:
+            shift_logits = logits[:, :-1, :].contiguous()
+            shift_labels = labels[:, 1:].contiguous()
+            loss = F.cross_entropy(
+                shift_logits.view(-1, shift_logits.size(-1)),
+                shift_labels.view(-1),
+                ignore_index=-100,
+            )
+        if not return_dict:
+            result = (logits,)
+            if loss is not None:
+                result = (loss,) + result
+            return result
+        return CausalLMOutputWithPast(loss=loss, logits=logits)

special_tokens_map.json

@@ -0,0 +1,16 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "unk_token": "<unk>",
+  "pad_token": "<pad>",
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|mode:direct|>",
+    "<|mode:think|>",
+    "<think>",
+    "</think>",
+    "<answer>",
+    "</answer>"
+  ]
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c095b5e3b1b804920c2e482323371d204fbb234d9b07c8d0fe34dca93bb21b89
+size 647397

tokenizer_config.json

@@ -0,0 +1,124 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|mode:direct|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|mode:think|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<answer>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "</answer>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "backend": "custom",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|mode:direct|>",
+    "<|mode:think|>",
+    "<think>",
+    "</think>",
+    "<answer>",
+    "</answer>"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "HanForgeTokenizer",
+  "unk_token": "<unk>",
+  "vocab_size": 24000,
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenizer_hanforge.HanForgeTokenizer",
+      null
+    ]
+  }
+}

tokenizer_hanforge.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+
+import sentencepiece as spm
+from transformers import PreTrainedTokenizer
+
+
+class HanForgeTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "tokenizer.model"}
+    model_input_names = ["input_ids", "attention_mask"]
+
+    def __init__(
+        self,
+        vocab_file: str,
+        bos_token: str = "<s>",
+        eos_token: str = "</s>",
+        unk_token: str = "<unk>",
+        pad_token: str = "<pad>",
+        additional_special_tokens: list[str] | None = None,
+        **kwargs,
+    ):
+        self.vocab_file = vocab_file
+        self.sp_model = spm.SentencePieceProcessor(model_file=vocab_file)
+        super().__init__(
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            pad_token=pad_token,
+            additional_special_tokens=additional_special_tokens or [],
+            **kwargs,
+        )
+
+    @property
+    def vocab_size(self) -> int:
+        return int(self.sp_model.vocab_size())
+
+    def get_vocab(self) -> dict[str, int]:
+        vocab = {self.sp_model.id_to_piece(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text: str) -> list[str]:
+        return list(self.sp_model.encode(text, out_type=str))
+
+    def _convert_token_to_id(self, token: str) -> int:
+        return int(self.sp_model.piece_to_id(token))
+
+    def _convert_id_to_token(self, index: int) -> str:
+        return str(self.sp_model.id_to_piece(index))
+
+    def convert_tokens_to_string(self, tokens: list[str]) -> str:
+        return self.sp_model.decode_pieces(tokens)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
+        output = [self.bos_token_id] + list(token_ids_0)
+        if token_ids_1 is not None:
+            output += list(token_ids_1)
+        output += [self.eos_token_id]
+        return output
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: str | None = None):
+        save_dir = Path(save_directory)
+        save_dir.mkdir(parents=True, exist_ok=True)
+        out_name = f"{filename_prefix + '-' if filename_prefix else ''}tokenizer.model"
+        out_path = save_dir / out_name
+        if Path(self.vocab_file).resolve() != out_path.resolve():
+            shutil.copy2(self.vocab_file, out_path)
+        vocab_src = Path(self.vocab_file).with_suffix(".vocab")
+        if vocab_src.exists():
+            vocab_out = save_dir / f"{filename_prefix + '-' if filename_prefix else ''}tokenizer.vocab"
+            if vocab_src.resolve() != vocab_out.resolve():
+                shutil.copy2(vocab_src, vocab_out)
+        return (str(out_path),)