Instructions to use pathcosmos/JANSORI with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use pathcosmos/JANSORI with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="pathcosmos/JANSORI")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("pathcosmos/JANSORI") model = AutoModelForCausalLM.from_pretrained("pathcosmos/JANSORI", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use pathcosmos/JANSORI with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf pathcosmos/JANSORI:Q4_K_M
Use Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use pathcosmos/JANSORI with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "pathcosmos/JANSORI" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- SGLang
How to use pathcosmos/JANSORI with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "pathcosmos/JANSORI" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "pathcosmos/JANSORI" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Ollama
How to use pathcosmos/JANSORI with Ollama:
ollama run hf.co/pathcosmos/JANSORI:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use pathcosmos/JANSORI with Docker Model Runner:
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- Lemonade
How to use pathcosmos/JANSORI with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull pathcosmos/JANSORI:Q4_K_M
Run and chat with the model
lemonade run user.JANSORI-Q4_K_M
List all available models
lemonade list
- Atomic Chat
File size: 16,315 Bytes
e68bfa3 82f5b3e e68bfa3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 | ---
language: [ko, en]
pipeline_tag: text-generation
license: apache-2.0
library_name: transformers
tags: [gguf]
---
# JANSORI Β· νκ΅μ΄ μ€μ¬ 3B Dense Base
νκ΅μ΄μ μμ΄ λ¬Έμ₯μ μ΄μ΄ μ°λ μ¬μ νμ΅ λͺ¨λΈμ΄λ€. μ΄ν ν¬κΈ°λ 65,536, κ²μ¦ν λ¬Έλ§₯μ 4,096ν ν°μ΄λ€. λνμ© μν ν ν° μμ΄ μΌλ° ν
μ€νΈλ₯Ό μ
λ ₯νλ€.
## vLLMμΌλ‘ BF16 μ€ν
vLLM 0.11.2μμ H200 ν μ₯μ μ¬μ©νλ€. μλ λͺ
λ Ήμ λ€μ΄λ‘λ μμ μ commit SHAλ₯Ό μ‘°νν΄ κ°μ€μΉμ ν ν¬λμ΄μ λ₯Ό κ°μ revisionμΌλ‘ κ³ μ νλ€. μ΄μ λ²μ μ μ¬νν λλ κΈ°λ‘ν΄ λ SHAλ₯Ό REVμ μ§μ νλ€.
```bash
REV=$(python -c 'from huggingface_hub import HfApi; print(HfApi().model_info("pathcosmos/JANSORI").sha)')
CUDA_VISIBLE_DEVICES=0 vllm serve pathcosmos/JANSORI --revision "$REV" --tokenizer-revision "$REV" --load-format safetensors --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 4096 --host 127.0.0.1
```
λ€λ₯Έ μ
Έμμ νκ΅μ΄ μ΄μ΄μ°κΈ°λ₯Ό μμ²νλ€.
```bash
curl -s http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' -d '{"model":"pathcosmos/JANSORI","prompt":"νκ΅μ μλλ","max_tokens":32,"temperature":0}'
```
## Ollamaλ‘ Q8_0 λλ Q4_K_M μ€ν
Ollama 0.40.16μ μ¬μ©νλ€. H200 ν μ₯μμ νκ΅μ΄ μ΄μ΄μ°κΈ°λ₯Ό μνν λλ μ¬μ©ν μμν νλλ₯Ό μ ννλ€.
```bash
ollama run hf.co/pathcosmos/JANSORI:Q8_0 "νκ΅μ μλλ"
ollama run hf.co/pathcosmos/JANSORI:Q4_K_M "νκ΅μ μλλ"
```
루νΈμ `template`μ μ
λ ₯ ν둬ννΈλ₯Ό κ·Έλλ‘ μ λ¬νλ€. μμν νκ·Έλ λ³κ²½λ μ μλ€. μ€ν λΉμ μ μ₯μ commitκ³Ό λ΄λ €λ°μ GGUFμ SHA-256μ ν¨κ» κΈ°λ‘νλ©΄ μ¬μ©ν κ°μ€μΉλ₯Ό νμΈν μ μλ€.
μΈ νμμ κ°μ λ°°ν¬ commitμ λ€μ΄ μλ€. μ΄ μ μ₯μμ BF16μμ κ³ μ λ° GGUF(SHA-256 `c69c8a77121940e0c1f06f668ba0a80e57edba1afaac777cae3ea3f923d69b7b`)λ₯Ό ν λ² μμ±ν λ€ Q8_0κ³Ό Q4_K_MμΌλ‘ κ°κ° μμννλ€. BF16 shardμ κ° GGUFμ SHA-256μ κ°μ commitμ [SHA256SUMS](./SHA256SUMS)μ μλ€.
## μ€μ 3B νκ°
κ°μ holdoutμ νκ΅μ΄ 663,905κ°Β·μμ΄ 705,634κ° νμ μ μ±μ νλ€. μλμ VRAMμ H200μμ 첫 μμ°¨ νμ°¨μ κ³ μ ν둬ννΈ 6κ°(μ΅λ 64ν ν°)λ‘ μΈ‘μ νλ€.
| νμ | ν¨ν€μ§ λ°μ΄νΈΒΉ | νκ΅μ΄ PPL | μμ΄ PPL | μμ² κΈ°μ€ ν ν°/μ΄ | μμ± κ΅¬κ° GPU μ΅κ³ (MiB) |
|---|---:|---:|---:|---:|---:|
| BF16 | 6,448,116,613 | 9.089808 | 10.215738 | 100.27 | 8805.9 |
| Q8_0 | 3,426,185,184 | 9.090860 | 10.217286 | 14.49 | 5055.9 |
| Q4_K_M | 1,969,663,968 | 9.196303 | 10.317744 | 9.95 | 3761.9 |
ΒΉ BF16μ κ°μ€μΉΒ·μ€μ Β·ν ν¬λμ΄μ ν©κ³, GGUFλ κ° κ°μ€μΉ νμΌ ν¬κΈ°λ€.
native νκ΅μ΄ PPLμ 9.087589, μμ΄λ 10.213156μ΄μλ€. νμ€ HFμ μλ native BF16 μμΉ λΉκ΅μμ 4,096ν ν° λ‘μ§ 268,435,456κ° μ€ 297κ°κ° νμ©λμ μ΄κ³Όνλ€. RoPE λ°μ¬λ¦Όκ³Ό attention νΈμΆ λ°©μμ λ§μΆ native λμ‘°μμλ 8κ° μ
λ ₯μ μ 체 λ‘μ§μ΄ κ°μκ³ μΊμ λΉκ΅κ° ν΅κ³Όνλ€.
### κ³ μ ν둬ννΈμμ κ΄μ°°λ νκ³
- λΉμ κ΅΄μ μ€λͺ
μμ μΈ νμ λͺ¨λ λ§€μ§ κ²½κ³μ κ΄ν 문ꡬλ₯Ό λ°λ³΅νκ³ κ΅΄μ μμΈμ μ€λͺ
νμ§ λͺ»νλ€.
- μ¬κ³Ό 12κ°λ₯Ό 3λͺ
μκ² λλλ νκ΅μ΄ λ¬Έμ μμ μΈ νμ λͺ¨λ 12κ°λΌκ³ λ΅νλ€. μ λ΅μ 4κ°λ€.
- νκ΅μ΄ add ν¨μμ 첫 λ°νμμ λͺ¨λ a + bμλ€. BF16Β·Q8μ λ²νΈ μ£Όμμ λ°λ³΅νκ³ , Q4λ λ μΈμ ν¨μμ μΈ κ° μ΄μμ μΈμλ₯Ό λκΈ°λ νΈμΆμ μΆκ°νλ€.
- μμ΄ μ΄μΌκΈ°μμλ BF16Β·Q8μ΄ κ°λν κ³ΌλΆλ₯Ό, Q4κ° λΆμ ν λ¨νΈκ³Ό μ¬λ Saraλ₯Ό λ±μ₯μμΌ°λ€. μΈ μΆλ ₯ λͺ¨λ ν λ¬Έμ₯μ΄λ ννμ λ°λ³΅νλ€.
- μμ΄ λ°©μ μμμ BF16Β·Q8μ μ²μ 4λΌκ³ μ΄ λ€ νμ΄μμ 5λ‘ μ μ νλ€. Q4λ 첫 μλΆν° 12 β 7 = 5λ‘ κ³μ°νλ€.
- μ§μ νλ³ ν¨μμμ μΈ νμ λͺ¨λ λ€μ¬μ°κΈ°λ₯Ό μμλ€. BF16Β·Q8μ μ§μ λΆκΈ°μ Trueλ₯Ό μΌκ³ Q4λ Falseλ₯Ό μ¨μ νμ λ λ€μ§νλ€.
### μ΄μ΄μ°κΈ° μμ
μ
λ ₯
```text
νκ΅μ΄λ‘ λΉμ΄ κ΅΄μ λλ μ΄μ λ₯Ό κ°λ¨ν μ€λͺ
νλ©΄
```
μμ± (BF16)
```text
λ€μκ³Ό κ°λ€.
1. λΉμ κ΅΄μ
λΉμ νλμ΄λ€. νλμ λ§€μ§μ κ²½κ³μμ μ§λνλ νλμ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€.
```
μ
λ ₯
```text
Once upon a time, in a quiet village
```
μμ± (BF16)
```text
, there lived a poor old woman. She was a widow, and she had no children. She was very old, and she was very poor. She had no money, and she had no food. She was very lonely, and she was very poor. She was very old, and she was very poor. She
```
## H200μμ μΈ‘μ ν νμ§κ³Ό μ€ν μ€μ
μ΄ κ²°κ³Όλ 47,000μ€ν
μ [JANSORI λ°°ν¬ μ»€λ°](https://huggingface.co/pathcosmos/JANSORI/tree/e68bfa3cb31e9589f4481a2e31d6d75c731fec9d)μ μ μ©νλ€. BF16μ vLLM 0.11.2, Q8_0Β·Q4_K_Mμ Ollama 0.40.16μΌλ‘ κ°μ 물리 H200 ν μ₯μμ κ°κ° μ€ννλ€. μ΄μ λ°°ν¬ κ²μ¦μ 6κ° κ³ μ μ
λ ₯μ νμλ³λ‘ 6/6 μ¬νλλ€.
### νκ΅μ΄Β·μμ΄Β·μνΒ·μ½λ
μ체 μμ± μνμ νμλΉ 98κ° μ
λ ₯μ΄λ€. μ§§μ μ¬μ€ μ§λ¬Έ 24κ°, μμ λ
ν΄ 16κ°, μ°μ 32κ°, ν¨μ 12κ°, μ΄μΌκΈ° μ΄μ΄μ°κΈ° 8κ°, νκ· μ
λ ₯ 6κ°λ₯Ό μ¬μ©νλ€. μΈ νμμ μ
λ ₯κ³Ό μμ± μνμ κ°κ² νλ€. μ¬μ€Β·λ
ν΄λ 첫 λ΅μ μ κ·νν΄ νμ© λ΅κ³Ό λΉκ΅νλ€. μ°μλ μꡬν μ μκ° μ²« μ€μ λ΅μΌλ‘ λμλμ§ μ±μ νλ€. λ¬Έν μκ° μμΌλ―λ‘ ν λ¬Έν μ°¨μ΄λ‘ νμμ μ°μ΄μ μ νκΈ° μ΄λ ΅λ€.
| νμ | νκ΅μ΄Β·μμ΄ μ¬μ€ (κ° 12) | νκ΅μ΄Β·μμ΄ λ
ν΄ (κ° 8) | κΈ°λ³ΈΒ·λ λ¨κ³ μ°μ (κ° 16) | ν¨μ μ 체 ν΅κ³Ό, μ€λͺ
λ§ / μμ+λ°νλ¬Έ (κ° 12) | BoolQ μ λ΅ / 1,404 | BoolQ macro-F1 |
|---|---:|---:|---:|---:|---:|---:|
| BF16 | 10 Β· 10 | 7 Β· 7 | 15 Β· 0 | 0 / 9 | 730 (52.0%) | 0.413 |
| Q8_0 | 10 Β· 10 | 8 Β· 7 | 15 Β· 0 | 0 / 9 | 729 (51.9%) | 0.410 |
| Q4_K_M | 10 Β· 10 | 7 Β· 7 | 14 Β· 1 | 0 / 9 | 733 (52.2%) | 0.410 |
λ μμ ν¨μμ λμ ν¨μμ `return`κΉμ§ μ
λ ₯ν 쑰건μ κ°μ 12κ° ν¨μμ ννμ μμ± λ¬Έμ λ€. λͺ¨λΈμ΄ λλ¨Έμ§ λ³Έλ¬Έ μ 체λ₯Ό μμ±νλ μ‘°κ±΄λ³΄λ€ μ½λ€. μΆλ ₯μ λ€μ¬μ°κΈ°λ₯Ό μ리νμ§ μκ³ μ²« ν¨μλ§ μΆμΆν΄ νμ±νκ³ , ν¨μλΉ 4κ° λ
립 μ¬λ‘μ κ°κ³Ό λ°ν νμ
μ κ²μ¬νλ€. μΈ νμ λͺ¨λ κ°μ₯ ν° ν¬μ λ₯λ₯Ό μ½λΌλ¦¬λΌκ³ λ΅νλ€. μ°μ μ΄ μ λ΅μ λͺ¨λ 15/32μμ§λ§ μ λ΅μ λΈ λ¬Ένμ΄ μΌλΆ λ¬λλ€.
νκ΅μ΄ KoBEST BoolQλ ν
μ€νΈ 1,404κ° μ 체μμ μ§λ¬ΈΒ·μ§λ¬Έ λ€ λ¨μΌ ν ν°μΈ β κ±°μ§βΒ·β μ°Έβμ μ‘°κ±΄λΆ λ‘κ·Ένλ₯ μ λΉκ΅ν zero-shot μνμ΄λ€. BF16μλ κ°μ λ°°ν¬ κ°μ€μΉμ Transformers BF16 forward, GGUFμλ ν΄λΉ νμΌμ llama.cpp μ±μ κΈ°λ₯Ό μ¬μ©νλ€. λ€μ ν΄λμ€λ μ°Έ 705/1,404(50.2%)λ€. λ€μ νμ μ°Έ μμΈ‘ λΉλμ macro-F1μ ν¨κ» ν΄μνλ€. λ°μ΄ν° λΌμ΄μ μ€λ CC BY-SA 4.0μ΄λ©° μνμ νμ΅ λ°μ΄ν° μ€λ³΅ μ¬λΆλ λ―ΈνμΈμ΄λ€. [KoBEST κ³ μ revision](https://huggingface.co/datasets/skt/kobest_v1/tree/a5ea15e3ac77ed694b79f6204eb31889a2ba989f)
| νμ | BoolQ μ°Έ μμΈ‘ μ | BF16 λλΉ λ§μβνλ¦Ό / νλ¦Όβλ§μ | νκ΅μ΄ / μμ΄ μ΄μΌκΈ° λ°λ³΅ 4-gram | νκ΅μ΄ / μμ΄μμ μ₯λ©΄μ μ κ°ν μΆλ ₯ (κ° 4κ°) |
|---|---:|---:|---:|---:|
| BF16 | 1299 | 0 / 0 | 53.7% / 58.6% | 1 / 1 |
| Q8_0 | 1302 | 5 / 4 | 42.6% / 61.9% | 2 / 0 |
| Q4_K_M | 1312 | 19 / 22 | 66.4% / 68.4% | 0 / 0 |
λ°λ³΅λ₯ μ 64ν ν° μμ±λ¬Όμ λ€μ ν ν°νν΄ μ²μ λ±μ₯ν μ΄νμ μ€λ³΅ 4-gram μλ₯Ό μ 체 4-gram μλ‘ λλ κ°μ΄λ€. μ΄μ΄μ°κΈ° 8κ°λ νμλͺ
μ κ°λ¦¬κ³ κ°μ μ
λ ₯Β·κ°μ μΆλ ₯μ ν©μ³ λ¨μΌ AI νλ
μκ° 0β2μ μΌλ‘ λΆλ₯νλ€. νμλ μλ‘ λ€λ₯Έ μΈλΆ λ΄μ©μΌλ‘ μ₯λ©΄μ΄ μ κ°λ 2μ μΆλ ₯ μλ₯Ό νμνλ€. μλ₯Ό λ€μ΄ λΉκ° κ·ΈμΉ μ΄λμ₯μ΄λΌλ μμ΄ μ
λ ₯μμλ μλ λ¬μ¬κ° κ°μ λ¬Έμ₯μΌλ‘ λ°λ³΅λλ€. μ¬λ νκ°μ κ° μΌμΉλλ μΈ‘μ νμ§ μμλ€.
### μΈμ΄λ³ holdout PPL
κ²μ¦λ 1,466λ¬Έμμ λμΌ ν ν° IDΒ·λ§μ€ν¬Β·λ¬Έμλ³ NLL ν©μ μ¬μ¬μ©νλ€. νκ΅μ΄ 663,905κ°, μμ΄ 705,634κ° νμ μ΄λ€. κ΄νΈ μμ BF16 λλΉ μλ μμΉλ₯ μ 95% ꡬκ°μ΄λ€. λ¬Έμλ₯Ό μΈμ΄λ³λ‘ μ§μ§μ΄ 2,000ν 볡μ μΆμΆνκ³ λ§€λ² νμ ν ν° κ°μ€ νκ· NLLλ‘ PPLμ ꡬνλ€.
| νμ | νκ΅μ΄ PPL | νκ΅μ΄ μμΉλ₯ (95% ꡬκ°) | μμ΄ PPL | μμ΄ μμΉλ₯ (95% ꡬκ°) |
|---|---:|---:|---:|---:|
| BF16 | 9.08981 | κΈ°μ€ | 10.21574 | κΈ°μ€ |
| Q8_0 | 9.09086 | +0.012% (+0.002 ~ +0.022%) | 10.21729 | +0.015% (+0.005 ~ +0.025%) |
| Q4_K_M | 9.19630 | +1.172% (+1.110 ~ +1.235%) | 10.31774 | +0.999% (+0.952 ~ +1.042%) |
### λ¬Έλ§₯ κΈΈμ΄μ λ©λͺ¨λ¦¬
μλ²λ₯Ό μμ΄ν λ€ μ
λ ₯ κΈΈμ΄Β·μΆλ ₯ μνλ³λ‘ 5νμ© μμλ₯Ό μμλ€. μ±λ₯ μ
λ ₯μ ν λ¨μ΄λΉ ν ν ν°μΈ κ³ μ μ΄νμμ seedλ‘ λ½μμΌλ©° νκ΅μ΄ νμ§ μν μ
λ ₯κ³Ό λ³λλ€. κ° νμμ κ°μ ν둬ννΈλ₯Ό μ μ‘νκ³ μ±κ³΅ μλ΅μ μ
λ ₯Β·μΆλ ₯ ν ν° μλ₯Ό μ μ νμΈνλ€. μλλ μΆλ ₯ 128ν ν°, λ¨μΌ μμ²μ μ€μκ°μ΄λ€. 첫 ν
μ€νΈκ° λμ°©ν λκΉμ§κ° TTFTμ΄κ³ μμ± μλλ μ΄ν λ¨μ ν ν°μ μλ£ν λκΉμ§μ μλλ€. μ΅κ³ VRAMμ μμ² κ²½κ³μ 0.5μ΄ κ°κ²©μμ κ΄μΈ‘ν μ₯μΉ μ 체 μ¬μ©λμ΄λ€.
| μ
λ ₯ ν ν° | νμ | μ±κ³΅ / μλ | TTFT (ms) | μ 체 μ§μ° (μ΄) | μμ± ν ν°/μ΄ | μ΅κ³ VRAM (MiB) |
|---:|---|---:|---:|---:|---:|---:|
| 128 | BF16 | 5/5 | 19.7 | 1.280 | 100.8 | 8805.9 |
| 128 | Q8_0 | 5/5 | 65.9 | 0.445 | 335.2 | 5145.9 |
| 128 | Q4_K_M | 5/5 | 65.5 | 0.418 | 361.2 | 3851.9 |
| 1,024 | BF16 | 5/5 | 26.2 | 1.290 | 100.5 | 8805.9 |
| 1,024 | Q8_0 | 5/5 | 112.7 | 0.490 | 335.4 | 5145.9 |
| 1,024 | Q4_K_M | 5/5 | 131.5 | 0.483 | 362.0 | 3851.9 |
| 3,072 | BF16 | 5/5 | 57.6 | 1.321 | 100.5 | 8805.9 |
| 3,072 | Q8_0 | 5/5 | 265.8 | 0.663 | 320.2 | 5145.9 |
| 3,072 | Q4_K_M | 5/5 | 292.1 | 0.666 | 343.2 | 3851.9 |
| 3,968 | BF16 | 5/5 | 72.3 | 1.334 | 100.8 | 8805.9 |
| 3,968 | Q8_0 | 5/5 | 362.4 | 0.753 | 324.8 | 5145.9 |
| 3,968 | Q4_K_M | 4/5 | 404.1 | 0.768 | 349.0 | 3851.9 |
Q4_K_Mμ 3,968+128 쑰건μμ ν μμ²μ΄ λ°λ³΅ ν ν° μ νμΌλ‘ μ’
λ£λμΌλ©° κ°μ μ
λ ₯μ λ³λ μ¬νμμλ κ°μ μ€λ₯κ° λ°μνλ€. μ΄ νμ μ§μ°Β·μλλ μ±κ³΅ν 4νλ‘ κ³μ°νκ³ VRAMμλ μ€ν¨ μμ²λ ν¬ν¨νλ€. 4,096μ μ
λ ₯κ³Ό μΆλ ₯μ ν©ν νλλ€. μλ²κ° μμ΄λ μνμμ KV μΊμμ μ€ν λ²νΌλ₯Ό μ¬μ¬μ©νλ―λ‘ μ
λ ₯ κΈΈμ΄κ° λ°λμ΄λ κ΄μΈ‘ μ΅κ³ VRAMμ΄ λΉμ·ν μ μλ€.
### λμ μμ²κ³Ό νμ μ ν
μ
λ ₯ 1,024ν ν°Β·μΆλ ₯ 32ν ν°μμ λμμ 1Β·2Β·4κ° μμ²μ 보λλ€. νμ μ²λ¦¬λμ μμ² λ¬Άμ μ 체μ μΆλ ₯ ν ν° μλ₯Ό 첫 μμ² μμλΆν° λ§μ§λ§ μλ£κΉμ§μ μκ°μΌλ‘ λλ κ°μ μ€μκ°μ΄λ€. Ollamaλ μλ² λ³λ ¬ μ¬λ‘― 1κ°λ‘ ν λκΈ°λ₯Ό ν¬ν¨νλ€. vLLMμ μ΅λ 2κ° μνμ€λ₯Ό ν¨κ» μ²λ¦¬νλ€.
| νμ | λμ 1 / 2 / 4 μ 체 μ²λ¦¬λ (ν ν°/μ΄) | λμ 4 μμ² μ§μ° μ€μκ° / κ΄μΈ‘ μ΅λκ° (μ΄) |
|---|---:|---:|
| BF16 | 95.0 / 179.3 / 185.1 | 0.517 / 0.690 |
| Q8_0 | 102.2 / 143.9 / 140.8 | 0.586 / 0.990 |
| Q4_K_M | 102.2 / 144.9 / 142.0 | 0.584 / 0.975 |
νμλ³ κΆμ₯ μμ μ€μ μ λ€μκ³Ό κ°λ€. μλ λΉκ΅μλ κ°μ€μΉ μ λ°λλΏ μλλΌ λ°νμ·컀λΒ·μΊμ μ€μ μ°¨μ΄λ ν¬ν¨λλ€.
- **BF16 Β· μ νλ λΉκ΅ κΈ°μ€, μ§§μ 첫 μλ΅ λλ λμ μμ².** H200 1μ₯μ `--dtype bfloat16 --load-format safetensors --tensor-parallel-size 1 --max-model-len 4096 --max-num-seqs 2 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.25 --num-gpu-blocks-override 512 --enforce-eager --no-enable-prefix-caching --generation-config vllm --seed 47000`μ μ¬μ©νλ€. `VLLM_ATTENTION_BACKEND=FLASH_ATTN`κ³Ό `VLLM_USE_FLASHINFER_SAMPLER=0`μ μ μ©νλ€. `/v1/completions`λ‘ `temperature=0`κ³Ό λͺ©μ μ λ§λ `max_tokens`λ₯Ό μ λ¬νλ€.
- **Q8_0 Β· Ollamaμ νμ§ μ°μ κΈ°λ³Έκ°.** BF16μ κ°κΉμ΄ PPLμ΄ νμν κ²½μ° `hf.co/pathcosmos/JANSORI:Q8_0`μ μ ννλ€. `/api/generate`μμ `raw=true`, `stream=true`, `num_ctx=4096`, `num_predict=32`(μ§§μ λ΅) λλ `128`(μ΄μ΄μ°κΈ°), `temperature=0`, `repeat_penalty=1`, `seed=47000`μ μ¬μ©νλ€. μ΄ H200 μνμ `num_gpu=99`, `num_thread=4`, `OLLAMA_NUM_PARALLEL=1`μ΄μλ€.
- **Q4_K_M Β· λ©λͺ¨λ¦¬ μ°μ .** κ°μ Ollama μ΅μ
μ νκ·Έλ₯Ό `Q4_K_M`μΌλ‘ λ°κΎΌλ€. κΈ΄ μ
λ ₯μμλ μ΅λ κΈΈμ΄λ³΄λ€ μ¬μ λ₯Ό λκ³ μμ± μνμ λͺ
μνλ€. μ΄λ² 3,072+128 μνμ 5/5 μλ£λλ€. Q4κ° λͺ¨λ μ
λ ₯ κΈΈμ΄μμ κ°μ₯ λΉ λ₯΄λ€λ 보μ₯μ μλ€.
- **μ°μΒ·μ½λ.** μ§§μ μ λ΅μ μꡬνλ λ¨μ μ°μμ λ λ¨κ³ μ°μ κ²°κ³Όλ₯Ό ꡬλΆνλ€. κ°λ¨ν Python ν¨μ μμ±μλ ν¨μ μμ λ κ°μ λͺ©ν ν¨μμ `return` μ λμ΄λ₯Ό μ μ©ν λ€ λ³λ ν
μ€νΈλ‘ κ²μ¦νλ€. μ΄ κ²°κ³Όλ μμ νλ‘κ·Έλ¨ μμ± λ₯λ ₯μ νκ°νμ§ μλλ€.
### H200 λ μ₯μμ μμ²μ λ³νν κ²½μ°
GPU 0μ BF16, GPU 1μ Ollama ν νμμ μ μ¬ν λ€ 1,024ν ν° μ
λ ₯κ³Ό 32ν ν° μΆλ ₯μ λ μλ²μ κ°κ° μμ²νλ€. νμ μ‘°ν©λ³λ‘ μλ‘ λ€λ₯Έ μ
λ ₯ 5κ°λ₯Ό μ¬μ©νλ€. κ°μ μ
λ ₯μ μμ°¨μ λ³νμ λͺ¨λ μ¬μ©νκ³ μ΄λ μͺ½μ λ¨Όμ μΈ‘μ ν μ§λ κ΅λνλ€. κ° μΈ‘μ μ λ³λ 1,024ν ν° μμ²μΌλ‘ μ§μ μ
λ ₯ μΊμκ° κ·Έλλ‘ μ¬μ¬μ©λλ κ²½μ°λ₯Ό μ€μλ€. μλ² μμΒ·λͺ¨λΈ λ‘λ©μ μ΄ νμμ μ μΈνλ€.
| λμ μμ£Ό νμ | μμ°¨ / λ³ν λ¬Άμ μ§μ° μ€μκ° (μ΄) | λ³ν μμ² μ€μ²© μ€μκ° (μ΄) | λ μΆλ ₯ λͺ¨λ λμΌν μ
λ ₯ | λμΌ μΆλ ₯ μλ§μ μμ°¨ / λ³ν μ€μκ° (μ΄) | μ΅κ³ λμ ν©μ° VRAM, μμ°¨ / λ³ν (MiB) |
|---|---:|---:|---:|---:|---:|
| BF16 + Q8_0 | 0.568 / 0.323 | 0.240 | 4/5 | 0.551 / 0.324 | 13889.8 / 13889.8 |
| BF16 + Q4_K_M | 0.556 / 0.324 | 0.235 | 3/5 | 0.559 / 0.324 | 12595.8 / 12595.8 |
μμͺ½ λͺ¨λΈμ΄ μμ£Όν μνμ΄λ―λ‘ μμ°¨ μμ²λ λ λͺ¨λΈμ VRAMμ μ μ§νλ€. λμΌ μΆλ ₯ μ μκ° μ μ λλ μλ μ°¨μ΄λ₯Ό λμΌ νμ§μμ 보μ₯λλ κ°μ μΌλ‘ ν΄μνκΈ° μ΄λ ΅λ€. κΈ°μ‘΄ κ³΅κ° κ²μ¦μ μλ² μμλΆν° μ’
λ£κΉμ§ μ° μμΉλ λ€λ₯Έ μ§νλ€.
## νμ΅ λ°μ΄ν°
νκ΅μ΄ μ€μ¬ 3B Dense Base λͺ¨λΈμ 47,000μ€ν
μμ λμ 98,566,144,000ν ν°μ νμ΅νλ€. μλ μμΉλ νμ΅ μ λ°λ³΅ λ
ΈμΆμ ν¬ν¨ν μΆμ²λ³ μ
λ ₯ ν ν° μλ€.
| λ°μ΄ν°μ
| μ¬μ© λ²μ | νμ΅ ν ν° | λΉμ€ |
|---|---|---:|---:|
| [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia/blob/b04c8d1ceb2f5cd4588862100d08de323dccfbaa/README.md) | 2023-11-01 νκ΅μ΄ μν€λ°±κ³Ό λ€ν 3κ° μ€λ | 494,833,664 | 0.502% |
| [HuggingFaceFW/fineweb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2/blob/af9c13333eb981300149d5ca60a8e9d659b276b9/README.md) | νκ΅μ΄ kor_Hang 25κ° μ€λ | 29,038,194,688 | 29.461% |
| [HPLT/HPLT2.0_cleaned](https://huggingface.co/datasets/HPLT/HPLT2.0_cleaned/blob/d1324a5283f762ee62c2a5c81de08fc6450ea540/README.md) | νκ΅μ΄ kor_Hang 464κ° μ€λ | 19,756,912,640 | 20.044% |
| [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/blob/87f09149ef4734204d70ed1d046ddc9ca3f2b8f9/README.md) | 100BT νλ³Έμμ νμΌλͺ
μ μ 56κ° μ€λ | 29,557,293,056 | 29.987% |
| [open-web-math/open-web-math](https://huggingface.co/datasets/open-web-math/open-web-math/blob/fde8ef8de2300f5e778f56261843dab89f230815/README.md) | 114κ° μ€λ | 9,861,763,072 | 10.005% |
| [codeparrot/github-code](https://huggingface.co/datasets/codeparrot/github-code/blob/b5661e6b17396364b2bcf8e68977b0d28e1ebd19/README.md) | 1,126κ° μ€ κ· λ± κ°κ²©μΌλ‘ μ μ ν 64κ° μ€λ | 9,857,146,880 | 10.001% |
|