Instructions to use pathcosmos/JANSORI with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use pathcosmos/JANSORI with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="pathcosmos/JANSORI")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("pathcosmos/JANSORI") model = AutoModelForCausalLM.from_pretrained("pathcosmos/JANSORI", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use pathcosmos/JANSORI with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf pathcosmos/JANSORI:Q4_K_M
Use Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use pathcosmos/JANSORI with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "pathcosmos/JANSORI" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- SGLang
How to use pathcosmos/JANSORI with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "pathcosmos/JANSORI" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "pathcosmos/JANSORI" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Ollama
How to use pathcosmos/JANSORI with Ollama:
ollama run hf.co/pathcosmos/JANSORI:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use pathcosmos/JANSORI with Docker Model Runner:
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- Lemonade
How to use pathcosmos/JANSORI with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull pathcosmos/JANSORI:Q4_K_M
Run and chat with the model
lemonade run user.JANSORI-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Document detailed JANSORI evaluations and recommended inference settings
Browse files- README.md +84 -0
- SHA256SUMS +1 -1
README.md
CHANGED
|
@@ -89,6 +89,90 @@ Once upon a time, in a quiet village
|
|
| 89 |
, there lived a poor old woman. She was a widow, and she had no children. She was very old, and she was very poor. She had no money, and she had no food. She was very lonely, and she was very poor. She was very old, and she was very poor. She
|
| 90 |
```
|
| 91 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
## νμ΅ λ°μ΄ν°
|
| 93 |
|
| 94 |
νκ΅μ΄ μ€μ¬ 3B Dense Base λͺ¨λΈμ 47,000μ€ν
μμ λμ 98,566,144,000ν ν°μ νμ΅νλ€. μλ μμΉλ νμ΅ μ λ°λ³΅ λ
ΈμΆμ ν¬ν¨ν μΆμ²λ³ μ
λ ₯ ν ν° μλ€.
|
|
|
|
| 89 |
, there lived a poor old woman. She was a widow, and she had no children. She was very old, and she was very poor. She had no money, and she had no food. She was very lonely, and she was very poor. She was very old, and she was very poor. She
|
| 90 |
```
|
| 91 |
|
| 92 |
+
## H200μμ μΈ‘μ ν νμ§κ³Ό μ€ν μ€μ
|
| 93 |
+
|
| 94 |
+
μ΄ κ²°κ³Όλ 47,000μ€ν
μ [JANSORI λ°°ν¬ μ»€λ°](https://huggingface.co/pathcosmos/JANSORI/tree/e68bfa3cb31e9589f4481a2e31d6d75c731fec9d)μ μ μ©νλ€. BF16μ vLLM 0.11.2, Q8_0Β·Q4_K_Mμ Ollama 0.40.16μΌλ‘ κ°μ 물리 H200 ν μ₯μμ κ°κ° μ€ννλ€. μ΄μ λ°°ν¬ κ²μ¦μ 6κ° κ³ μ μ
λ ₯μ νμλ³λ‘ 6/6 μ¬νλλ€.
|
| 95 |
+
|
| 96 |
+
### νκ΅μ΄Β·μμ΄Β·μνΒ·μ½λ
|
| 97 |
+
|
| 98 |
+
μ체 μμ± μνμ νμλΉ 98κ° μ
λ ₯μ΄λ€. μ§§μ μ¬μ€ μ§λ¬Έ 24κ°, μμ λ
ν΄ 16κ°, μ°μ 32κ°, ν¨μ 12κ°, μ΄μΌκΈ° μ΄μ΄μ°κΈ° 8κ°, νκ· μ
λ ₯ 6κ°λ₯Ό μ¬μ©νλ€. μΈ νμμ μ
λ ₯κ³Ό μμ± μνμ κ°κ² νλ€. μ¬μ€Β·λ
ν΄λ 첫 λ΅μ μ κ·νν΄ νμ© λ΅κ³Ό λΉκ΅νλ€. μ°μλ μꡬν μ μκ° μ²« μ€μ λ΅μΌλ‘ λμλμ§ μ±μ νλ€. λ¬Έν μκ° μμΌλ―λ‘ ν λ¬Έν μ°¨μ΄λ‘ νμμ μ°μ΄μ μ νκΈ° μ΄λ ΅λ€.
|
| 99 |
+
|
| 100 |
+
| νμ | νκ΅μ΄Β·μμ΄ μ¬μ€ (κ° 12) | νκ΅μ΄Β·μμ΄ λ
ν΄ (κ° 8) | κΈ°λ³ΈΒ·λ λ¨κ³ μ°μ (κ° 16) | ν¨μ μ 체 ν΅κ³Ό, μ€λͺ
λ§ / μμ+λ°νλ¬Έ (κ° 12) | BoolQ μ λ΅ / 1,404 | BoolQ macro-F1 |
|
| 101 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 102 |
+
| BF16 | 10 Β· 10 | 7 Β· 7 | 15 Β· 0 | 0 / 9 | 730 (52.0%) | 0.413 |
|
| 103 |
+
| Q8_0 | 10 Β· 10 | 8 Β· 7 | 15 Β· 0 | 0 / 9 | 729 (51.9%) | 0.410 |
|
| 104 |
+
| Q4_K_M | 10 Β· 10 | 7 Β· 7 | 14 Β· 1 | 0 / 9 | 733 (52.2%) | 0.410 |
|
| 105 |
+
|
| 106 |
+
λ μμ ν¨μμ λμ ν¨μμ `return`κΉμ§ μ
λ ₯ν 쑰건μ κ°μ 12κ° ν¨μμ ννμ μμ± λ¬Έμ λ€. λͺ¨λΈμ΄ λλ¨Έμ§ λ³Έλ¬Έ μ 체λ₯Ό μμ±νλ μ‘°κ±΄λ³΄λ€ μ½λ€. μΆλ ₯μ λ€μ¬μ°κΈ°λ₯Ό μ리νμ§ μκ³ μ²« ν¨μλ§ μΆμΆν΄ νμ±νκ³ , ν¨μλΉ 4κ° λ
립 μ¬λ‘μ κ°κ³Ό λ°ν νμ
μ κ²μ¬νλ€. μΈ νμ λͺ¨λ κ°μ₯ ν° ν¬μ λ₯λ₯Ό μ½λΌλ¦¬λΌκ³ λ΅νλ€. μ°μ μ΄ μ λ΅μ λͺ¨λ 15/32μμ§λ§ μ λ΅μ λΈ λ¬Ένμ΄ μΌλΆ λ¬λλ€.
|
| 107 |
+
|
| 108 |
+
νκ΅μ΄ KoBEST BoolQλ ν
μ€νΈ 1,404κ° μ 체μμ μ§λ¬ΈΒ·μ§λ¬Έ λ€ λ¨μΌ ν ν°μΈ β κ±°μ§βΒ·β μ°Έβμ μ‘°κ±΄λΆ λ‘κ·Ένλ₯ μ λΉκ΅ν zero-shot μνμ΄λ€. BF16μλ κ°μ λ°°ν¬ κ°μ€μΉμ Transformers BF16 forward, GGUFμλ ν΄λΉ νμΌμ llama.cpp μ±μ κΈ°λ₯Ό μ¬μ©νλ€. λ€μ ν΄λμ€λ μ°Έ 705/1,404(50.2%)λ€. λ€μ νμ μ°Έ μμΈ‘ λΉλμ macro-F1μ ν¨κ» ν΄μνλ€. λ°μ΄ν° λΌμ΄μ μ€λ CC BY-SA 4.0μ΄λ©° μνμ νμ΅ λ°μ΄ν° μ€λ³΅ μ¬λΆλ λ―ΈνμΈμ΄λ€. [KoBEST κ³ μ revision](https://huggingface.co/datasets/skt/kobest_v1/tree/a5ea15e3ac77ed694b79f6204eb31889a2ba989f)
|
| 109 |
+
|
| 110 |
+
| νμ | BoolQ μ°Έ μμΈ‘ μ | BF16 λλΉ λ§μβνλ¦Ό / νλ¦Όβλ§μ | νκ΅μ΄ / μμ΄ μ΄μΌκΈ° λ°λ³΅ 4-gram | νκ΅μ΄ / μμ΄μμ μ₯λ©΄μ μ κ°ν μΆλ ₯ (κ° 4κ°) |
|
| 111 |
+
|---|---:|---:|---:|---:|
|
| 112 |
+
| BF16 | 1299 | 0 / 0 | 53.7% / 58.6% | 1 / 1 |
|
| 113 |
+
| Q8_0 | 1302 | 5 / 4 | 42.6% / 61.9% | 2 / 0 |
|
| 114 |
+
| Q4_K_M | 1312 | 19 / 22 | 66.4% / 68.4% | 0 / 0 |
|
| 115 |
+
|
| 116 |
+
λ°λ³΅λ₯ μ 64ν ν° μμ±λ¬Όμ λ€μ ν ν°νν΄ μ²μ λ±μ₯ν μ΄νμ μ€λ³΅ 4-gram μλ₯Ό μ 체 4-gram μλ‘ λλ κ°μ΄λ€. μ΄μ΄μ°κΈ° 8κ°λ νμλͺ
μ κ°λ¦¬κ³ κ°μ μ
λ ₯Β·κ°μ μΆλ ₯μ ν©μ³ λ¨μΌ AI νλ
μκ° 0β2μ μΌλ‘ λΆλ₯νλ€. νμλ μλ‘ λ€λ₯Έ μΈλΆ λ΄μ©μΌλ‘ μ₯λ©΄μ΄ μ κ°λ 2μ μΆλ ₯ μλ₯Ό νμνλ€. μλ₯Ό λ€μ΄ λΉκ° κ·ΈμΉ μ΄λμ₯μ΄λΌλ μμ΄ μ
λ ₯μμλ μλ λ¬μ¬κ° κ°μ λ¬Έμ₯μΌλ‘ λ°λ³΅λλ€. μ¬λ νκ°μ κ° μΌμΉλλ μΈ‘μ νμ§ μμλ€.
|
| 117 |
+
|
| 118 |
+
### μΈμ΄λ³ holdout PPL
|
| 119 |
+
|
| 120 |
+
κ²μ¦λ 1,466λ¬Έμμ λμΌ ν ν° IDΒ·λ§μ€ν¬Β·λ¬Έμλ³ NLL ν©μ μ¬μ¬μ©νλ€. νκ΅μ΄ 663,905κ°, μμ΄ 705,634κ° νμ μ΄λ€. κ΄νΈ μμ BF16 λλΉ μλ μμΉλ₯ μ 95% ꡬκ°μ΄λ€. λ¬Έμλ₯Ό μΈμ΄λ³λ‘ μ§μ§μ΄ 2,000ν 볡μ μΆμΆνκ³ λ§€λ² νμ ν ν° κ°μ€ νκ· NLLλ‘ PPLμ ꡬνλ€.
|
| 121 |
+
|
| 122 |
+
| νμ | νκ΅μ΄ PPL | νκ΅μ΄ μμΉλ₯ (95% ꡬκ°) | μμ΄ PPL | μμ΄ μμΉλ₯ (95% ꡬκ°) |
|
| 123 |
+
|---|---:|---:|---:|---:|
|
| 124 |
+
| BF16 | 9.08981 | κΈ°μ€ | 10.21574 | κΈ°μ€ |
|
| 125 |
+
| Q8_0 | 9.09086 | +0.012% (+0.002 ~ +0.022%) | 10.21729 | +0.015% (+0.005 ~ +0.025%) |
|
| 126 |
+
| Q4_K_M | 9.19630 | +1.172% (+1.110 ~ +1.235%) | 10.31774 | +0.999% (+0.952 ~ +1.042%) |
|
| 127 |
+
|
| 128 |
+
### λ¬Έλ§₯ κΈΈμ΄μ λ©λͺ¨λ¦¬
|
| 129 |
+
|
| 130 |
+
μλ²λ₯Ό μμ΄ν λ€ μ
λ ₯ κΈΈμ΄Β·μΆλ ₯ μνλ³λ‘ 5νμ© μμλ₯Ό μμλ€. μ±λ₯ μ
λ ₯μ ν λ¨μ΄λΉ ν ν ν°μΈ κ³ μ μ΄νμμ seedλ‘ λ½μμΌλ©° νκ΅μ΄ νμ§ μν μ
λ ₯κ³Ό λ³λλ€. κ° νμμ κ°μ ν둬ννΈλ₯Ό μ μ‘νκ³ μ±κ³΅ μλ΅μ μ
λ ₯Β·μΆλ ₯ ν ν° μλ₯Ό μ μ νμΈνλ€. μλλ μΆλ ₯ 128ν ν°, λ¨μΌ μμ²μ μ€μκ°μ΄λ€. 첫 ν
μ€νΈκ° λμ°©ν λκΉμ§κ° TTFTμ΄κ³ μμ± μλλ μ΄ν λ¨μ ν ν°μ μλ£ν λκΉμ§μ μλλ€. μ΅κ³ VRAMμ μμ² κ²½οΏ½οΏ½μ 0.5μ΄ κ°κ²©μμ κ΄μΈ‘ν μ₯μΉ μ 체 μ¬μ©λμ΄λ€.
|
| 131 |
+
|
| 132 |
+
| μ
λ ₯ ν ν° | νμ | μ±κ³΅ / μλ | TTFT (ms) | μ 체 μ§μ° (μ΄) | μμ± ν ν°/μ΄ | μ΅κ³ VRAM (MiB) |
|
| 133 |
+
|---:|---|---:|---:|---:|---:|---:|
|
| 134 |
+
| 128 | BF16 | 5/5 | 19.7 | 1.280 | 100.8 | 8805.9 |
|
| 135 |
+
| 128 | Q8_0 | 5/5 | 65.9 | 0.445 | 335.2 | 5145.9 |
|
| 136 |
+
| 128 | Q4_K_M | 5/5 | 65.5 | 0.418 | 361.2 | 3851.9 |
|
| 137 |
+
| 1,024 | BF16 | 5/5 | 26.2 | 1.290 | 100.5 | 8805.9 |
|
| 138 |
+
| 1,024 | Q8_0 | 5/5 | 112.7 | 0.490 | 335.4 | 5145.9 |
|
| 139 |
+
| 1,024 | Q4_K_M | 5/5 | 131.5 | 0.483 | 362.0 | 3851.9 |
|
| 140 |
+
| 3,072 | BF16 | 5/5 | 57.6 | 1.321 | 100.5 | 8805.9 |
|
| 141 |
+
| 3,072 | Q8_0 | 5/5 | 265.8 | 0.663 | 320.2 | 5145.9 |
|
| 142 |
+
| 3,072 | Q4_K_M | 5/5 | 292.1 | 0.666 | 343.2 | 3851.9 |
|
| 143 |
+
| 3,968 | BF16 | 5/5 | 72.3 | 1.334 | 100.8 | 8805.9 |
|
| 144 |
+
| 3,968 | Q8_0 | 5/5 | 362.4 | 0.753 | 324.8 | 5145.9 |
|
| 145 |
+
| 3,968 | Q4_K_M | 4/5 | 404.1 | 0.768 | 349.0 | 3851.9 |
|
| 146 |
+
|
| 147 |
+
Q4_K_Mμ 3,968+128 쑰건μμ ν μμ²μ΄ λ°λ³΅ ν ν° μ νμΌλ‘ μ’
λ£λμΌλ©° κ°μ μ
λ ₯μ λ³λ μ¬νμμλ κ°μ μ€λ₯κ° λ°μνλ€. μ΄ νμ μ§μ°Β·μλλ μ±κ³΅ν 4νλ‘ κ³μ°νκ³ VRAMμλ μ€ν¨ μμ²λ ν¬ν¨νλ€. 4,096μ μ
λ ₯κ³Ό μΆλ ₯μ ν©ν νλλ€. μλ²κ° μμ΄λ μνμμ KV μΊμμ μ€ν λ²νΌλ₯Ό μ¬μ¬μ©νλ―λ‘ μ
λ ₯ κΈΈμ΄κ° λ°λμ΄λ κ΄μΈ‘ μ΅κ³ VRAMμ΄ λΉμ·ν μ μλ€.
|
| 148 |
+
|
| 149 |
+
### λμ μμ²κ³Ό νμ μ ν
|
| 150 |
+
|
| 151 |
+
μ
λ ₯ 1,024ν ν°Β·μΆλ ₯ 32ν ν°μμ λμμ 1Β·2Β·4κ° μμ²μ 보λλ€. νμ μ²λ¦¬λμ μμ² λ¬Άμ μ 체μ μΆλ ₯ ν ν° μλ₯Ό 첫 μμ² μμλΆν° λ§μ§λ§ μλ£κΉμ§μ μκ°μΌλ‘ λλ κ°μ μ€μκ°μ΄λ€. Ollamaλ μλ² λ³λ ¬ μ¬λ‘― 1κ°λ‘ ν λκΈ°λ₯Ό ν¬ν¨νλ€. vLLMμ μ΅λ 2κ° μνμ€λ₯Ό ν¨κ» μ²λ¦¬νλ€.
|
| 152 |
+
|
| 153 |
+
| νμ | λμ 1 / 2 / 4 μ 체 μ²λ¦¬λ (ν ν°/μ΄) | λμ 4 μμ² μ§μ° μ€μκ° / κ΄μΈ‘ μ΅λκ° (μ΄) |
|
| 154 |
+
|---|---:|---:|
|
| 155 |
+
| BF16 | 95.0 / 179.3 / 185.1 | 0.517 / 0.690 |
|
| 156 |
+
| Q8_0 | 102.2 / 143.9 / 140.8 | 0.586 / 0.990 |
|
| 157 |
+
| Q4_K_M | 102.2 / 144.9 / 142.0 | 0.584 / 0.975 |
|
| 158 |
+
|
| 159 |
+
νμλ³ κΆμ₯ μμ μ€μ μ λ€μκ³Ό κ°λ€. μλ λΉκ΅μλ κ°μ€μΉ μ λ°λλΏ μλλΌ λ°νμ·컀λΒ·μΊμ μ€μ μ°¨μ΄λ ν¬ν¨λλ€.
|
| 160 |
+
|
| 161 |
+
- **BF16 Β· μ νλ λΉκ΅ κΈ°μ€, μ§§μ 첫 μλ΅ λλ λμ μμ².** H200 1μ₯μ `--dtype bfloat16 --load-format safetensors --tensor-parallel-size 1 --max-model-len 4096 --max-num-seqs 2 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.25 --num-gpu-blocks-override 512 --enforce-eager --no-enable-prefix-caching --generation-config vllm --seed 47000`μ μ¬μ©νλ€. `VLLM_ATTENTION_BACKEND=FLASH_ATTN`κ³Ό `VLLM_USE_FLASHINFER_SAMPLER=0`μ μ μ©νλ€. `/v1/completions`λ‘ `temperature=0`κ³Ό λͺ©μ μ λ§λ `max_tokens`λ₯Ό μ λ¬νλ€.
|
| 162 |
+
- **Q8_0 Β· Ollamaμ νμ§ μ°μ κΈ°λ³Έκ°.** BF16μ κ°κΉμ΄ PPLμ΄ νμν κ²½μ° `hf.co/pathcosmos/JANSORI:Q8_0`μ μ ννλ€. `/api/generate`μμ `raw=true`, `stream=true`, `num_ctx=4096`, `num_predict=32`(μ§§μ λ΅) λλ `128`(μ΄μ΄μ°κΈ°), `temperature=0`, `repeat_penalty=1`, `seed=47000`μ μ¬μ©νλ€. μ΄ H200 μνμ `num_gpu=99`, `num_thread=4`, `OLLAMA_NUM_PARALLEL=1`μ΄μλ€.
|
| 163 |
+
- **Q4_K_M Β· λ©λͺ¨λ¦¬ μ°μ .** κ°μ Ollama μ΅μ
μ νκ·Έλ₯Ό `Q4_K_M`μΌλ‘ λ°κΎΌλ€. κΈ΄ μ
λ ₯μμλ μ΅λ κΈΈμ΄λ³΄λ€ μ¬μ λ₯Ό λκ³ μμ± μνμ λͺ
μνλ€. μ΄λ² 3,072+128 μνμ 5/5 μλ£λλ€. Q4κ° λͺ¨λ μ
λ ₯ κΈΈμ΄μμ κ°μ₯ λΉ λ₯΄λ€λ 보μ₯μ μλ€.
|
| 164 |
+
- **μ°μΒ·μ½λ.** μ§§μ μ λ΅μ μꡬνλ λ¨μ μ°μμ λ λ¨κ³ μ°μ κ²°κ³Όλ₯Ό ꡬλΆνλ€. κ°λ¨ν Python ν¨μ μμ±μλ ν¨μ μμ λ κ°μ λͺ©ν ν¨μμ `return` μ λμ΄λ₯Ό μ μ©ν λ€ λ³λ ν
μ€νΈλ‘ κ²μ¦νλ€. μ΄ κ²°κ³Όλ μμ νλ‘κ·Έλ¨ μμ± λ₯λ ₯μ νκ°νμ§ μλλ€.
|
| 165 |
+
### H200 λ μ₯μμ μμ²μ λ³νν κ²½μ°
|
| 166 |
+
|
| 167 |
+
GPU 0μ BF16, GPU 1μ Ollama ν νμμ μ μ¬ν λ€ 1,024ν ν° μ
λ ₯κ³Ό 32ν ν° μΆλ ₯μ λ μλ²μ κ°κ° μμ²νλ€. νμ μ‘°ν©λ³λ‘ μλ‘ λ€λ₯Έ μ
λ ₯ 5κ°λ₯Ό μ¬μ©νλ€. κ°μ μ
λ ₯μ μμ°¨μ λ³νμ λͺ¨λ μ¬μ©νκ³ μ΄λ μͺ½μ λ¨Όμ μΈ‘μ ν μ§λ κ΅λνλ€. κ° μΈ‘μ μ λ³λ 1,024ν ν° μμ²μΌλ‘ μ§μ μ
λ ₯ μΊμκ° κ·Έλλ‘ μ¬μ¬μ©λλ κ²½μ°λ₯Ό μ€μλ€. μλ² μμΒ·λͺ¨λΈ λ‘λ©μ μ΄ νμμ μ μΈνλ€.
|
| 168 |
+
|
| 169 |
+
| λμ μμ£Ό νμ | μμ°¨ / λ³ν λ¬Άμ μ§μ° μ€μκ° (μ΄) | λ³ν μμ² μ€μ²© μ€μκ° (μ΄) | λ μΆλ ₯ λͺ¨λ λμΌν μ
λ ₯ | λμΌ μΆλ ₯ μλ§μ μμ°¨ / λ³ν μ€μκ° (μ΄) | μ΅κ³ λμ ν©μ° VRAM, μμ°¨ / λ³ν (MiB) |
|
| 170 |
+
|---|---:|---:|---:|---:|---:|
|
| 171 |
+
| BF16 + Q8_0 | 0.568 / 0.323 | 0.240 | 4/5 | 0.551 / 0.324 | 13889.8 / 13889.8 |
|
| 172 |
+
| BF16 + Q4_K_M | 0.556 / 0.324 | 0.235 | 3/5 | 0.559 / 0.324 | 12595.8 / 12595.8 |
|
| 173 |
+
|
| 174 |
+
μμͺ½ λͺ¨λΈμ΄ μμ£Όν μνμ΄λ―λ‘ μμ°¨ μμ²λ λ λͺ¨λΈμ VRAMμ μ μ§νλ€. λμΌ μΆλ ₯ μ μκ° μ μ λλ μλ μ°¨μ΄λ₯Ό λμΌ νμ§μμ 보μ₯λλ κ°μ μΌλ‘ ν΄μνκΈ° μ΄λ ΅λ€. κΈ°μ‘΄ κ³΅κ° κ²μ¦μ μλ² μμλΆν° μ’
λ£κΉμ§ μ° μμΉλ λ€λ₯Έ μ§νλ€.
|
| 175 |
+
|
| 176 |
## νμ΅ λ°μ΄ν°
|
| 177 |
|
| 178 |
νκ΅μ΄ μ€μ¬ 3B Dense Base λͺ¨λΈμ 47,000μ€ν
μμ λμ 98,566,144,000ν ν°μ νμ΅νλ€. μλ μμΉλ νμ΅ μ λ°λ³΅ λ
ΈμΆμ ν¬ν¨ν μΆμ²λ³ μ
λ ₯ ν ν° μλ€.
|
SHA256SUMS
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
e656e13897e3ef9c16a5bd506b16cc1cd557212bd0c55e10994be2e70e01bf47 .gitattributes
|
| 2 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 3 |
057f847c2bb770535d520ca66c61eff6de69f8095385cf1d5f69f48b471f96d6 NOTICE
|
| 4 |
-
|
| 5 |
b94eb89d477bb22489a5730b77ea4cdf92af9c63dbddc3dc5cd7967e87d4749e config.json
|
| 6 |
21f8c90ade582967a8acb1991f73bea5f62757ae2d430500da41ba458f0f52a0 dense-3b-47k-Q4_K_M.gguf
|
| 7 |
eb7053112886dab60d6012d4635dc2c7cdd07c902317e202a91ba36da33fed41 dense-3b-47k-Q8_0.gguf
|
|
|
|
| 1 |
e656e13897e3ef9c16a5bd506b16cc1cd557212bd0c55e10994be2e70e01bf47 .gitattributes
|
| 2 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 3 |
057f847c2bb770535d520ca66c61eff6de69f8095385cf1d5f69f48b471f96d6 NOTICE
|
| 4 |
+
57b502f333dc02f6fb6064a6cae1626876501ebc7f6a6f9fd950da9bb3d15d91 README.md
|
| 5 |
b94eb89d477bb22489a5730b77ea4cdf92af9c63dbddc3dc5cd7967e87d4749e config.json
|
| 6 |
21f8c90ade582967a8acb1991f73bea5f62757ae2d430500da41ba458f0f52a0 dense-3b-47k-Q4_K_M.gguf
|
| 7 |
eb7053112886dab60d6012d4635dc2c7cdd07c902317e202a91ba36da33fed41 dense-3b-47k-Q8_0.gguf
|