Instructions to use pathcosmos/JANSORI with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use pathcosmos/JANSORI with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="pathcosmos/JANSORI")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("pathcosmos/JANSORI") model = AutoModelForCausalLM.from_pretrained("pathcosmos/JANSORI", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use pathcosmos/JANSORI with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf pathcosmos/JANSORI:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf pathcosmos/JANSORI:Q4_K_M
Use Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use pathcosmos/JANSORI with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "pathcosmos/JANSORI" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- SGLang
How to use pathcosmos/JANSORI with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "pathcosmos/JANSORI" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "pathcosmos/JANSORI" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pathcosmos/JANSORI", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Ollama
How to use pathcosmos/JANSORI with Ollama:
ollama run hf.co/pathcosmos/JANSORI:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use pathcosmos/JANSORI with Docker Model Runner:
docker model run hf.co/pathcosmos/JANSORI:Q4_K_M
- Lemonade
How to use pathcosmos/JANSORI with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull pathcosmos/JANSORI:Q4_K_M
Run and chat with the model
lemonade run user.JANSORI-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pathcosmos/JANSORI:Q4_K_M# Run inference directly in the terminal:
llama cli -hf pathcosmos/JANSORI:Q4_K_MBuild from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf pathcosmos/JANSORI:Q4_K_M# Run inference directly in the terminal:
./build/bin/llama cli -hf pathcosmos/JANSORI:Q4_K_MUse Docker
docker model run hf.co/pathcosmos/JANSORI:Q4_K_MJANSORI Β· νκ΅μ΄ μ€μ¬ 3B Dense Base
νκ΅μ΄μ μμ΄ λ¬Έμ₯μ μ΄μ΄ μ°λ μ¬μ νμ΅ λͺ¨λΈμ΄λ€. μ΄ν ν¬κΈ°λ 65,536, κ²μ¦ν λ¬Έλ§₯μ 4,096ν ν°μ΄λ€. λνμ© μν ν ν° μμ΄ μΌλ° ν μ€νΈλ₯Ό μ λ ₯νλ€.
vLLMμΌλ‘ BF16 μ€ν
vLLM 0.11.2μμ H200 ν μ₯μ μ¬μ©νλ€. μλ λͺ λ Ήμ λ€μ΄λ‘λ μμ μ commit SHAλ₯Ό μ‘°νν΄ κ°μ€μΉμ ν ν¬λμ΄μ λ₯Ό κ°μ revisionμΌλ‘ κ³ μ νλ€. μ΄μ λ²μ μ μ¬νν λλ κΈ°λ‘ν΄ λ SHAλ₯Ό REVμ μ§μ νλ€.
REV=$(python -c 'from huggingface_hub import HfApi; print(HfApi().model_info("pathcosmos/JANSORI").sha)')
CUDA_VISIBLE_DEVICES=0 vllm serve pathcosmos/JANSORI --revision "$REV" --tokenizer-revision "$REV" --load-format safetensors --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 4096 --host 127.0.0.1
λ€λ₯Έ μ Έμμ νκ΅μ΄ μ΄μ΄μ°κΈ°λ₯Ό μμ²νλ€.
curl -s http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' -d '{"model":"pathcosmos/JANSORI","prompt":"νκ΅μ μλλ","max_tokens":32,"temperature":0}'
Ollamaλ‘ Q8_0 λλ Q4_K_M μ€ν
Ollama 0.40.16μ μ¬μ©νλ€. H200 ν μ₯μμ νκ΅μ΄ μ΄μ΄μ°κΈ°λ₯Ό μνν λλ μ¬μ©ν μμν νλλ₯Ό μ ννλ€.
ollama run hf.co/pathcosmos/JANSORI:Q8_0 "νκ΅μ μλλ"
ollama run hf.co/pathcosmos/JANSORI:Q4_K_M "νκ΅μ μλλ"
루νΈμ templateμ μ
λ ₯ ν둬ννΈλ₯Ό κ·Έλλ‘ μ λ¬νλ€. μμν νκ·Έλ λ³κ²½λ μ μλ€. μ€ν λΉμ μ μ₯μ commitκ³Ό λ΄λ €λ°μ GGUFμ SHA-256μ ν¨κ» κΈ°λ‘νλ©΄ μ¬μ©ν κ°μ€μΉλ₯Ό νμΈν μ μλ€.
μΈ νμμ κ°μ λ°°ν¬ commitμ λ€μ΄ μλ€. μ΄ μ μ₯μμ BF16μμ κ³ μ λ° GGUF(SHA-256 c69c8a77121940e0c1f06f668ba0a80e57edba1afaac777cae3ea3f923d69b7b)λ₯Ό ν λ² μμ±ν λ€ Q8_0κ³Ό Q4_K_MμΌλ‘ κ°κ° μμννλ€. BF16 shardμ κ° GGUFμ SHA-256μ κ°μ commitμ SHA256SUMSμ μλ€.
μ€μ 3B νκ°
κ°μ holdoutμ νκ΅μ΄ 663,905κ°Β·μμ΄ 705,634κ° νμ μ μ±μ νλ€. μλμ VRAMμ H200μμ 첫 μμ°¨ νμ°¨μ κ³ μ ν둬ννΈ 6κ°(μ΅λ 64ν ν°)λ‘ μΈ‘μ νλ€.
| νμ | ν¨ν€μ§ λ°μ΄νΈΒΉ | νκ΅μ΄ PPL | μμ΄ PPL | μμ² κΈ°μ€ ν ν°/μ΄ | μμ± κ΅¬κ° GPU μ΅κ³ (MiB) |
|---|---|---|---|---|---|
| BF16 | 6,448,116,613 | 9.089808 | 10.215738 | 100.27 | 8805.9 |
| Q8_0 | 3,426,185,184 | 9.090860 | 10.217286 | 14.49 | 5055.9 |
| Q4_K_M | 1,969,663,968 | 9.196303 | 10.317744 | 9.95 | 3761.9 |
ΒΉ BF16μ κ°μ€μΉΒ·μ€μ Β·ν ν¬λμ΄μ ν©κ³, GGUFλ κ° κ°μ€μΉ νμΌ ν¬κΈ°λ€.
native νκ΅μ΄ PPLμ 9.087589, μμ΄λ 10.213156μ΄μλ€. νμ€ HFμ μλ native BF16 μμΉ λΉκ΅μμ 4,096ν ν° λ‘μ§ 268,435,456κ° μ€ 297κ°κ° νμ©λμ μ΄κ³Όνλ€. RoPE λ°μ¬λ¦Όκ³Ό attention νΈμΆ λ°©μμ λ§μΆ native λμ‘°μμλ 8κ° μ λ ₯μ μ 체 λ‘μ§μ΄ κ°μκ³ μΊμ λΉκ΅κ° ν΅κ³Όνλ€.
κ³ μ ν둬ννΈμμ κ΄μ°°λ νκ³
- λΉμ κ΅΄μ μ€λͺ μμ μΈ νμ λͺ¨λ λ§€μ§ κ²½κ³μ κ΄ν 문ꡬλ₯Ό λ°λ³΅νκ³ κ΅΄μ μμΈμ μ€λͺ νμ§ λͺ»νλ€.
- μ¬κ³Ό 12κ°λ₯Ό 3λͺ μκ² λλλ νκ΅μ΄ λ¬Έμ μμ μΈ νμ λͺ¨λ 12κ°λΌκ³ λ΅νλ€. μ λ΅μ 4κ°λ€.
- νκ΅μ΄ add ν¨μμ 첫 λ°νμμ λͺ¨λ a + bμλ€. BF16Β·Q8μ λ²νΈ μ£Όμμ λ°λ³΅νκ³ , Q4λ λ μΈμ ν¨μμ μΈ κ° μ΄μμ μΈμλ₯Ό λκΈ°λ νΈμΆμ μΆκ°νλ€.
- μμ΄ μ΄μΌκΈ°μμλ BF16Β·Q8μ΄ κ°λν κ³ΌλΆλ₯Ό, Q4κ° λΆμ ν λ¨νΈκ³Ό μ¬λ Saraλ₯Ό λ±μ₯μμΌ°λ€. μΈ μΆλ ₯ λͺ¨λ ν λ¬Έμ₯μ΄λ ννμ λ°λ³΅νλ€.
- μμ΄ λ°©μ μμμ BF16Β·Q8μ μ²μ 4λΌκ³ μ΄ λ€ νμ΄μμ 5λ‘ μ μ νλ€. Q4λ 첫 μλΆν° 12 β 7 = 5λ‘ κ³μ°νλ€.
- μ§μ νλ³ ν¨μμμ μΈ νμ λͺ¨λ λ€μ¬μ°κΈ°λ₯Ό μμλ€. BF16Β·Q8μ μ§μ λΆκΈ°μ Trueλ₯Ό μΌκ³ Q4λ Falseλ₯Ό μ¨μ νμ λ λ€μ§νλ€.
μ΄μ΄μ°κΈ° μμ
μ λ ₯
νκ΅μ΄λ‘ λΉμ΄ κ΅΄μ λλ μ΄μ λ₯Ό κ°λ¨ν μ€λͺ
νλ©΄
μμ± (BF16)
λ€μκ³Ό κ°λ€.
1. λΉμ κ΅΄μ
λΉμ νλμ΄λ€. νλμ λ§€μ§μ κ²½κ³μμ μ§λνλ νλμ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€. νλμ κ²½κ³λ λ§€μ§μ κ²½κ³μ΄λ€.
μ λ ₯
Once upon a time, in a quiet village
μμ± (BF16)
, there lived a poor old woman. She was a widow, and she had no children. She was very old, and she was very poor. She had no money, and she had no food. She was very lonely, and she was very poor. She was very old, and she was very poor. She
H200μμ μΈ‘μ ν νμ§κ³Ό μ€ν μ€μ
μ΄ κ²°κ³Όλ 47,000μ€ν μ JANSORI λ°°ν¬ μ»€λ°μ μ μ©νλ€. BF16μ vLLM 0.11.2, Q8_0Β·Q4_K_Mμ Ollama 0.40.16μΌλ‘ κ°μ 물리 H200 ν μ₯μμ κ°κ° μ€ννλ€. μ΄μ λ°°ν¬ κ²μ¦μ 6κ° κ³ μ μ λ ₯μ νμλ³λ‘ 6/6 μ¬νλλ€.
νκ΅μ΄Β·μμ΄Β·μνΒ·μ½λ
μ체 μμ± μνμ νμλΉ 98κ° μ λ ₯μ΄λ€. μ§§μ μ¬μ€ μ§λ¬Έ 24κ°, μμ λ ν΄ 16κ°, μ°μ 32κ°, ν¨μ 12κ°, μ΄μΌκΈ° μ΄μ΄μ°κΈ° 8κ°, νκ· μ λ ₯ 6κ°λ₯Ό μ¬μ©νλ€. μΈ νμμ μ λ ₯κ³Ό μμ± μνμ κ°κ² νλ€. μ¬μ€Β·λ ν΄λ 첫 λ΅μ μ κ·νν΄ νμ© λ΅κ³Ό λΉκ΅νλ€. μ°μλ μꡬν μ μκ° μ²« μ€μ λ΅μΌλ‘ λμλμ§ μ±μ νλ€. λ¬Έν μκ° μμΌλ―λ‘ ν λ¬Έν μ°¨μ΄λ‘ νμμ μ°μ΄μ μ νκΈ° μ΄λ ΅λ€.
| νμ | νκ΅μ΄Β·μμ΄ μ¬μ€ (κ° 12) | νκ΅μ΄Β·μμ΄ λ ν΄ (κ° 8) | κΈ°λ³ΈΒ·λ λ¨κ³ μ°μ (κ° 16) | ν¨μ μ 체 ν΅κ³Ό, μ€λͺ λ§ / μμ+λ°νλ¬Έ (κ° 12) | BoolQ μ λ΅ / 1,404 | BoolQ macro-F1 |
|---|---|---|---|---|---|---|
| BF16 | 10 Β· 10 | 7 Β· 7 | 15 Β· 0 | 0 / 9 | 730 (52.0%) | 0.413 |
| Q8_0 | 10 Β· 10 | 8 Β· 7 | 15 Β· 0 | 0 / 9 | 729 (51.9%) | 0.410 |
| Q4_K_M | 10 Β· 10 | 7 Β· 7 | 14 Β· 1 | 0 / 9 | 733 (52.2%) | 0.410 |
λ μμ ν¨μμ λμ ν¨μμ returnκΉμ§ μ
λ ₯ν 쑰건μ κ°μ 12κ° ν¨μμ ννμ μμ± λ¬Έμ λ€. λͺ¨λΈμ΄ λλ¨Έμ§ λ³Έλ¬Έ μ 체λ₯Ό μμ±νλ μ‘°κ±΄λ³΄λ€ μ½λ€. μΆλ ₯μ λ€μ¬μ°κΈ°λ₯Ό μ리νμ§ μκ³ μ²« ν¨μλ§ μΆμΆν΄ νμ±νκ³ , ν¨μλΉ 4κ° λ
립 μ¬λ‘μ κ°κ³Ό λ°ν νμ
μ κ²μ¬νλ€. μΈ νμ λͺ¨λ κ°μ₯ ν° ν¬μ λ₯λ₯Ό μ½λΌλ¦¬λΌκ³ λ΅νλ€. μ°μ μ΄ μ λ΅μ λͺ¨λ 15/32μμ§λ§ μ λ΅μ λΈ λ¬Ένμ΄ μΌλΆ λ¬λλ€.
νκ΅μ΄ KoBEST BoolQλ ν μ€νΈ 1,404κ° μ 체μμ μ§λ¬ΈΒ·μ§λ¬Έ λ€ λ¨μΌ ν ν°μΈ β κ±°μ§βΒ·β μ°Έβμ μ‘°κ±΄λΆ λ‘κ·Ένλ₯ μ λΉκ΅ν zero-shot μνμ΄λ€. BF16μλ κ°μ λ°°ν¬ κ°μ€μΉμ Transformers BF16 forward, GGUFμλ ν΄λΉ νμΌμ llama.cpp μ±μ κΈ°λ₯Ό μ¬μ©νλ€. λ€μ ν΄λμ€λ μ°Έ 705/1,404(50.2%)λ€. λ€μ νμ μ°Έ μμΈ‘ λΉλμ macro-F1μ ν¨κ» ν΄μνλ€. λ°μ΄ν° λΌμ΄μ μ€λ CC BY-SA 4.0μ΄λ©° μνμ νμ΅ λ°μ΄ν° μ€λ³΅ μ¬λΆλ λ―ΈνμΈμ΄λ€. KoBEST κ³ μ revision
| νμ | BoolQ μ°Έ μμΈ‘ μ | BF16 λλΉ λ§μβνλ¦Ό / νλ¦Όβλ§μ | νκ΅μ΄ / μμ΄ μ΄μΌκΈ° λ°λ³΅ 4-gram | νκ΅μ΄ / μμ΄μμ μ₯λ©΄μ μ κ°ν μΆλ ₯ (κ° 4κ°) |
|---|---|---|---|---|
| BF16 | 1299 | 0 / 0 | 53.7% / 58.6% | 1 / 1 |
| Q8_0 | 1302 | 5 / 4 | 42.6% / 61.9% | 2 / 0 |
| Q4_K_M | 1312 | 19 / 22 | 66.4% / 68.4% | 0 / 0 |
λ°λ³΅λ₯ μ 64ν ν° μμ±λ¬Όμ λ€μ ν ν°νν΄ μ²μ λ±μ₯ν μ΄νμ μ€λ³΅ 4-gram μλ₯Ό μ 체 4-gram μλ‘ λλ κ°μ΄λ€. μ΄μ΄μ°κΈ° 8κ°λ νμλͺ μ κ°λ¦¬κ³ κ°μ μ λ ₯Β·κ°μ μΆλ ₯μ ν©μ³ λ¨μΌ AI νλ μκ° 0β2μ μΌλ‘ λΆλ₯νλ€. νμλ μλ‘ λ€λ₯Έ μΈλΆ λ΄μ©μΌλ‘ μ₯λ©΄μ΄ μ κ°λ 2μ μΆλ ₯ μλ₯Ό νμνλ€. μλ₯Ό λ€μ΄ λΉκ° κ·ΈμΉ μ΄λμ₯μ΄λΌλ μμ΄ μ λ ₯μμλ μλ λ¬μ¬κ° κ°μ λ¬Έμ₯μΌλ‘ λ°λ³΅λλ€. μ¬λ νκ°μ κ° μΌμΉλλ μΈ‘μ νμ§ μμλ€.
μΈμ΄λ³ holdout PPL
κ²μ¦λ 1,466λ¬Έμμ λμΌ ν ν° IDΒ·λ§μ€ν¬Β·λ¬Έμλ³ NLL ν©μ μ¬μ¬μ©νλ€. νκ΅μ΄ 663,905κ°, μμ΄ 705,634κ° νμ μ΄λ€. κ΄νΈ μμ BF16 λλΉ μλ μμΉλ₯ μ 95% ꡬκ°μ΄λ€. λ¬Έμλ₯Ό μΈμ΄λ³λ‘ μ§μ§μ΄ 2,000ν 볡μ μΆμΆνκ³ λ§€λ² νμ ν ν° κ°μ€ νκ· NLLλ‘ PPLμ ꡬνλ€.
| νμ | νκ΅μ΄ PPL | νκ΅μ΄ μμΉλ₯ (95% ꡬκ°) | μμ΄ PPL | μμ΄ μμΉλ₯ (95% ꡬκ°) |
|---|---|---|---|---|
| BF16 | 9.08981 | κΈ°μ€ | 10.21574 | κΈ°μ€ |
| Q8_0 | 9.09086 | +0.012% (+0.002 ~ +0.022%) | 10.21729 | +0.015% (+0.005 ~ +0.025%) |
| Q4_K_M | 9.19630 | +1.172% (+1.110 ~ +1.235%) | 10.31774 | +0.999% (+0.952 ~ +1.042%) |
λ¬Έλ§₯ κΈΈμ΄μ λ©λͺ¨λ¦¬
μλ²λ₯Ό μμ΄ν λ€ μ λ ₯ κΈΈμ΄Β·μΆλ ₯ μνλ³λ‘ 5νμ© μμλ₯Ό μμλ€. μ±λ₯ μ λ ₯μ ν λ¨μ΄λΉ ν ν ν°μΈ κ³ μ μ΄νμμ seedλ‘ λ½μμΌλ©° νκ΅μ΄ νμ§ μν μ λ ₯κ³Ό λ³λλ€. κ° νμμ κ°μ ν둬ννΈλ₯Ό μ μ‘νκ³ μ±κ³΅ μλ΅μ μ λ ₯Β·μΆλ ₯ ν ν° μλ₯Ό μ μ νμΈνλ€. μλλ μΆλ ₯ 128ν ν°, λ¨μΌ μμ²μ μ€μκ°μ΄λ€. 첫 ν μ€νΈκ° λμ°©ν λκΉμ§κ° TTFTμ΄κ³ μμ± μλλ μ΄ν λ¨μ ν ν°μ μλ£ν λκΉμ§μ μλλ€. μ΅κ³ VRAMμ μμ² κ²½κ³μ 0.5μ΄ κ°κ²©μμ κ΄μΈ‘ν μ₯μΉ μ 체 μ¬μ©λμ΄λ€.
| μ λ ₯ ν ν° | νμ | μ±κ³΅ / μλ | TTFT (ms) | μ 체 μ§μ° (μ΄) | μμ± ν ν°/μ΄ | μ΅κ³ VRAM (MiB) |
|---|---|---|---|---|---|---|
| 128 | BF16 | 5/5 | 19.7 | 1.280 | 100.8 | 8805.9 |
| 128 | Q8_0 | 5/5 | 65.9 | 0.445 | 335.2 | 5145.9 |
| 128 | Q4_K_M | 5/5 | 65.5 | 0.418 | 361.2 | 3851.9 |
| 1,024 | BF16 | 5/5 | 26.2 | 1.290 | 100.5 | 8805.9 |
| 1,024 | Q8_0 | 5/5 | 112.7 | 0.490 | 335.4 | 5145.9 |
| 1,024 | Q4_K_M | 5/5 | 131.5 | 0.483 | 362.0 | 3851.9 |
| 3,072 | BF16 | 5/5 | 57.6 | 1.321 | 100.5 | 8805.9 |
| 3,072 | Q8_0 | 5/5 | 265.8 | 0.663 | 320.2 | 5145.9 |
| 3,072 | Q4_K_M | 5/5 | 292.1 | 0.666 | 343.2 | 3851.9 |
| 3,968 | BF16 | 5/5 | 72.3 | 1.334 | 100.8 | 8805.9 |
| 3,968 | Q8_0 | 5/5 | 362.4 | 0.753 | 324.8 | 5145.9 |
| 3,968 | Q4_K_M | 4/5 | 404.1 | 0.768 | 349.0 | 3851.9 |
Q4_K_Mμ 3,968+128 쑰건μμ ν μμ²μ΄ λ°λ³΅ ν ν° μ νμΌλ‘ μ’ λ£λμΌλ©° κ°μ μ λ ₯μ λ³λ μ¬νμμλ κ°μ μ€λ₯κ° λ°μνλ€. μ΄ νμ μ§μ°Β·μλλ μ±κ³΅ν 4νλ‘ κ³μ°νκ³ VRAMμλ μ€ν¨ μμ²λ ν¬ν¨νλ€. 4,096μ μ λ ₯κ³Ό μΆλ ₯μ ν©ν νλλ€. μλ²κ° μμ΄λ μνμμ KV μΊμμ μ€ν λ²νΌλ₯Ό μ¬μ¬μ©νλ―λ‘ μ λ ₯ κΈΈμ΄κ° λ°λμ΄λ κ΄μΈ‘ μ΅κ³ VRAMμ΄ λΉμ·ν μ μλ€.
λμ μμ²κ³Ό νμ μ ν
μ λ ₯ 1,024ν ν°Β·μΆλ ₯ 32ν ν°μμ λμμ 1Β·2Β·4κ° μμ²μ 보λλ€. νμ μ²λ¦¬λμ μμ² λ¬Άμ μ 체μ μΆλ ₯ ν ν° μλ₯Ό 첫 μμ² μμλΆν° λ§μ§λ§ μλ£κΉμ§μ μκ°μΌλ‘ λλ κ°μ μ€μκ°μ΄λ€. Ollamaλ μλ² λ³λ ¬ μ¬λ‘― 1κ°λ‘ ν λκΈ°λ₯Ό ν¬ν¨νλ€. vLLMμ μ΅λ 2κ° μνμ€λ₯Ό ν¨κ» μ²λ¦¬νλ€.
| νμ | λμ 1 / 2 / 4 μ 체 μ²λ¦¬λ (ν ν°/μ΄) | λμ 4 μμ² μ§μ° μ€μκ° / κ΄μΈ‘ μ΅λκ° (μ΄) |
|---|---|---|
| BF16 | 95.0 / 179.3 / 185.1 | 0.517 / 0.690 |
| Q8_0 | 102.2 / 143.9 / 140.8 | 0.586 / 0.990 |
| Q4_K_M | 102.2 / 144.9 / 142.0 | 0.584 / 0.975 |
νμλ³ κΆμ₯ μμ μ€μ μ λ€μκ³Ό κ°λ€. μλ λΉκ΅μλ κ°μ€μΉ μ λ°λλΏ μλλΌ λ°νμ·컀λΒ·μΊμ μ€μ μ°¨μ΄λ ν¬ν¨λλ€.
- BF16 Β· μ νλ λΉκ΅ κΈ°μ€, μ§§μ 첫 μλ΅ λλ λμ μμ². H200 1μ₯μ
--dtype bfloat16 --load-format safetensors --tensor-parallel-size 1 --max-model-len 4096 --max-num-seqs 2 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.25 --num-gpu-blocks-override 512 --enforce-eager --no-enable-prefix-caching --generation-config vllm --seed 47000μ μ¬μ©νλ€.VLLM_ATTENTION_BACKEND=FLASH_ATTNκ³ΌVLLM_USE_FLASHINFER_SAMPLER=0μ μ μ©νλ€./v1/completionsλ‘temperature=0κ³Ό λͺ©μ μ λ§λmax_tokensλ₯Ό μ λ¬νλ€. - Q8_0 Β· Ollamaμ νμ§ μ°μ κΈ°λ³Έκ°. BF16μ κ°κΉμ΄ PPLμ΄ νμν κ²½μ°
hf.co/pathcosmos/JANSORI:Q8_0μ μ ννλ€./api/generateμμraw=true,stream=true,num_ctx=4096,num_predict=32(μ§§μ λ΅) λλ128(μ΄μ΄μ°κΈ°),temperature=0,repeat_penalty=1,seed=47000μ μ¬μ©νλ€. μ΄ H200 μνμnum_gpu=99,num_thread=4,OLLAMA_NUM_PARALLEL=1μ΄μλ€. - Q4_K_M Β· λ©λͺ¨λ¦¬ μ°μ . κ°μ Ollama μ΅μ
μ νκ·Έλ₯Ό
Q4_K_MμΌλ‘ λ°κΎΌλ€. κΈ΄ μ λ ₯μμλ μ΅λ κΈΈμ΄λ³΄λ€ μ¬μ λ₯Ό λκ³ μμ± μνμ λͺ μνλ€. μ΄λ² 3,072+128 μνμ 5/5 μλ£λλ€. Q4κ° λͺ¨λ μ λ ₯ κΈΈμ΄μμ κ°μ₯ λΉ λ₯΄λ€λ 보μ₯μ μλ€. - μ°μΒ·μ½λ. μ§§μ μ λ΅μ μꡬνλ λ¨μ μ°μμ λ λ¨κ³ μ°μ κ²°κ³Όλ₯Ό ꡬλΆνλ€. κ°λ¨ν Python ν¨μ μμ±μλ ν¨μ μμ λ κ°μ λͺ©ν ν¨μμ
returnμ λμ΄λ₯Ό μ μ©ν λ€ λ³λ ν μ€νΈλ‘ κ²μ¦νλ€. μ΄ κ²°κ³Όλ μμ νλ‘κ·Έλ¨ μμ± λ₯λ ₯μ νκ°νμ§ μλλ€.
H200 λ μ₯μμ μμ²μ λ³νν κ²½μ°
GPU 0μ BF16, GPU 1μ Ollama ν νμμ μ μ¬ν λ€ 1,024ν ν° μ λ ₯κ³Ό 32ν ν° μΆλ ₯μ λ μλ²μ κ°κ° μμ²νλ€. νμ μ‘°ν©λ³λ‘ μλ‘ λ€λ₯Έ μ λ ₯ 5κ°λ₯Ό μ¬μ©νλ€. κ°μ μ λ ₯μ μμ°¨μ λ³νμ λͺ¨λ μ¬μ©νκ³ μ΄λ μͺ½μ λ¨Όμ μΈ‘μ ν μ§λ κ΅λνλ€. κ° μΈ‘μ μ λ³λ 1,024ν ν° μμ²μΌλ‘ μ§μ μ λ ₯ μΊμκ° κ·Έλλ‘ μ¬μ¬μ©λλ κ²½μ°λ₯Ό μ€μλ€. μλ² μμΒ·λͺ¨λΈ λ‘λ©μ μ΄ νμμ μ μΈνλ€.
| λμ μμ£Ό νμ | μμ°¨ / λ³ν λ¬Άμ μ§μ° μ€μκ° (μ΄) | λ³ν μμ² μ€μ²© μ€μκ° (μ΄) | λ μΆλ ₯ λͺ¨λ λμΌν μ λ ₯ | λμΌ μΆλ ₯ μλ§μ μμ°¨ / λ³ν μ€μκ° (μ΄) | μ΅κ³ λμ ν©μ° VRAM, μμ°¨ / λ³ν (MiB) |
|---|---|---|---|---|---|
| BF16 + Q8_0 | 0.568 / 0.323 | 0.240 | 4/5 | 0.551 / 0.324 | 13889.8 / 13889.8 |
| BF16 + Q4_K_M | 0.556 / 0.324 | 0.235 | 3/5 | 0.559 / 0.324 | 12595.8 / 12595.8 |
μμͺ½ λͺ¨λΈμ΄ μμ£Όν μνμ΄λ―λ‘ μμ°¨ μμ²λ λ λͺ¨λΈμ VRAMμ μ μ§νλ€. λμΌ μΆλ ₯ μ μκ° μ μ λλ μλ μ°¨μ΄λ₯Ό λμΌ νμ§μμ 보μ₯λλ κ°μ μΌλ‘ ν΄μνκΈ° μ΄λ ΅λ€. κΈ°μ‘΄ κ³΅κ° κ²μ¦μ μλ² μμλΆν° μ’ λ£κΉμ§ μ° μμΉλ λ€λ₯Έ μ§νλ€.
νμ΅ λ°μ΄ν°
νκ΅μ΄ μ€μ¬ 3B Dense Base λͺ¨λΈμ 47,000μ€ν μμ λμ 98,566,144,000ν ν°μ νμ΅νλ€. μλ μμΉλ νμ΅ μ λ°λ³΅ λ ΈμΆμ ν¬ν¨ν μΆμ²λ³ μ λ ₯ ν ν° μλ€.
| λ°μ΄ν°μ | μ¬μ© λ²μ | νμ΅ ν ν° | λΉμ€ |
|---|---|---|---|
| wikimedia/wikipedia | 2023-11-01 νκ΅μ΄ μν€λ°±κ³Ό λ€ν 3κ° μ€λ | 494,833,664 | 0.502% |
| HuggingFaceFW/fineweb-2 | νκ΅μ΄ kor_Hang 25κ° μ€λ | 29,038,194,688 | 29.461% |
| HPLT/HPLT2.0_cleaned | νκ΅μ΄ kor_Hang 464κ° μ€λ | 19,756,912,640 | 20.044% |
| HuggingFaceFW/fineweb-edu | 100BT νλ³Έμμ νμΌλͺ μ μ 56κ° μ€λ | 29,557,293,056 | 29.987% |
| open-web-math/open-web-math | 114κ° μ€λ | 9,861,763,072 | 10.005% |
| codeparrot/github-code | 1,126κ° μ€ κ· λ± κ°κ²©μΌλ‘ μ μ ν 64κ° μ€λ | 9,857,146,880 | 10.001% |
- Downloads last month
- -
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf pathcosmos/JANSORI:Q4_K_M# Run inference directly in the terminal: llama cli -hf pathcosmos/JANSORI:Q4_K_M