File size: 3,227 Bytes
567dda7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | ---
base_model:
- mkd-hossain/keural-alpha
---
---
library_name: transformers
model_name: keural-alpha
---
language:
- en
- kr
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- text-generation
- chat
- vllm
- bf16
- keural-alpha
- mkd
---
# Keural-Alpha
**Keural-Alpha-Chat v0.1** is an instruction-tuned causal language model developed by **MKD Co., Ltd.**
It is optimized for **completion-style prompting** and **local deployment using vLLM** with an OpenAI-compatible API.
This release focuses on **stable, concise assistant responses** using a GPT-style prompt format.
---
## Model Description
- **Model name:** Keural-Alpha-Chat v0.1
- **Architecture:** Causal Language Model (GPT-style)
- **Framework:** PyTorch / Transformers
- **Inference engine:** vLLM
- **Precision:** bfloat16
- **Context length:** 2048 tokens
- **Primary language:** English
This model is designed for **instruction-following and assistant-style text generation** using explicit prompt formatting rather than a built-in chat template.
---
## Intended Use
Keural-Alpha-Chat v0.1 is intended for:
- Instruction-style prompting
- Assistant-like conversations (completion-based)
- Internal chat tools and research
- API-based inference using vLLM
This model is **not a native chat-template model**.
For best results, use `/v1/completions` with a structured prompt.
---
## Prompt Format (Important)
Recommended prompt structure:
### Framework versions
- TRL: 0.27.0
- Transformers: 4.57.6
- Pytorch: 2.9.1
- Datasets: 4.5.0
- Tokenizers: 0.22.1
## vllm run command
python -m vllm.entrypoints.openai.api_server \
--model mkd-hossain/keural-alpha-chat-v0.1 \
--served-model-name keural-alpha-chat-v0.1 \
--dtype bfloat16 \
--tensor-parallel-size 2 \
--max-model-len 2048 \
--gpu-memory-utilization 0.9 \
--port 10008
## respose command
HISTORY="System: You are Keural-Alpha, an AI assistant developed by MKD Co., Ltd."
MODEL="keural-alpha-chat-v0.1"
while true; do
read -p "User: " USER
[[ "$USER" == "exit" ]] && break
PROMPT="$HISTORY\nUser: $USER\nAssistant:"
RESPONSE=$(curl -s http://localhost:10008/v1/completions \
-H "Content-Type: application/json" \
-d "{
\"model\": \"$MODEL\",
\"prompt\": \"$PROMPT\",
\"max_tokens\": 120,
\"temperature\": 0.9,
\"top_p\": 0.85,
\"repetition_penalty\": 1.15,
\"stop\": [\"\\nUser:\"]
}")
ASSISTANT=$(echo "$RESPONSE" | python -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['text'].strip())")
echo "Assistant: $ASSISTANT"
HISTORY="$HISTORY\nUser: $USER\nAssistant: $ASSISTANT"
done
## start vLLM with a chat template
python -m vllm.entrypoints.openai.api_server \
--model mkd-hossain/keural-alpha-chat-v0.1 \
--served-model-name keural-alpha-chat-v0.1 \
--dtype bfloat16 \
--tensor-parallel-size 1 \
--max-model-len 2048 \
--gpu-memory-utilization 0.9 \
--port 10008 \
--chat-template "{% for m in messages %}{% if m.role == 'system' %}System: {{ m.content }}\n{% elif m.role == 'user' %}User: {{ m.content }}\n{% elif m.role == 'assistant' %}Assistant: {{ m.content }}\n{% endif %}{% endfor %}Assistant:"
|