| --- |
| base_model: |
| - mkd-hossain/keural-alpha |
| --- |
| --- |
| library_name: transformers |
| model_name: keural-alpha |
|
|
|
|
| --- |
| language: |
| - en |
| - kr |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - causal-lm |
| - text-generation |
| - chat |
| - vllm |
| - bf16 |
| - keural-alpha |
| - mkd |
|
|
| --- |
|
|
| # Keural-Alpha |
|
|
| **Keural-Alpha-Chat v0.1** is an instruction-tuned causal language model developed by **MKD Co., Ltd.** |
| It is optimized for **completion-style prompting** and **local deployment using vLLM** with an OpenAI-compatible API. |
|
|
| This release focuses on **stable, concise assistant responses** using a GPT-style prompt format. |
|
|
| --- |
|
|
| ## Model Description |
|
|
| - **Model name:** Keural-Alpha-Chat v0.1 |
| - **Architecture:** Causal Language Model (GPT-style) |
| - **Framework:** PyTorch / Transformers |
| - **Inference engine:** vLLM |
| - **Precision:** bfloat16 |
| - **Context length:** 2048 tokens |
| - **Primary language:** English |
|
|
| This model is designed for **instruction-following and assistant-style text generation** using explicit prompt formatting rather than a built-in chat template. |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| Keural-Alpha-Chat v0.1 is intended for: |
|
|
| - Instruction-style prompting |
| - Assistant-like conversations (completion-based) |
| - Internal chat tools and research |
| - API-based inference using vLLM |
|
|
| This model is **not a native chat-template model**. |
| For best results, use `/v1/completions` with a structured prompt. |
|
|
| --- |
|
|
| ## Prompt Format (Important) |
|
|
| Recommended prompt structure: |
|
|
|
|
| ### Framework versions |
|
|
| - TRL: 0.27.0 |
| - Transformers: 4.57.6 |
| - Pytorch: 2.9.1 |
| - Datasets: 4.5.0 |
| - Tokenizers: 0.22.1 |
|
|
|
|
| ## vllm run command |
| python -m vllm.entrypoints.openai.api_server \ |
| --model mkd-hossain/keural-alpha-chat-v0.1 \ |
| --served-model-name keural-alpha-chat-v0.1 \ |
| --dtype bfloat16 \ |
| --tensor-parallel-size 2 \ |
| --max-model-len 2048 \ |
| --gpu-memory-utilization 0.9 \ |
| --port 10008 |
| |
| |
| ## respose command |
| HISTORY="System: You are Keural-Alpha, an AI assistant developed by MKD Co., Ltd." |
| MODEL="keural-alpha-chat-v0.1" |
| |
| while true; do |
| read -p "User: " USER |
| [[ "$USER" == "exit" ]] && break |
| |
| PROMPT="$HISTORY\nUser: $USER\nAssistant:" |
| |
| RESPONSE=$(curl -s http://localhost:10008/v1/completions \ |
| -H "Content-Type: application/json" \ |
| -d "{ |
| \"model\": \"$MODEL\", |
| \"prompt\": \"$PROMPT\", |
| \"max_tokens\": 120, |
| \"temperature\": 0.9, |
| \"top_p\": 0.85, |
| \"repetition_penalty\": 1.15, |
| \"stop\": [\"\\nUser:\"] |
| }") |
| |
| ASSISTANT=$(echo "$RESPONSE" | python -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['text'].strip())") |
|
|
| echo "Assistant: $ASSISTANT" |
|
|
| HISTORY="$HISTORY\nUser: $USER\nAssistant: $ASSISTANT" |
| done |
|
|
|
|
|
|
|
|
|
|
|
|
| ## start vLLM with a chat template |
|
|
| python -m vllm.entrypoints.openai.api_server \ |
| --model mkd-hossain/keural-alpha-chat-v0.1 \ |
| --served-model-name keural-alpha-chat-v0.1 \ |
| --dtype bfloat16 \ |
| --tensor-parallel-size 1 \ |
| --max-model-len 2048 \ |
| --gpu-memory-utilization 0.9 \ |
| --port 10008 \ |
| --chat-template "{% for m in messages %}{% if m.role == 'system' %}System: {{ m.content }}\n{% elif m.role == 'user' %}User: {{ m.content }}\n{% elif m.role == 'assistant' %}Assistant: {{ m.content }}\n{% endif %}{% endfor %}Assistant:" |
| |
