Text Generation
Transformers
Safetensors
English
qwen2
code-generation
python
lora
rejection-sampling
conversational
text-generation-inference
Instructions to use layer8labs8/OrdoPy-Coder-0.5B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use layer8labs8/OrdoPy-Coder-0.5B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="layer8labs8/OrdoPy-Coder-0.5B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B") model = AutoModelForCausalLM.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use layer8labs8/OrdoPy-Coder-0.5B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "layer8labs8/OrdoPy-Coder-0.5B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "layer8labs8/OrdoPy-Coder-0.5B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/layer8labs8/OrdoPy-Coder-0.5B
- SGLang
How to use layer8labs8/OrdoPy-Coder-0.5B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "layer8labs8/OrdoPy-Coder-0.5B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "layer8labs8/OrdoPy-Coder-0.5B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "layer8labs8/OrdoPy-Coder-0.5B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "layer8labs8/OrdoPy-Coder-0.5B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use layer8labs8/OrdoPy-Coder-0.5B with Docker Model Runner:
docker model run hf.co/layer8labs8/OrdoPy-Coder-0.5B
Commit ·
0846bbc
0
Parent(s):
Initial release: OrdoPy-Coder-0.5B
Browse files- .gitattributes +36 -0
- README.md +85 -0
- chat_template.jinja +54 -0
- config.json +58 -0
- eval_benchmarks.json +88 -0
- generation_config.json +14 -0
- model.safetensors +3 -0
- tokenizer.json +3 -0
- tokenizer_config.json +30 -0
.gitattributes
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
|
| 4 |
+
language:
|
| 5 |
+
- en
|
| 6 |
+
library_name: transformers
|
| 7 |
+
pipeline_tag: text-generation
|
| 8 |
+
tags:
|
| 9 |
+
- qwen2
|
| 10 |
+
- code-generation
|
| 11 |
+
- python
|
| 12 |
+
- reasoning
|
| 13 |
+
- coding
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# OrdoPy-Coder-0.5B
|
| 17 |
+
|
| 18 |
+
**OrdoPy-Coder-0.5B** is a compact 0.5B parameter specialist model tuned on `Qwen/Qwen2.5-Coder-0.5B-Instruct` exclusively for **Python programming**, **English instruction following**, and **step-by-step logical reasoning**.
|
| 19 |
+
|
| 20 |
+
Focusing the full capacity of the 0.5B backbone on a single coherent skill cluster (**English + Logic + Verified Python**) delivers strong functional code execution accuracy while remaining fast and deployable in < 1.5 GB VRAM (`float16`).
|
| 21 |
+
|
| 22 |
+
## Official Execution Benchmarks (`Pass@1` — 421 Tasks)
|
| 23 |
+
|
| 24 |
+
Evaluated greedily (`do_sample=False`) inside isolated `3.0s`-timeout `sys.executable -I` subprocesses against full unit-test suites (`openai/openai_humaneval` `164` tasks + `google-research-datasets/mbpp` `sanitized` `257` test tasks):
|
| 25 |
+
|
| 26 |
+
| Benchmark Suite | Baseline (`Qwen2.5-Coder-0.5B-Instruct`) | `OrdoPy-Coder-0.5B` | Gain |
|
| 27 |
+
| :--- | :---: | :---: | :---: |
|
| 28 |
+
| **HumanEval (`164` tasks)** | `9.76%` (`16/164`) | **`40.85%` (`67/164`)** | **`+31.09%` (`4.19x`)** |
|
| 29 |
+
| **MBPP Sanitized Test (`257` tasks)** | `36.58%` (`94/257`) | **`46.30%` (`119/257`)** | **`+9.72%` (`+25` tasks)** |
|
| 30 |
+
| **Combined (`421` tasks)** | `26.13%` (`110/421`) | **`44.18%` (`186/421`)** | **`+18.05%` (`+76` tasks)** |
|
| 31 |
+
|
| 32 |
+
### Official EvalPlus Suite (`evalplus` Greedy `Pass@1`)
|
| 33 |
+
|
| 34 |
+
| Benchmark Suite | `OrdoPy-Coder-0.5B` `Pass@1` | Passed Tasks |
|
| 35 |
+
| :--- | :---: | :---: |
|
| 36 |
+
| **HumanEval (`base`)** | **`48.8%`** (`0.4878`) | `80 / 164` |
|
| 37 |
+
| **HumanEval+ (`base + extra tests`)** | **`46.3%`** (`0.4634`) | `76 / 164` |
|
| 38 |
+
| **MBPP (`base`)** | **`48.4%`** (`0.4841`) | `183 / 378` |
|
| 39 |
+
| **MBPP+ (`base + extra tests`)** | **`42.1%`** (`0.4206`) | `159 / 378` |
|
| 40 |
+
|
| 41 |
+
## Held-Out Domain Evaluation
|
| 42 |
+
|
| 43 |
+
| Domain (Held-Out Split) | Baseline Loss (PPL / Token Acc) | `OrdoPy-Coder-0.5B` Loss (PPL / Token Acc) |
|
| 44 |
+
| :--- | :---: | :---: |
|
| 45 |
+
| **`python_code`** (`MBPP Sanitized Val`) | `1.0226` (`2.78` / `76.44%`) | **`0.9843` (`2.68` / `78.40%`)** |
|
| 46 |
+
| **`logic_math`** (`GSM8K Official Test`) | `1.4752` (`4.37` / `73.21%`) | **`1.4096` (`4.09` / `74.68%`)** |
|
| 47 |
+
| **`english_debug`** (`Alpaca Held-Out`) | `1.6227` (`5.07` / `59.53%`) | **`1.4409` (`4.22` / `61.58%`)** |
|
| 48 |
+
|
| 49 |
+
## Model Details
|
| 50 |
+
|
| 51 |
+
- **Architecture**: `Qwen2ForCausalLM` (`24` layers, `hidden_size=896`, `494M` parameters, `32K` context window)
|
| 52 |
+
- **Base Model**: `Qwen/Qwen2.5-Coder-0.5B-Instruct`
|
| 53 |
+
- **Specialization**: Python code generation, algorithmic problem solving, debugging & step-by-step reasoning in English
|
| 54 |
+
- **Format**: Safetensors (`model.safetensors`) + Qwen ChatML tokenizer template
|
| 55 |
+
|
| 56 |
+
## Quickstart Usage
|
| 57 |
+
|
| 58 |
+
```python
|
| 59 |
+
import torch
|
| 60 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 61 |
+
|
| 62 |
+
model_id = "layer8labs8/OrdoPy-Coder-0.5B"
|
| 63 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
| 64 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 65 |
+
model_id, torch_dtype=torch.float16, device_map="auto"
|
| 66 |
+
)
|
| 67 |
+
|
| 68 |
+
messages = [
|
| 69 |
+
{
|
| 70 |
+
"role": "system",
|
| 71 |
+
"content": (
|
| 72 |
+
"You are OrdoPy, an expert Python software engineer and logical problem solver. "
|
| 73 |
+
"Think clearly step-by-step in English and write clean, bug-free Python code."
|
| 74 |
+
),
|
| 75 |
+
},
|
| 76 |
+
{
|
| 77 |
+
"role": "user",
|
| 78 |
+
"content": "Write a Python function `is_palindrome_clean(s: str) -> bool` ignoring spaces and punctuation.",
|
| 79 |
+
},
|
| 80 |
+
]
|
| 81 |
+
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 82 |
+
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
| 83 |
+
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
|
| 84 |
+
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 85 |
+
```
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- if tools %}
|
| 2 |
+
{{- '<|im_start|>system\n' }}
|
| 3 |
+
{%- if messages[0]['role'] == 'system' %}
|
| 4 |
+
{{- messages[0]['content'] }}
|
| 5 |
+
{%- else %}
|
| 6 |
+
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
|
| 7 |
+
{%- endif %}
|
| 8 |
+
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
| 9 |
+
{%- for tool in tools %}
|
| 10 |
+
{{- "\n" }}
|
| 11 |
+
{{- tool | tojson }}
|
| 12 |
+
{%- endfor %}
|
| 13 |
+
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
| 14 |
+
{%- else %}
|
| 15 |
+
{%- if messages[0]['role'] == 'system' %}
|
| 16 |
+
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
|
| 17 |
+
{%- else %}
|
| 18 |
+
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
|
| 19 |
+
{%- endif %}
|
| 20 |
+
{%- endif %}
|
| 21 |
+
{%- for message in messages %}
|
| 22 |
+
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
|
| 23 |
+
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
|
| 24 |
+
{%- elif message.role == "assistant" %}
|
| 25 |
+
{{- '<|im_start|>' + message.role }}
|
| 26 |
+
{%- if message.content %}
|
| 27 |
+
{{- '\n' + message.content }}
|
| 28 |
+
{%- endif %}
|
| 29 |
+
{%- for tool_call in message.tool_calls %}
|
| 30 |
+
{%- if tool_call.function is defined %}
|
| 31 |
+
{%- set tool_call = tool_call.function %}
|
| 32 |
+
{%- endif %}
|
| 33 |
+
{{- '\n<tool_call>\n{"name": "' }}
|
| 34 |
+
{{- tool_call.name }}
|
| 35 |
+
{{- '", "arguments": ' }}
|
| 36 |
+
{{- tool_call.arguments | tojson }}
|
| 37 |
+
{{- '}\n</tool_call>' }}
|
| 38 |
+
{%- endfor %}
|
| 39 |
+
{{- '<|im_end|>\n' }}
|
| 40 |
+
{%- elif message.role == "tool" %}
|
| 41 |
+
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
|
| 42 |
+
{{- '<|im_start|>user' }}
|
| 43 |
+
{%- endif %}
|
| 44 |
+
{{- '\n<tool_response>\n' }}
|
| 45 |
+
{{- message.content }}
|
| 46 |
+
{{- '\n</tool_response>' }}
|
| 47 |
+
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
| 48 |
+
{{- '<|im_end|>\n' }}
|
| 49 |
+
{%- endif %}
|
| 50 |
+
{%- endif %}
|
| 51 |
+
{%- endfor %}
|
| 52 |
+
{%- if add_generation_prompt %}
|
| 53 |
+
{{- '<|im_start|>assistant\n' }}
|
| 54 |
+
{%- endif %}
|
config.json
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"Qwen2ForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_dropout": 0.0,
|
| 6 |
+
"bos_token_id": 151643,
|
| 7 |
+
"dtype": "float32",
|
| 8 |
+
"eos_token_id": 151645,
|
| 9 |
+
"hidden_act": "silu",
|
| 10 |
+
"hidden_size": 896,
|
| 11 |
+
"initializer_range": 0.02,
|
| 12 |
+
"intermediate_size": 4864,
|
| 13 |
+
"layer_types": [
|
| 14 |
+
"full_attention",
|
| 15 |
+
"full_attention",
|
| 16 |
+
"full_attention",
|
| 17 |
+
"full_attention",
|
| 18 |
+
"full_attention",
|
| 19 |
+
"full_attention",
|
| 20 |
+
"full_attention",
|
| 21 |
+
"full_attention",
|
| 22 |
+
"full_attention",
|
| 23 |
+
"full_attention",
|
| 24 |
+
"full_attention",
|
| 25 |
+
"full_attention",
|
| 26 |
+
"full_attention",
|
| 27 |
+
"full_attention",
|
| 28 |
+
"full_attention",
|
| 29 |
+
"full_attention",
|
| 30 |
+
"full_attention",
|
| 31 |
+
"full_attention",
|
| 32 |
+
"full_attention",
|
| 33 |
+
"full_attention",
|
| 34 |
+
"full_attention",
|
| 35 |
+
"full_attention",
|
| 36 |
+
"full_attention",
|
| 37 |
+
"full_attention"
|
| 38 |
+
],
|
| 39 |
+
"max_position_embeddings": 32768,
|
| 40 |
+
"max_window_layers": 24,
|
| 41 |
+
"model_type": "qwen2",
|
| 42 |
+
"num_attention_heads": 14,
|
| 43 |
+
"num_hidden_layers": 24,
|
| 44 |
+
"num_key_value_heads": 2,
|
| 45 |
+
"pad_token_id": null,
|
| 46 |
+
"rms_norm_eps": 1e-06,
|
| 47 |
+
"rope_parameters": {
|
| 48 |
+
"rope_theta": 1000000.0,
|
| 49 |
+
"rope_type": "default"
|
| 50 |
+
},
|
| 51 |
+
"sliding_window": null,
|
| 52 |
+
"tie_word_embeddings": true,
|
| 53 |
+
"transformers_version": "5.20.2",
|
| 54 |
+
"use_cache": true,
|
| 55 |
+
"use_sliding_window": false,
|
| 56 |
+
"vocab_size": 151936,
|
| 57 |
+
"torch_dtype": "float32"
|
| 58 |
+
}
|
eval_benchmarks.json
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_name": "OrdoPy-Coder-0.5B",
|
| 3 |
+
"base_model": "Qwen/Qwen2.5-Coder-0.5B-Instruct",
|
| 4 |
+
"scope": [
|
| 5 |
+
"English",
|
| 6 |
+
"Logical Reasoning",
|
| 7 |
+
"Python"
|
| 8 |
+
],
|
| 9 |
+
"baseline_loss": {
|
| 10 |
+
"python_code": {
|
| 11 |
+
"loss": 1.0226,
|
| 12 |
+
"perplexity": 2.78,
|
| 13 |
+
"token_accuracy_pct": 76.44
|
| 14 |
+
},
|
| 15 |
+
"logic_math": {
|
| 16 |
+
"loss": 1.4752,
|
| 17 |
+
"perplexity": 4.37,
|
| 18 |
+
"token_accuracy_pct": 73.21
|
| 19 |
+
},
|
| 20 |
+
"english_debug": {
|
| 21 |
+
"loss": 1.6227,
|
| 22 |
+
"perplexity": 5.07,
|
| 23 |
+
"token_accuracy_pct": 59.53
|
| 24 |
+
}
|
| 25 |
+
},
|
| 26 |
+
"final_best_loss": {
|
| 27 |
+
"python_code": {
|
| 28 |
+
"loss": 0.9843,
|
| 29 |
+
"perplexity": 2.68,
|
| 30 |
+
"token_accuracy_pct": 78.4
|
| 31 |
+
},
|
| 32 |
+
"logic_math": {
|
| 33 |
+
"loss": 1.4096,
|
| 34 |
+
"perplexity": 4.09,
|
| 35 |
+
"token_accuracy_pct": 74.68
|
| 36 |
+
},
|
| 37 |
+
"english_debug": {
|
| 38 |
+
"loss": 1.4409,
|
| 39 |
+
"perplexity": 4.22,
|
| 40 |
+
"token_accuracy_pct": 61.58
|
| 41 |
+
}
|
| 42 |
+
},
|
| 43 |
+
"baseline_full_exec_421": {
|
| 44 |
+
"humaneval_passed": 16,
|
| 45 |
+
"humaneval_total": 164,
|
| 46 |
+
"humaneval_pass_at_1_pct": 9.76,
|
| 47 |
+
"mbpp_passed": 94,
|
| 48 |
+
"mbpp_total": 257,
|
| 49 |
+
"mbpp_pass_at_1_pct": 36.58,
|
| 50 |
+
"combined_pass_at_1_pct": 26.13
|
| 51 |
+
},
|
| 52 |
+
"final_full_exec_421": {
|
| 53 |
+
"humaneval_passed": 67,
|
| 54 |
+
"humaneval_total": 164,
|
| 55 |
+
"humaneval_pass_at_1_pct": 40.85,
|
| 56 |
+
"mbpp_passed": 119,
|
| 57 |
+
"mbpp_total": 257,
|
| 58 |
+
"mbpp_pass_at_1_pct": 46.3,
|
| 59 |
+
"combined_pass_at_1_pct": 44.18
|
| 60 |
+
},
|
| 61 |
+
"evalplus": {
|
| 62 |
+
"version": "0.4.0.dev44",
|
| 63 |
+
"datasets": {
|
| 64 |
+
"humaneval": {
|
| 65 |
+
"total": 164,
|
| 66 |
+
"base": {
|
| 67 |
+
"passed": 80,
|
| 68 |
+
"pass@1": 0.4878048780487805
|
| 69 |
+
},
|
| 70 |
+
"plus": {
|
| 71 |
+
"passed": 76,
|
| 72 |
+
"pass@1": 0.4634146341463415
|
| 73 |
+
}
|
| 74 |
+
},
|
| 75 |
+
"mbpp": {
|
| 76 |
+
"total": 378,
|
| 77 |
+
"base": {
|
| 78 |
+
"passed": 183,
|
| 79 |
+
"pass@1": 0.48412698412698413
|
| 80 |
+
},
|
| 81 |
+
"plus": {
|
| 82 |
+
"passed": 159,
|
| 83 |
+
"pass@1": 0.42063492063492064
|
| 84 |
+
}
|
| 85 |
+
}
|
| 86 |
+
}
|
| 87 |
+
}
|
| 88 |
+
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 151643,
|
| 3 |
+
"do_sample": true,
|
| 4 |
+
"eos_token_id": [
|
| 5 |
+
151645,
|
| 6 |
+
151643
|
| 7 |
+
],
|
| 8 |
+
"pad_token_id": 151643,
|
| 9 |
+
"repetition_penalty": 1.05,
|
| 10 |
+
"temperature": 0.7,
|
| 11 |
+
"top_k": 20,
|
| 12 |
+
"top_p": 0.8,
|
| 13 |
+
"transformers_version": "5.20.2"
|
| 14 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c2a45803542e6f99d998f8beae68e1e0be97bc02379087d01fd2635cbac9a46c
|
| 3 |
+
size 1976163472
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f7f96da3a872b5e901575b2067c744ad336c3a3d77a21584d20024557b1bd7f0
|
| 3 |
+
size 11422059
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"backend": "tokenizers",
|
| 4 |
+
"bos_token": null,
|
| 5 |
+
"clean_up_tokenization_spaces": false,
|
| 6 |
+
"eos_token": "<|im_end|>",
|
| 7 |
+
"errors": "replace",
|
| 8 |
+
"extra_special_tokens": [
|
| 9 |
+
"<|im_start|>",
|
| 10 |
+
"<|im_end|>",
|
| 11 |
+
"<|object_ref_start|>",
|
| 12 |
+
"<|object_ref_end|>",
|
| 13 |
+
"<|box_start|>",
|
| 14 |
+
"<|box_end|>",
|
| 15 |
+
"<|quad_start|>",
|
| 16 |
+
"<|quad_end|>",
|
| 17 |
+
"<|vision_start|>",
|
| 18 |
+
"<|vision_end|>",
|
| 19 |
+
"<|vision_pad|>",
|
| 20 |
+
"<|image_pad|>",
|
| 21 |
+
"<|video_pad|>"
|
| 22 |
+
],
|
| 23 |
+
"is_local": false,
|
| 24 |
+
"local_files_only": false,
|
| 25 |
+
"model_max_length": 32768,
|
| 26 |
+
"pad_token": "<|endoftext|>",
|
| 27 |
+
"split_special_tokens": false,
|
| 28 |
+
"tokenizer_class": "Qwen2Tokenizer",
|
| 29 |
+
"unk_token": null
|
| 30 |
+
}
|