File size: 2,387 Bytes
ab4a043 23d9c35 ab4a043 23d9c35 ab4a043 23d9c35 ab4a043 23d9c35 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 |
---
license: apache-2.0
language:
- en
library_name: peft
tags:
- base_model:adapter:distilgpt2
- lora
- transformers
base_model: distilgpt2
pipeline_tag: text-generation
---
# NextStep-Coder-MoE
A high-performance tiny coding LLM with **Interleaved Thinking** capability for advanced reasoning and agentic workflows.
## Model Description
NextStep-Coder-MoE is a LoRA fine-tuned model based on Qwen2.5-Coder-1.5B, optimized for:
- Chain-of-Thought reasoning with `<think>...</think>` tags
- Multi-step coding tasks
- Agentic workflows (plan → act → reflect)
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load model
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B")
model = PeftModel.from_pretrained(base_model, "OsamaBinLikhon/NextStep-Coder-MoE")
tokenizer = AutoTokenizer.from_pretrained("OsamaBinLikhon/NextStep-Coder-MoE")
# Generate
prompt = "Write a Python function to check if a number is prime.\n<think>"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))
```
## Training Details
| Parameter | Value |
|-----------|-------|
| Base Model | Qwen/Qwen2.5-Coder-1.5B |
| Method | LoRA (r=16, alpha=32) |
| Trainable Params | 18.4M (1.18%) |
| Precision | bf16 |
| Framework | Transformers + PEFT |
## Interleaved Thinking Format
The model uses `<think>` tags to show reasoning:
```
User: Write a binary search function.
Model: <think>
I need to implement binary search on a sorted array.
Key steps: find middle, compare, narrow search space.
Edge case: empty array returns -1.
</think>
def binary_search(arr, target):
left, right = 0, len(arr) - 1
while left <= right:
mid = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
left = mid + 1
else:
right = mid - 1
return -1
```
## Intended Use
- Code generation with step-by-step reasoning
- Debugging and code review
- Algorithm design and explanation
- Educational coding assistance
## Limitations
- Best for Python, may vary for other languages
- Requires `<think>` tag retention in conversation history
- 2048 token context limit
## Author
**OsamaBinLikhon**
## License
Apache 2.0
### Framework versions
- PEFT 0.18.0 |