| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - code-generation |
| - codefeedback |
| - hrm-text |
| base_model: sapientai/HRM-Text-1B |
| --- |
| |
| # HRM-Text-1B-Code-Feedback |
|
|
| Fine-tuned version of [HRM-Text-1B](https://huggingface.co/sapientai/HRM-Text-1B) on the CodeFeedback dataset for code generation. |
|
|
| ## Model Details |
|
|
| - **Base Model:** sapientai/HRM-Text-1B (1B parameters, hierarchical reasoning model) |
| - **Training Data:** CodeFeedback dataset (~131k samples, filtered to <= 4096 tokens) |
| - **Training:** 2 epochs, ~8 hours on L40S GPU |
| - **Architecture:** Hierarchical Reasoning Model with H_cycles=2, L_cycles=3 |
|
|
| ## Training Data Distribution |
|
|
| | Language | Samples | |
| |----------|---------| |
| | Python | ~80k | |
| | JavaScript | ~7.6k | |
| | React | ~550 | |
|
|
| ## Performance |
|
|
| | Task | Base | Fine-tuned | |
| |------|------|------------| |
| | C++ factorial | Broken (repeating includes) | Correct | |
| | JS reverse | Wrong syntax | Correct syntax | |
| | Java max | Wrong type | Better structure | |
|
|
| ## Usage |
|
|
|
|
|
|
| ## Training Details |
|
|
| - **Framework:** PyTorch with FlashAttention 3 |
| - **Loss:** Cross-entropy |
| - **Hardware:** AWS L40S GPU |
| - **Training Time:** ~8 hours |
|
|
| ## Limitations |
|
|
| - Maximum sequence length: 4096 tokens |
| - Requires FlashAttention 3 for inference (Ada Lovelace or newer GPUs) |
| - Limited React/TypeScript performance due to small training data |
| - Best performance on Python code generation |
|
|
| ## License |
|
|
| MIT License |
|
|
