| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - ruby |
| - rails |
| - code-generation |
| - gguf |
| - fine-tuned |
| - lora |
| - unsloth |
| pipeline_tag: text-generation |
| base_model: Qwen/Qwen3-Coder-30B-A3B-Instruct |
| model-index: |
| - name: qwen3-coder-30b-rails |
| results: [] |
| --- |
| |
| # qwen3-coder-30b-rails |
|
|
| A 31B parameter Mixture-of-Experts model fine-tuned for **Ruby on Rails code generation**. Trained on 111,000 samples extracted from our own internal Rails projects. |
|
|
| Built by [Bytecode](https://bytecode.hr). |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Base model | [Qwen3-Coder-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) | |
| | Architecture | Qwen3 MoE (31B total, 3B active) | |
| | Training method | QLoRA (rank 16) via [Unsloth](https://github.com/unslothai/unsloth) | |
| | Training data | 111K samples from internal Rails projects | |
| | Training cost | ~$32 (A100 80GB, ~26 hours) | |
| | Quantization | GGUF Q4_K_M (18.6 GB), Q5_K_M (21.7 GB) | |
|
|
| ## What it does |
|
|
| This model writes idiomatic Ruby on Rails code following specific conventions: |
|
|
| - Devise authentication |
| - Namespaced concerns instead of service objects |
| - Sidekiq instead of Solid Queue |
| - State-as-records instead of boolean flags |
| - DaisyUI drawer layouts instead of ActiveAdmin |
|
|
| It generates code that follows these patterns without prompt engineering — the conventions are baked into the weights. |
|
|
| ## Usage with Ollama |
|
|
| ```bash |
| # Download and run |
| ollama run bytecodehr/qwen3-coder-30b-rails |
| |
| # Example prompt |
| ollama run bytecodehr/qwen3-coder-30b-rails "Write a Rails controller for managing user subscriptions with state transitions" |
| ``` |
|
|
| ### Memory requirements |
|
|
| | Format | GGUF Size | Min RAM | Recommended | |
| |---|---|---|---| |
| | Q5_K_M | 21.7 GB | 24 GB | 32 GB | |
| | Q4_K_M | 18.6 GB | 20 GB | 24 GB | |
|
|
| Rule of thumb: GGUF file size + 2–4 GB for KV cache and overhead. |
|
|
| ## Training |
|
|
| Trained with LoRA (rank 16, alpha 16) on attention projection layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`). Only 0.78% of parameters were trained. |
|
|
| The dataset pipeline: |
| 1. Extracted code from our internal Rails projects |
| 2. 15-step cleaning and deduplication pipeline |
| 3. 111K final training samples |
| 4. Includes 29 contrastive pairs (wrong way vs right way) |
| 5. Source diversity cap at 20% per repository |
|
|
| Full details in our blog posts: |
| - [Part 1: Dataset Engineering](https://bytecode.hr/posts/training-rails-llms-part-1-dataset-engineering) |
| - [Part 2: Training, Quantization, and Deployment](https://bytecode.hr/posts/training-rails-llms-part-2-training-quantization-deployment) |
|
|
| ## Why Ruby for LLMs? |
|
|
| Ruby uses 42–45% fewer tokens than TypeScript across every major LLM tokenizer. That means more code fits in the context window, generations are faster, and costs are lower. Read our analysis: [Why Ruby Is the Better Language for LLM-Powered Development](https://bytecode.hr/posts/why-ruby-is-the-better-language-for-llm-powered-development). |
|
|
| ## Other models |
|
|
| - [bytecodehr/qwen3-8b-rails](https://huggingface.co/bytecodehr/qwen3-8b-rails) — 8B dense model, runs on laptops (5 GB) |
| - [bytecodehr/qwen2.5-coder-7b-rails](https://huggingface.co/bytecodehr/qwen2.5-coder-7b-rails) — 7B LoRA adapter |
| - [bytecodehr/qwen2.5-coder-3b-rails](https://huggingface.co/bytecodehr/qwen2.5-coder-3b-rails) — 3B LoRA adapter |
