Qwen3.6-35B-A3B "The Judge — Code Edition"

A LoRA adapter for Qwen3.6-35B-A3B fine-tuned for code generation, code evaluation, and task orchestration.

Requires a pinned output schema for reliable judge behavior. The model memorized training-template shapes and will emit synthetic orchestration records under a loose prompt. Using the canonical JSON schema below is not optional — it's the difference between chance-level (kappa 0.00) and reliable (kappa 0.80) judging.

Performance

Benchmark	Base Model	+Adapter	Δ
Coder Quick (10-task)	70.0%	80.0%	+10.0 pp

⚠️ McNemar p = 1.00 on 10 tasks. A single-task difference. Do not treat this as statistically significant. Proper execution-graded benchmarks (300-400 tasks) are forthcoming.

Judge Sanity (10-task, constrained format):

Metric	Base Model	+Adapter	Δ
Accuracy	70%	90%	+20 pp
Cohen's κ	0.40	0.80	+0.40
False rejects (of 5 good)	3	1	−2
False accepts (of 5 bad)	0	0	—

⚠️ n=10, McNemar p = 0.50. Direction is consistent and mechanism is clear (fewer false rejections, zero false accepts), but not yet statistically validated. A larger benchmark is the priority.

Canonical Output Schema (Required for Judge)

This is the single most important thing to get right. The adapter was trained on template-generated records. Without a pinned output format, it emits training-set artifacts instead of evaluating code. Use this exact schema:

{
  "judgment": "accept | reject | fix | escalate",
  "confidence": 0.0-1.0,
  "issues": [
    {
      "severity": "error | warning | info",
      "description": "What is wrong and why",
      "line_numbers": [0]
    }
  ],
  "strengths": ["What the code does well"],
  "suggested_fix": "Corrected code if applicable",
  "explanation": "Overall assessment of code quality"
}

Field semantics:

judgment: accept = passes, reject = fails, fix = minor fix needed, escalate = needs human review
confidence: 0.0-1.0 scale. Below 0.6 → set judgment to escalate
issues: empty array if no issues found
suggested_fix: only required when judgment is fix
strengths: at least 1 item for accept judgments

Recommended System Prompt

You are a code evaluation assistant. Your job is to evaluate code for correctness, edge cases, and style.

You MUST respond with valid JSON only, following this exact schema:

{
  "judgment": "accept | reject | fix | escalate",
  "confidence": 0.0-1.0,
  "issues": [
    {
      "severity": "error | warning | info",
      "description": "What is wrong and why",
      "line_numbers": [0]
    }
  ],
  "strengths": ["What the code does well"],
  "suggested_fix": "Corrected code if applicable",
  "explanation": "Overall assessment of code quality"
}

Rules:
- judgment "accept" = code is correct. "reject" = fundamentally wrong. "fix" = minor issue. "escalate" = you are unsure.
- If confidence < 0.6, set judgment to "escalate".
- issues must be empty for "accept". strengths must be empty for "reject".
- suggested_fix only required for "fix" judgments.
- Be specific about what is wrong and why. Reference line numbers when possible.
- Never output anything outside the JSON object.

Usage

Installation

pip install torch transformers peft accelerate

Load with adapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "Qwen/Qwen3.6-35B-A3B"
adapter_path = "CertainLogicAI/qwen3.6-35b-a3b-the-judge-code-edition"  # Note: CertainLogicAI, not CertainLogic

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(base_model_name)

# Apply adapter
model = PeftModel.from_pretrained(model, adapter_path)

Using as a judge (canonical format)

import json

system_prompt = """You are a code evaluation assistant. Your job is to evaluate code for correctness, edge cases, and style.

You MUST respond with valid JSON only, following this exact schema:

{
  "judgment": "accept | reject | fix | escalate",
  "confidence": 0.0-1.0,
  "issues": [
    {
      "severity": "error | warning | info",
      "description": "What is wrong and why",
      "line_numbers": [0]
    }
  ],
  "strengths": ["What the code does well"],
  "suggested_fix": "Corrected code if applicable",
  "explanation": "Overall assessment of code quality"
}

Rules:
- judgment "accept" = code is correct. "reject" = fundamentally wrong. "fix" = minor issue. "escalate" = you are unsure.
- If confidence < 0.6, set judgment to "escalate".
- issues must be empty for "accept". strengths must be empty for "reject".
- suggested_fix only required for "fix" judgments.
- Be specific about what is wrong and why. Reference line numbers when possible.
- Never output anything outside the JSON object."""

user_prompt = """Evaluate this code for correctness:

```python
def divide(a, b):
    return a / b

Does this code handle edge cases properly?"""

messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt} ]

inputs = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_tensors="pt" ).to("cuda")

outputs = model.generate(**inputs, max_new_tokens=512) result = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)

Always parse as JSON

try: parsed = json.loads(result) print(f"Judgment: {parsed['judgment']} (confidence: {parsed['confidence']})") for issue in parsed.get('issues', []): print(f" - [{issue['severity']}] {issue['description']}") except json.JSONDecodeError: print(f"WARNING: Non-JSON output. Model may need a stricter prompt.\nRaw: {result}")


### Using as a code generator

```python
prompt = "def fibonacci(n):\n    \"\"\"Return the nth Fibonacci number.\"\"\""
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Architecture Notes

MoE Structure

The base model Qwen3.6-35B-A3B uses 256 experts with 8 routed per token plus 1 shared expert. The README on earlier versions incorrectly stated "8/32 active MoE" — the correct count is 256 experts, 8 routed + 1 shared.

Target Modules

The adapter was trained on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Important caveat: In Qwen3.6 MoE, gate_proj, up_proj, and down_proj only target the shared expert and attention layers in 10 of 40 layers. The 256 routed expert weights (~629M adaptable parameters) were never adapted because PEFT could not hook into the fused MoE tensor structure. This means the adapter's influence is concentrated in attention projections and the shared expert, not the routed MoE experts.

Sequence Length

The training sequence length was 2048 tokens, which is tight for code review tasks that include both the solution and the evaluation. For production use, consider increasing max_new_tokens to 4096 or more. The base model supports up to 262,144 tokens.

Training Details

Parameter	Value
Base model	Qwen/Qwen3.6-35B-A3B (34.7B params, 256 experts, 8 routed + 1 shared)
Method	LoRA (r=8, alpha=16, dropout=0.05)
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training records	1,685 train / 81 validation
Data sources	6 curated template-generated sources
Compute	Single H100 (80GB), ~61 minutes
Precision	BF16 mixed precision
Sequence length	2048 (unpacked)
Epochs	1
Eval loss	0.5793

Refusal Safety

Tested on 6 refusal probe cases: 0/6 false refusals. The adapter does not introduce excessive refusal behavior.

Known Limitations

Output schema is critical. Without the pinned JSON schema, the model emits training-template artifacts. Always use the canonical format.
Small benchmark. The 10-task coder and 10-task judge evaluations are directional, not statistically significant. Larger benchmarks (300-400 tasks) are needed.
Routed experts not adapted. The adapter only touches attention projections and the shared expert. A future training run with expert-targeted adaptation may yield further gains.
Sequence length. 2048 training tokens is tight for code review. Increase max_new_tokens for production use.
MoE PEFT fragility. Merging the LoRA into the base model can be fragile due to the fused MoE tensor structure. If merge fails, use the adapter as a standalone LoRA without merging.

License

Apache 2.0. The base model (Qwen3.6-35B-A3B) carries its own license terms.

About

Built by CertainLogic. We build deterministic AI tools for agents that execute real work.

The training dataset is proprietary and not included.

Downloads last month: 161

Model tree for CertainLogicAI/qwen3.6-35b-a3b-the-judge-code-edition

Base model

Qwen/Qwen3.6-35B-A3B

Adapter

(240)

this model