wowsocool123/qwen3-4b-regex-lora

Qwen3-4B Regex LoRA

A LoRA adapter that teaches Qwen/Qwen3-4B-Instruct-2507 to compose regular expressions — stacked lookaheads, backreferences, quantified groups, exclusion classes — rather than recall canonical ones.

Trained on 360 programmatically generated examples, each verified by executing the pattern against its own test cases. ~5 minutes on a free Colab T4.

Results

Evaluated on 24 hand-written held-out tasks in two tiers. A task passes only if its pattern satisfies every held-out test case; some test cases are shown in the prompt and the rest are withheld, so literal-matching scores zero.

tier	what it tests	before	after
hard	composition (lookaheads, backrefs, counting)	0.250	0.417
seed	canonical patterns (postcode, IPv4, email)	0.333	0.167

Fine-tuning traded memorised canonical patterns for compositional skill. The hard tier improved 67% relative. The seed tier regressed — those are patterns the base model already half-knew from pretraining, and training on composition tasks partially overwrote them. Both numbers are reported because the trade is the actual finding.

For reference, Gemini Flash-Lite scores 1.000 / 0.750 on the same tiers.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "wowsocool123/qwen3-4b-regex-lora")

prompt = """Write a Python regular expression for this task.

Task: Match a string of 8 to 16 characters with no whitespace, containing at \
least one lowercase letter, at least one uppercase letter, and at least one digit.

The pattern is applied with re.fullmatch (it must match the ENTIRE string).

Reply with ONLY the regular expression pattern. No explanation, no code fences,
no quotes, no re.compile() call. Just the raw pattern."""

text = tok.apply_chat_template([{"role": "user", "content": prompt}],
                               tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=256, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Prompt format matters. The adapter was trained on the exact template above — task description, re.fullmatch or re.search note, a few example strings, then the "reply with ONLY the pattern" instruction. Straying far from it will degrade output.

Known issues

The model prefixes every answer with an empty <think>\n\n</think> block. Qwen3's chat template injects a reasoning block into the final assistant turn, so all training targets carried one and the model learned to reproduce it. Strip it before using the output:

import re
pattern = re.sub(r"<think>.*?</think>", "", raw, flags=re.DOTALL).strip()

Other limitations: 24 held-out tasks is a small evaluation, so a one-task swing moves the score by 0.083; results come from a single training run with no seed-variance estimate; and training instructions were templated rather than natural English, so unusual phrasing may degrade output.

Training

Method	QLoRA (4-bit NF4 base, LoRA adapter)
Rank / alpha	16 / 32
Target modules	q, k, v, o, gate, up, down
Learning rate	2e-4, linear schedule
Epochs	3
Effective batch	8 (2 × 4 grad accum)
Loss	responses only (prompt masked)
Hardware	Colab T4, ~5 min

adapter_config.json records the base as unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit — Unsloth's pre-quantised mirror of Qwen/Qwen3-4B-Instruct-2507, which is what was actually trained against. It loads fine on either.

Data

Training data was generated programmatically, not by an LLM. Nine templates (one per regex technique) emit the instruction, the reference pattern, and the test cases together, then every candidate is rejected unless the pattern passes its own tests.

An earlier LLM-based generator produced 9 verified tasks in ~20 minutes with six of nine techniques stuck at zero — asking a model to "use a backreference" does not make it use one. Templates produced 400 verified tasks in 0.4 seconds with every technique evenly represented.

Links

Code, evaluation harness, and full write-up: github.com/asadusman123/regex-finetune