README.md

---
license: apache-2.0
library_name: peft
pipeline_tag: text-generation
base_model: Qwen/Qwen3-4B-Base
base_model_relation: adapter
tags:
  - peft
  - lora
  - transformers
  - safetensors
  - qwen3
  - code
  - text-generation
model-index:
  - name: Qwen3-4B-Base-ThinkCode-A-U025 PEFT Adapter
    results:
      - task:
          type: text-generation
          name: Code Generation
        dataset:
          name: EvalScope Full1055 corrected (development-only)
          type: evalscope-full1055-corrected-development
        metrics:
          - type: pass@1
            name: code_only pass@1 (seed 3407)
            value: 25.21
---

# Qwen3-4B-Base-ThinkCode-A-U025 — PEFT Adapter

This repository contains a **PEFT LoRA adapter only**. It does not contain the
Qwen3 base-model weights and cannot be loaded as a standalone causal language
model.

The required base is
[`Qwen/Qwen3-4B-Base`](https://huggingface.co/Qwen/Qwen3-4B-Base) at the fixed
revision `906bfd4b4dc7f14ee4320094d8b41684abff8539`.

## Adapter construction

`A-U025` is the Phase A uniform-scale arm. Starting from the completed source
LoRA, every selected LoRA `B` tensor—including the `lm_head` adapter—is
multiplied by `0.25` in FP32. LoRA `A` tensors are unchanged. With
`lora_alpha=128` and `r=64`, PEFT therefore applies the exact intended
`0.25×` source delta to all 253 adapted modules.

Because Qwen3 ties `lm_head.weight` to `embed_tokens.weight`, the released
standard-PEFT representation stores the head factors as transposed
`embed_tokens` LoRA factors and sets `ensure_weight_tying=true`. PEFT then
shares that adapter with the tied output layer, matching both input-embedding
and output-head effects without storing any base-layer tensor.

The repository includes `MODULE_SCALE_MANIFEST.json`, which records every
logical module, tensor key, physical base weight, and scale. This release is
from the completed Phase A delta-scaling line; it is **not** the later failed
NEXTGEN route and does not include subsequent protocol-repair experiments.

## Loading with PEFT

Use recent `transformers` and `peft` versions. Load the fixed base first, then
attach this adapter:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Base"
base_revision = "906bfd4b4dc7f14ee4320094d8b41684abff8539"
adapter_id = "modrill/Qwen3-4B-Base-ThinkCode-A-U025"

tokenizer = AutoTokenizer.from_pretrained(base_id, revision=base_revision)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    revision=base_revision,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)

messages = [{"role": "user", "content": "Write a Python function that checks whether a number is prime."}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
eos_ids = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|im_end|>"),
]
outputs = model.generate(**inputs, max_new_tokens=2048, eos_token_id=eos_ids)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
```

The base tokenizer's chat template supports `enable_thinking`. Disable it for
direct code generation matching the concise screening style, or enable it when
explicit reasoning is desired. Pass both `<|endoftext|>` and `<|im_end|>` as
EOS IDs. Keep the combined prompt and generated sequence within **32K tokens**,
the fixed base model configuration limit, unless a separate long-context
extension is validated.

## Development evaluation

On the corrected EvalScope Full1055 development suite, the preregistered
`seed=3407` `code_only` result was **266/1055 = 25.21%**. Independent forward
and reverse scoring produced **0 verdict flips**.

This is a **single-seed development screening result**, not formal
confirmation, a held-out estimate, or a multi-seed aggregate. No aggregate from
`A-NH025` is attributed to this adapter.

## Limitations

- This adapter requires the exact base model and should not be loaded alone.
- The published evidence is development-only and single-seed.
- Generated code can be incorrect, insecure, or non-compiling; sandbox and
  test it independently.
- No production safety, security, or suitability certification is implied.

## License

The fixed base card and included license identify Apache-2.0. This adapter
preserves that license text and metadata. Users should independently verify the
upstream Qwen3 license, notices, training-data terms, and applicability to their
use case.