README.md

---
license: apache-2.0
language:
  - en
  - ru
base_model: Qwen/Qwen3.5-9B-Base
library_name: peft
pipeline_tag: text-generation
tags:
  - qwen
  - qwen3.5
  - lora
  - peft
  - qlora
  - code
  - rag
  - instruction-following
  - translation
  - game-localization
---

# Qwen Existence Code

Qwen Existence Code is a PEFT LoRA adapter for `Qwen/Qwen3.5-9B-Base`.

The adapter is intended to reinforce practical instruction following, complete technical answers, code-oriented assistance, RAG-driven workflows, and context-aware translation/localization. It is not a full standalone model checkpoint. It must be loaded together with the base model.

## Status

This release is an adapter release with training metadata and logs available locally from the training package. It is not yet a benchmarked research release.

What is documented:

- Training data mix and example counts
- Approximate training token count from trainer logs
- LoRA/QLoRA hyperparameters
- Training steps, runtime, checkpoints, and final training loss
- Reproducible training script structure

What is not yet proven by this release:

- It has not yet been independently benchmarked against the base model.
- It has not yet been evaluated on HumanEval, MBPP, SWE-bench, translation benchmarks, or general capability regression tests.
- It should not be claimed to be globally better than `Qwen/Qwen3.5-9B-Base` until a separate evaluation report is added.

## Model Details

- Model name: `Qwen Existence Code`
- Repository name: `Qwen3.5-9B-Existence-Code`
- Base model: `Qwen/Qwen3.5-9B-Base`
- Model type: PEFT LoRA adapter
- Training method: QLoRA SFT
- Adapter file: `adapter_model.safetensors`
- Adapter size: about 58 MB
- License: Apache-2.0
- Primary languages: English and Russian

## Files

```text
README.md
adapter_model.safetensors
adapter_config.json
tokenizer.json
tokenizer_config.json
chat_template.jinja
training_args.bin
```

## Intended Use

Typical use cases:

- Code generation and code completion
- Refactoring and debugging assistance
- Technical documentation
- RAG-based project assistants
- Automation planning
- Game localization and dialogue translation
- Context-aware translation where tone, slang, mature language, or character voice must be preserved

For localization tasks, the adapter is intended to preserve the meaning and tone of source material when the user has the right to process that content. Fictional game dialogue may include profanity, slang, dark humor, or mature language, and removing it can damage the original intent.

## Training Data

The training set was built as a mixed SFT JSONL file:

```text
data/sft_mix.jsonl
```

Observed dataset statistics:

| Source | Examples |
|---|---:|
| `microsoft/orca-agentinstruct-1M-v1` | 500 |
| `HuggingFaceH4/ultrafeedback_binarized` | 250 |
| `nvidia/Nemotron-SFT-OpenCode-v1` | 250 |
| `synthetic_complete_code` | 80 |
| **Total** | **1080** |

Approximate text size:

```text
examples: 1080
characters: 3,985,263
average characters/example: 3,690.1
```

The trainer log reported approximately:

```text
num_tokens: 3.899e+06
```

at the final training step.

### Dataset Filtering

The dataset preparation script filtered out examples containing placeholder or incomplete-answer patterns, including:

```text
TODO
insert your code here
your code here
left as an exercise
you can continue
and so on
etc.
...
implement the rest
fill in
placeholder
```

Synthetic examples were added to emphasize complete runnable code and avoidance of placeholder-only answers.

## Training Procedure

Training used QLoRA supervised fine-tuning.

Base model path during training:

```text
/workspace/train/qwen35_diligent_lora/models/Qwen3.5-9B-Base
```

Public base model:

```text
Qwen/Qwen3.5-9B-Base
```

### LoRA Parameters

From `adapter_config.json`:

```text
peft_type: LORA
task_type: CAUSAL_LM
r: 16
lora_alpha: 32
lora_dropout: 0.05
bias: none
use_dora: false
use_rslora: false
```

Target modules:

```text
q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj
```

### Quantization

Training used 4-bit QLoRA:

```text
load_in_4bit: true
bnb_4bit_quant_type: nf4
bnb_4bit_compute_dtype: bfloat16
bnb_4bit_use_double_quant: true
```

### SFT Hyperparameters

```text
max_steps: 600
save_steps: 100
max_length / sequence length: 4096
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
effective batch size: 8
learning_rate: 1.5e-4
warmup_ratio: 0.03
lr_scheduler_type: cosine
logging_steps: 5
save_total_limit: 6
bf16: true
fp16: false
gradient_checkpointing: true
optimizer: paged_adamw_8bit
seed: 42
packing: false
```

### Runtime

Final trainer log:

```text
train_runtime: 6914 seconds
train_samples_per_second: 0.694
train_steps_per_second: 0.087
epoch: 4.444
```

## Loss Curve

The final training loss was:

```text
train_loss: 0.4767
```

Selected logged points from the final stage:

| Step | Epoch | Loss | Mean token accuracy | Learning rate |
|---:|---:|---:|---:|---:|
| 540 | 4.000 | 0.3872 | 0.8982 | 4.029e-06 |
| 545 | 4.037 | 0.2483 | 0.9253 | 3.401e-06 |
| 550 | 4.074 | 0.3037 | 0.9060 | 2.824e-06 |
| 560 | 4.148 | 0.3212 | 0.9060 | 1.829e-06 |
| 570 | 4.222 | 0.2903 | 0.9146 | 1.048e-06 |
| 580 | 4.296 | 0.3849 | 0.9048 | 4.813e-07 |
| 590 | 4.370 | 0.2959 | 0.9217 | 1.322e-07 |
| 600 | 4.444 | 0.3393 | 0.9026 | 1.093e-09 |

Interpretation: the loss indicates that the adapter fit the SFT mixture, but training loss alone does not prove downstream quality. External evaluation is still required.

## Evaluation

No formal benchmark evaluation is included in this release yet.

Recommended evaluation before making quality claims:

### Code

- HumanEval
- MBPP
- LiveCodeBench
- Repo-level patch tests
- Internal unit-test based coding tasks

### Translation and Localization

- Human review on game dialogue samples
- Terminology consistency tests
- Format preservation tests
- Variable/tag preservation tests
- Side-by-side comparison with the base model

### General Regression

- MMLU-style knowledge checks
- GSM/math samples
- Summarization and instruction-following checks
- Refusal/safety behavior checks if deployed publicly

Until these tests are added, the correct claim is:

```text
This adapter was trained to bias Qwen3.5-9B-Base toward more complete technical and localization-style outputs, but benchmarked improvements over the base model have not yet been established.
```

## Reproducibility

The original training package included:

```text
data/sft_mix.jsonl
scripts/prepare_sft_mix.py
scripts/train_qlora.py
scripts/start_train_qlora.sh
logs/train_qwen35_diligent_lora_*.log
outputs/qwen35_diligent_lora_v1/checkpoint-100
outputs/qwen35_diligent_lora_v1/checkpoint-200
outputs/qwen35_diligent_lora_v1/checkpoint-300
outputs/qwen35_diligent_lora_v1/checkpoint-400
outputs/qwen35_diligent_lora_v1/checkpoint-500
outputs/qwen35_diligent_lora_v1/checkpoint-600
outputs/qwen35_diligent_lora_v1/final_adapter
```

Minimal reproduction command:

```bash
source /venv/main/bin/activate
CUDA_VISIBLE_DEVICES=0 python scripts/train_qlora.py \
  --model /path/to/Qwen3.5-9B-Base \
  --dataset data/sft_mix.jsonl \
  --output outputs/qwen35_diligent_lora_v1 \
  --max-steps 600 \
  --save-steps 100 \
  --seq-len 4096 \
  --lr 1.5e-4
```

## Loading With Transformers + PEFT

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "Qwen/Qwen3.5-9B-Base"
adapter_path = "taylonmcfly/Qwen3.5-9B-Existence-Code"

tokenizer = AutoTokenizer.from_pretrained(adapter_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

messages = [
    {
        "role": "system",
        "content": "You are Qwen Existence Code, a precise technical assistant.",
    },
    {
        "role": "user",
        "content": "Write a complete Python script that scans a folder and prints file sizes.",
    },
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.4,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Recommended Prompting

```text
You are Qwen Existence Code, a precise technical assistant.
Follow the user's task completely.
When writing code, provide complete runnable files or patches unless the user explicitly asks for a sketch.
Do not replace required logic with placeholders such as TODO, "insert code here", or "continue yourself".
Use the provided RAG context as the source of truth when it is relevant.
For translation/localization, preserve meaning, tone, character voice, formatting, variables, markup, and speaker intent.
```

## Recommended Generation Settings

Technical work:

```text
temperature: 0.2-0.5
top_p: 0.85-0.95
repetition_penalty: 1.05-1.15
max_new_tokens: high enough for complete output
```

Translation/localization:

```text
temperature: 0.3-0.7
top_p: 0.9
repetition_penalty: 1.05
```

## Limitations

- This is a LoRA adapter, not a full standalone model.
- It depends on `Qwen/Qwen3.5-9B-Base`.
- It can hallucinate or make mistakes.
- Training loss does not prove real-world superiority.
- Formal code, translation, and regression benchmarks are not included yet.
- Human review is recommended for production code and sensitive translations.
- It may require RAG context for project-specific facts.

## Responsible Use

This model is a tool. Users are responsible for their inputs, outputs, and deployment choices.

Do not use this model for illegal activity, unauthorized access, fraud, targeted harassment, or other harmful purposes. For legal, medical, financial, security-critical, or production-sensitive work, outputs should be reviewed by a qualified human.

## Citation

```text
Qwen Existence Code, PEFT LoRA adapter for Qwen/Qwen3.5-9B-Base.
```