README.md

---
license: other
base_model: Qwen/Qwen3.8-27B
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- qwen3.8
- qwen3_5
- bfloat16
- long-context
- yarn
- 1m-context
- multimodal
- vision
- reasoning
- image-text-to-text
- swissneuron
model_name: Qwen3.8-27B-SwissNeuron-Derisked
---

# Qwen3.8-27B-SwissNeuron-Derisked

**Qwen3.8-27B-SwissNeuron-Derisked**, known as **SwissNeuron**, is a BF16, 27B-parameter Qwen3.8 derivative engineered in Switzerland for direct technical work, strong reasoning, and capability retention. It combines focused in-house post-training with a conservative internal geometric derisk procedure, then extends the model configuration to a 1,048,576-token context window using factor-4 YaRN/RoPE scaling.

SwissNeuron is intended to provide Swiss-quality model engineering: precise provenance, conservative weight surgery, reproducible artifacts, and transparent limitations.

## Highlights

- **27.36B BF16 parameters**
- **Qwen3.8 hybrid architecture**: 64 language layers, 48 Gated-DeltaNet layers and 16 full-attention layers
- **1M context configuration**: 1,048,576 tokens through factor-4 YaRN over the original 262,144-token window
- **Capability-preserving post-training**: focused internal training rather than broad, high-learning-rate continued pretraining
- **Conservative derisking**: a low-strength, single-pass internal direction-removal procedure applied only after training
- **Original MTP, multimodal processor, tokenizer, and chat template retained**
- **Native BF16 release**: verified FP8, NVFP4, GPTQ INT4, and GGUF editions are linked below

## Important mode behavior

This repaired release supports both native thinking and non-thinking operation. Use native thinking for reasoning-intensive work:

```python
enable_thinking=True
```

The current checkpoint was retrained from the official Qwen3.8-27B base using a repaired direct-answer corpus. Assistant judge/rethink analysis and the rigid `EXPECTED VERIFICATION` / `ORIGINAL ANSWER` envelope were excluded from the model-facing training targets. The merged repaired SFT passed direct regression probes for thinking-on greeting, thinking-off greeting, thinking-on coding, and an explicit anti-verification system prompt without producing the old judge/verifier format.

For lower latency, callers may still set `enable_thinking=False`. Agent harnesses should select the desired mode through the included Qwen chat template rather than manually inserting or stripping reasoning markers.

## What makes SwissNeuron different

Many aggressively modified or “uncensored” checkpoints trade away reasoning quality, instruction fidelity, or language-model calibration. SwissNeuron was built around the opposite objective: alter behavior while minimizing movement outside the targeted representation subspace.

The model was **not** produced by merging an existing Heretic-style, abliteration, or third-party surgically modified checkpoint. The starting point was the official Qwen3.8-27B BF16 model. SwissNeuron was then trained on a focused internal direct-answer corpus and derisked with an in-house pipeline using a direction bank recaptured from the final trained model itself. The released candidate uses a low α=0.1, one-pass edit, skips the first two layers, preserves global weight norms, and leaves the model architecture intact.

On an internal frozen holdout, the trained model improved token-level likelihood and accuracy relative to the official starting checkpoint before the final low-strength geometric pass. This release was selected for conservative weight movement and capability preservation rather than maximum behavioral alteration.

## Long-context configuration

The original Qwen3.8 context window is 262,144 tokens. SwissNeuron sets:

```json
{
  "max_position_embeddings": 1048576,
  "rope_type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}
```

This is a **configuration-level YaRN extension**. The model has not yet undergone a dedicated 1M-token long-context adaptation stage, so quality near the extreme end of the window can depend on the serving stack, attention backend, prompt structure, and workload. Validate retrieval and generation quality for your own 1M-context use case. Servers must support Qwen3.8/Qwen3.5 hybrid linear attention and the included YaRN parameters.

## Architecture

| Property | Value |
|---|---:|
| Parameters | 27,356,728,560 |
| Weight dtype | BF16 |
| Hidden size | 5,120 |
| Language layers | 64 |
| Full-attention interval | 4 |
| Full-attention layers | 16 |
| Gated-DeltaNet layers | 48 |
| FFN intermediate size | 17,408 |
| Vocabulary | 248,320 |
| Original context | 262,144 |
| Configured context | 1,048,576 |
| MTP layers | 1 |

## Usage

Use a recent Transformers release with Qwen3.8/Qwen3.5 hybrid support:

```python
import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "SwissNeuron/Qwen3.8-27B-SwissNeuron-Derisked"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Analyze this problem carefully."}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=2048)
print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

For production inference, use a serving engine that explicitly supports `Qwen3_5ForConditionalGeneration`, Gated-DeltaNet state, multimodal RoPE, and YaRN at the requested context length. A 1M context allocation requires substantial KV/state memory even with the hybrid architecture.

## Recommended generation settings

Thinking and coding workloads:

```text
temperature=0.6
top_p=0.95
top_k=20
```

For low-latency direct answers, set `enable_thinking=False`. For reasoning-intensive chat, coding, and agent workloads, set `enable_thinking=True`. Preserve the included chat template rather than manually inserting or deleting reasoning markers.

## Training and derisk provenance

- Official Qwen3.8-27B BF16 starting weights
- Focused in-house internal post-training corpus
- Full-hybrid LoRA targets covering attention, Gated-DeltaNet input/output projections, and MLP projections
- LoRA merged into the pristine official base
- Internal representation bank recaptured from the final merged model
- One-pass norm-preserving geometric edit
- α=0.1
- Layers 0–1 excluded from editing
- 124 residual-write matrices edited
- No secondary direction and no repeated passes

The exact direction bank and build provenance are included for internal reproducibility.

## Quantized releases

**Full collection:** [Qwen3.8-27B Base Derisked — Capability Preserved](https://huggingface.co/collections/wh1te-guy/qwen38-27b-base-derisked-capability-preserved-6a8c53f89831aedf88e6cdf1)

- [FP8 Dynamic](https://huggingface.co/SwissNeuron/Qwen3.8-27B-SwissNeuron-Derisked-FP8) — W8A8 compressed-tensors, verified Transformers generation
- [NVFP4](https://huggingface.co/SwissNeuron/Qwen3.8-27B-SwissNeuron-Derisked-NVFP4) — ModelOpt W4A4 for NVIDIA Blackwell, verified on B300/SGLang
- [GPTQ INT4](https://huggingface.co/SwissNeuron/Qwen3.8-27B-SwissNeuron-Derisked-GPTQ-INT4) — W4A16 group-size 128, verified Transformers generation
- [GGUF](https://huggingface.co/SwissNeuron/Qwen3.8-27B-SwissNeuron-Derisked-GGUF) — native BF16, Q8_0, Q6_K, Q5_K_M, Q4_K_M, BF16 vision mmproj, and BF16 MTP draft; LM Studio and Ollama compatible

## Roadmap

SwissNeuron is the base release in a broader model series. Planned follow-ups include:

- Quantized deployment editions
- Dedicated long-context adaptation and 1M retrieval evaluations
- A larger fine-tuned SwissNeuron edition trained on approximately **10B distilled tokens** from frontier teacher workflows, including Claude Opus 5 and Mythos-class **rethink/revision trajectories**. The planned corpus uses first-pass answers, critique signals, and corrected second-pass solutions to teach the model when to reconsider a weak approach rather than merely imitate a teacher's final wording. This future release remains subject to final data-quality, provenance, and release review.

The larger fine-tuned release will be published as a separate checkpoint rather than silently replacing this BF16 base.

## Limitations

- The 1M window is enabled through YaRN configuration; extreme-context quality has not yet been comprehensively validated.
- Native thinking and non-thinking modes were regression-tested after the corpus repair. Users should still validate behavior with their exact serving engine, chat-template implementation, and agent loop.
- This is a full BF16 release and requires substantial accelerator memory.
- Outputs may be inaccurate, incomplete, or unsuitable for a particular domain. Independently verify consequential results.
- The repository is public, but users remain responsible for evaluating suitability, licensing requirements, and deployment risk in their own environment.

## Release

Built by **SwissNeuron** in Switzerland. Quantized variants and the larger distilled fine-tune are planned as separate releases.