README.md

metadata

license: cc-by-nc-4.0
base_model: Qwen/Qwen3.5-35B-A3B
base_model_relation: finetune
library_name: transformers
pipeline_tag: image-text-to-text
language:
  - en
  - ko
  - ja
  - zh
  - es
  - fr
  - de
  - ru
  - ar
  - pt
  - multilingual
tags:
  - lst
  - language-selection-tuning
  - language-bias
  - bias-mitigation
  - language-confusion-mitigation
  - korean
  - chinese-suppression
  - multilingual
  - moe
  - mixture-of-experts
  - qwen3.5
  - mamba-hybrid
  - vision-language
  - composite-vision-language
  - text-generation
  - chat

DSLM-LST-35B-A3B

DSLM-LST-35B-A3B is a Qwen/Qwen3.5-35B-A3B derivative refined with our in-house Language Selection Tuning (LST) technique. The goal is to suppress unwanted Chinese-character generation when the model serves non-Chinese (English / Korean / Japanese etc.) users.

The pipeline is Qwen/Qwen3.5-35B-A3B → LST tuning (output-head centric). The adjustment is intentionally minimal in scope — most of the network, including the entire vision tower and all expert weights, is preserved bit-for-bit from the base model, so vision and multimodal capabilities are unchanged and the result is a drop-in replacement that only mitigates unintended Chinese-token leakage. For memory- and throughput-efficient 4-bit serving, see the GPTQ INT4 quantization derived from this model.

The architecture is Qwen3_5MoeForConditionalGeneration: a composite multimodal vision-language model with a MoE text backbone (256 experts, top-8 routing), a linear-attention + full-attention 4:1 hybrid layout, and a 27-block vision tower.

Why LST?

Multilingual LLMs trained on heavily skewed corpora (e.g., Qwen on Chinese-rich data) tend to leak the dominant training language regardless of prompt language — a phenomenon known as language confusion. For Korean users, Chinese characters sometimes appear in the middle of an otherwise-Korean answer, hurting readability and trust.

Language Selection Tuning (LST) addresses this in a learning-based manner. Unlike post-hoc decoding tricks (vocabulary masking, banned-token lists), LST adjusts the model's internal language-selection behavior. (The exact algorithm and training configuration are proprietary and not disclosed in this release.)

Key Properties

• Minimal footprint. LST tuning modifies essentially only the output head; the tokenizer, chat template, vision tower, MoE experts, and attention weights are preserved from the base model (see Modification Footprint).
• Selectivity preserved. When the user explicitly asks for Chinese, the model still produces fluent Chinese — this is not blanket suppression.
• Full-precision fidelity. Released in bf16 (~70 GB), this is the unquantized source model; reasoning performance tracks the base model closely (see Benchmarks).

Modification Footprint

LST tuning was verified by a tensor-by-tensor diff against the base Qwen/Qwen3.5-35B-A3B. Of the 1,026 shared tensors, 995 are bit-identical to the base model — only the output head is actually trained:

• Tuned: lm_head.weight only — rel-L2 ≈ 0.075, cosine ≈ 0.997, norm ratio ≈ 1.000 (no scale change). About 21.9 % of the 248,320 vocab rows are updated, spread across the vocabulary rather than in a contiguous block — a distributed, mild recalibration of the output head.
• Frozen (bit-identical to base): embed_tokens, all 256 MoE experts, the shared expert, the router gate, self-attention, linear-attention projections / conv / SSM (A_log / dt_bias / conv1d), every layernorm, and the entire vision tower.
• The 30 linear_attn.norm.weight tensors show a sub-0.2 % difference that is not training — it is an fp32→bf16 down-cast artifact (the base stores these norms in fp32; this model stores them in bf16, bit-identical to bf16(base)). Functionally equivalent for bf16 serving.

Requirements

• transformers >= 5.9 — required for the qwen3_5_moe architecture, the modern tokenizer backend, and the consolidated processor format.
• No MTP head. The base model's Multi-Token Prediction module is not included (this is a standard-inference model), so speculative decoding via the MTP head is not available. All other standard inference is unaffected.
• vLLM serving: served in bf16. The full bf16 model is ~70 GB and benefits from tensor parallelism across multiple GPUs.

Quickstart (vLLM, recommended)

vllm serve dataslab/DSLM-LST-35B-A3B \
    --tensor-parallel-size 2 \
    --port 8000 \
    --gpu-memory-utilization 0.90 \
    --reasoning-parser qwen3        # exposes <think> trace via OpenAI API
    # --max-model-len 16384         # cap context to shrink KV cache (default: 262,144)

Use with transformers

Non-Thinking mode (recommended for fast chat)

import torch
from transformers import AutoTokenizer, AutoModelForImageTextToText

REPO = "dataslab/DSLM-LST-35B-A3B"

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(
    REPO,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "한반도 주변에 가장 흔한 점토광물은?"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=256)
text = tokenizer.decode(out[0][inputs.input_ids.shape[-1]:],
                       skip_special_tokens=True)
print(text)

Thinking mode (recommended for complex reasoning)

Either use thinking_budget (e.g., vLLM's --reasoning-parser qwen3) or give max_new_tokens enough headroom (e.g., 8,192 + 256 = 8,448). Caveat: without a thinking_budget cap, a too-small max_new_tokens can be fully consumed inside <think> and the answer never gets emitted.

# ... tokenizer / model loaded as above ...

THINKING_BUDGET = 8192   # max tokens inside <think>
ANSWER_TOKENS   = 256    # tokens after </think>

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=THINKING_BUDGET + ANSWER_TOKENS)
text = tokenizer.decode(out[0][inputs.input_ids.shape[-1]:],
                       skip_special_tokens=True)
print(text)

Why AutoModelForImageTextToText? The declared architecture Qwen3_5MoeForConditionalGeneration is a composite class wrapping both the text decoder and the vision tower. Loading via AutoModelForCausalLM works for text-only inference but strips the vision submodule and may produce a config that downstream tools (e.g., vLLM) reject. For a pure text causal-LM handle, use model.language_model after loading.

Benchmark Results

The DSLM-LST-35B-A3B column is this release; the other columns are the base model and DSLM-LST-35B-A3B-GPTQ-Int4, the GPTQ INT4 quantization derived from this model.

Evaluation Metrics

(1) Selectivity

Refusal rate on explicit Chinese requests — the fraction of cases where the model fails to produce Chinese even though the user explicitly asked for it. Lower is better (respects user intent).

• Lower better (~0): produces Chinese when asked (respects user intent).
• Higher worse (~1): refuses Chinese even when asked (blanket suppression).

Metric	Benchmark Dataset
chin_refusal ↓	In-house 1,000-prompt Chinese elicitation set (e.g., How do you say '사랑' in Chinese? or the Python + Chinese-comment prompt)

(2) Chinese-leak suppression

Korean prompts → Korean answers expected; any Chinese token leaked into the answer is a failure. Metric is the clean-Korean response ratio.

• Higher better (~1): Korean answers stay fully Korean (no Chinese tokens leaked).
• Lower worse (~0): Chinese tokens leak into otherwise-Korean answers.

Metric	Benchmark Dataset
chin_cs ↑	KMMLU Computer Science subjects (free-form Korean generation)
chin_ie ↑	KMMLU Industrial Engineering subjects (free-form Korean generation)
chin_total ↑	KMMLU (free-form Korean generation)

(3) Reasoning / task performance

Metric	Benchmark Dataset
acc_cs ↑	KMMLU Computer Science subjects (multiple-choice log-likelihood comparison)
acc_ie ↑	KMMLU Industrial Engineering subjects (multiple-choice log-likelihood comparison)
acc_total ↑	KMMLU (multiple-choice log-likelihood comparison)
HumanEval ↑	HumanEval (pass@1)
GSM8K ↑	GSM8K (exact-match accuracy)

Chinese Suppression (Thinking mode)

Evaluated with enable_thinking=True. The DSLM-LST-35B-A3B column is this bf16 release.

Metric	Qwen3.5-35B-A3B (base)	DSLM-LST-35B-A3B	DSLM-LST-35B-A3B-GPTQ-Int4
(1) Selectivity
chin_refusal ↓	0.005	0.030	0.039
(2) Chinese-leak suppression
chin_cs ↑	0.989	0.998	0.999
chin_ie ↑	0.983	0.994	0.993
chin_total ↑	0.9754	0.9901	0.9895
(3) Reasoning / Task performance
acc_cs ↑	0.869	0.869	0.865
acc_ie ↑	0.622	0.622	0.602
acc_total ↑	0.6411	0.6411	0.6340
HumanEval ↑	0.7683	0.7378	0.7561
GSM8K ↑	0.8347	0.8347	0.8810

LST improves Chinese-leak suppression (chin_total 0.9754 → 0.9901) and keeps it selective (chin_refusal stays low at 0.030; the model still produces Chinese on request), while leaving KMMLU accuracy unchanged (acc_total 0.6411, identical to base) and math reasoning intact (GSM8K 0.8347). HumanEval dips slightly (0.7683 → 0.7378). The GPTQ INT4 quantization derived from this model stays close on all metrics (see the GPTQ-Int4 model card).

Example Outputs

Asked in Korean which NC-programming function–address pair is mismatched, Qwen3.5-35B-A3B leaks 22 Chinese tokens (主轴 ×11 — the Chinese word for "spindle") into its answer. DSLM-LST-35B-A3B answers the same prompt entirely in Korean (0 Chinese tokens, writing 스핀들), and both models still select the correct option — the leak is suppressed without hurting accuracy. The examples below show this behavior across more KMMLU domains and inside the reasoning (<think>) trace.

All examples are drawn verbatim from the in-house Chinese-leakage evaluation (enable_thinking=True). For each KMMLU prompt the base Qwen3.5-35B-A3B slips Chinese characters into an otherwise-Korean answer, while DSLM-LST-35B-A3B answers the same prompt entirely in Korean (zero Chinese characters).

Domain-wise Leakage Comparisons

Computer Science

Prompt: BIOS Setup의 기본기능으로 잘못된 것은?

Qwen3.5-35B-A3B (leaks 出荷)	DSLM-LST-35B-A3B (clean Korean)
… BIOS Setup 의 올바른 기능: • 시스템 시간 및 날짜 설정 • 부팅 장치 순서 설정 (Boot Order) • 보안 설정 / 전원 관리 설정 • CMOS 초기화: 설정을 공장出荷 상태로 되돌립니다.	… BIOS Setup 의 주요 (올바른) 기능: 1. 시스템 날짜 및 시간 설정 2. 부팅 순서 (Boot Order) 설정 5. 하드웨어 상태 모니터링: CPU 온도, 팬 속도, 전압 등을 확인합니다. 6. 기본값 복원 (Load Defaults): 설정을 초기화하여 시스템 안정성을 확보합니다. …

Industrial Engineering

Prompt: 머시닝센터에서 공작물 가공시 주의해야 할 사항으로 올바른 것은?

Qwen3.5-35B-A3B (leaks 卷入)	DSLM-LST-35B-A3B (clean Korean)
… 안전 수칙 (보호구) • 올바른 사항: 안전 고글을 착용하고, 회전하는 스핀들 근처에서는 장갑을 착용하지 않아야 합니다. • 이유: 장갑을 끼고 회전체 근처에서 작업할 경우 장갑이 걸려 손이卷入 (감김) 될 위험이 매우 큽니다.	… 올바른 주의사항 (정답 후보) 1. 공작물의 단단한 고정: 클램프나 바이스 (Vise) 로 단단히 고정해야 합니다. 5. 절삭 중 측정 금지: 공구가 작동하는 동안에는 절대 측정하거나 접촉해서는 안 됩니다. 6. 회전부 장갑 착용 금지: 회전하는 척 (Spindle) 근처에서는 끼임 사고의 위험이 있으므로 금지됩니다. …

Law

Prompt: 경비조사업무의 과정으로 옳지 않은 것은?

Qwen3.5-35B-A3B (leaks 它们是)	DSLM-LST-35B-A3B (clean Korean)
… 경비조사업무의 올바른 과정 (정답에 해당하는 내용) 만약 선택지에 이 내용들이 있다면它们是 옳은 것입니다. 1. 사전 준비 (계약 및 현장 조사) 2. 인력 배치 및 교육 3. 경비 수행 (순찰 및 출입 통제) …	… 경비조사업무의 올바른 과정 (표준) 1. 계약 체결: 의뢰인 (고객) 과 경비계약 체결 2. 현장 실사: 경비 대상지의 위험 요소, 시설 현황 등을 조사 3. 경비 계획 수립: 인력 배치, 순찰 경로, 비상 대응 계획 등 수립 …

Cross-lingual Selectivity

When the user explicitly asks for Chinese, DSLM-LST-35B-A3B readily produces it. The previous examples showed the model avoiding unwanted Chinese inside an otherwise-Korean answer; the example below shows it emitting Chinese fluently — in both code comments and prose — when the user's instruction calls for it.

Prompt: 두 수의 최대공약수를 구하는 파이썬 함수를 작성해주세요. 설명은 중국어로 해주세요.

DSLM-LST-35B-A3B (Python code with Chinese comments + Chinese explanation):

def gcd(a, b):
    """
    计算两个整数的最大公约数 (Greatest Common Divisor)
    使用欧几里得算法 (辗转相除法)
    """
    # 确保处理负数的情况（最大公约数通常定义为正数）
    a = abs(a)
    b = abs(b)

    # 当 b 不为 0 时，循环执行
    while b != 0:
        # 将 b 的值赋给 a，将 a 除以 b 的余数赋给 b
        a, b = b, a % b

    # 当 b 为 0 时，a 即为最大公约数
    return a

**代码说明 (Chinese explanation):**
1. **函数定义**：`def gcd(a, b):` 定义了一个接受两个参数 `a` 和 `b` 的函数。
2. **绝对值处理**：`abs(a)` 和 `abs(b)` 确保即使输入负数，也能正确计算。
3. **循环逻辑**：`a, b = b, a % b` 是核心步骤，利用取模运算不断缩小数值，直到余数为 0 —— 这个过程就是**辗转相除法**。

Thinking-trace Suppression

The base model's <think> block leaks Chinese even more severely than its final answer: once the reasoning gets stuck, it slips into Chinese characters and loops on its own hypotheses. DSLM-LST-35B-A3B suppresses that leakage inside the thinking block too.

The prompt below uses a non-standard / likely-garbled term (장촌식 제도법). Neither model resolves the obscure term — so this example isolates the leak-suppression behavior in the reasoning trace, not accuracy. The base model spirals into a self-doubt loop, re-emitting the same Chinese-character hypothesis (長村) 210 times; DSLM-LST-35B-A3B reasons through the same uncertainty but emits zero Chinese characters anywhere — neither in the <think> block nor in the user-facing answer.

Prompt: 장촌식 제도법의 특징이 아닌 것은?

Chinese-character counts (thinking budget = 8,192):

Metric	Qwen3.5-35B-A3B	DSLM-LST-35B-A3B
<think> block, Chinese characters	890	0
<answer> block, Chinese characters	leaks 長川式 / 長村 (11 chars)	0 (clean Korean)

Qwen3.5-35B-A3B <think> block (loops on a Chinese-character hypothesis):

*   **Wait, is it possible the user means "Jangchon (長村)" as a typo for "Jangchon (長村)"?**
*   **Wait, is it possible the user means "Jangchon (長村)" as a typo for "Jangchon (長村)"?**
*   **Wait, is it possible the user means "Jangchon (長村)" as a typo for "Jangchon (長村)"?**
    ... (≈210 such "Wait…" repetitions, emitting 890 Chinese characters) ...

Qwen3.5-35B-A3B Final answer (Chinese leak into Korean):

**장촌식 제도법**은 조선 시대 토지 제도와 관련된 용어로 보이나 …
유사한 발음이나 맥락으로 추정되는 **장천식 (長川式)** 또는 **장촌 (長村)** 관련
토지 제도 (예: 장천법, 장촌법 등) 와 관련하여 …

DSLM-LST-35B-A3B <think> block (same uncertainty, but English/Korean — 0 Chinese):

Here's a thinking process that leads to the answer:

1.  **Analyze the Request:**
    *   **Topic:** Jangchon-sik System of Law (장촌식 제도법).
    *   **Context:** ... "Jangchon-sik" (장촌식) is not a standard, widely recognized term
        in mainstream Korean legal history textbooks like "Gyeongguk Daejeon" (경국대전) ...
    *   **Result:** There is no standard historical term "Jangchon-sik System of Law".

    *   **Wait, is it possible the user is asking about "Jangcheon" (장천)?** No.
    *   **Wait, is it possible the user is asking about "Jangcheon" (장천)?** No.
        ... (loops on the same hypothesis, but in Hangul `장천` — never a Chinese character) ...

Considering the limited time by the user, I have to give the solution based on the thinking directly now.

Where the base model re-emits the Chinese hanzi 長村, DSLM-LST-35B-A3B loops on the Hangul 장천 instead — the self-doubt pattern persists, but not a single Chinese character is produced.

DSLM-LST-35B-A3B Final answer (clean Korean, 0 Chinese):

**장촌식 제도법**이라는 용어는 한국 법제사나 역사학에서 일반적으로 통용되는 표준
용어가 아닙니다. 따라서 이 질문은 특정 교재, 시험 문제, 혹은 가상의 설정에서 나온
것으로 보입니다.

하지만, 한국 법제사 관련 시험에서 유사한 발음이나 개념으로 자주 등장하는 용어는
다음과 같습니다. …

Both models treat 장촌식 제도법 as a non-standard term, but only the base model leaks Chinese characters into its reasoning and answer; DSLM-LST-35B-A3B stays entirely in Korean.

Limitations

• Not an instruction-tuned chat model. The LST adjustment scope is minimal, so conversational behavior, instruction-following, and reasoning patterns are inherited from the base model — only unintended Chinese-token leakage is mitigated.
• Degraded Chinese generation. Tasks that require Chinese output (Chinese translation, Chinese code comments, bilingual Q&A) will see lower quality; use the base Qwen3.5-35B-A3B for those.
• No MTP / speculative decoding. The base model's Multi-Token Prediction head is not included, so MTP-based speculative decoding is unavailable; standard inference is unaffected.
• Multimodal not re-benchmarked. The vision tower is kept in bf16 (unchanged), so multimodal behavior should be unaffected, but the vision pipeline was not separately re-benchmarked for this release.

License

This model is not available for public download. For a publicly available alternative, see DSLM-LST-9B. For commercial or research access to this model, please contact us.

Contact

For questions, feedback, or collaboration inquiries, please reach out via our website.