DiffusionGemma Humanizer β Architecture & Strategy
Date: 2026-06-29 Budget: $250 Modal credits Target: GPTZero, GPTDetector, Pangram, Binoculars Objective: Fine-tune DiffusionGemma to humanize AI-generated text and evade detection
1. Why Diffusion Models Beat Detectors (Paper-Backed)
1.1 The Theoretical Ceiling (Sadasivan et al., 2023)
Theorem 1 proves mathematically: AUROC β€ 1/2 + TV(M,H) β TV(M,H)Β²/2, where TV(M,H) is the total variation distance between machine and human text distributions. As models improve and TV β 0, every detector converges to random guessing (AUROC β 0.5). This is not a technological limitation β it's a mathematical certainty. [arXiv:2303.11156]
Implication: 0% detection (AUROC = 0.5) is achievable asymptotically. A model that perfectly mimics human text distribution is undetectable by definition. We don't need perfection today β we just need TV(M,H) to be lower than what detectors have been trained to distinguish.
1.2 Pangram's Blind Spot: No Diffusion Text
Pangram's DAMAGE detector (Mistral NeMo 12B + LoRA + linear head) was trained exclusively on autoregressive text: GPT-3.5/4/4o, Claude 2/3, LLaMA 2/3/3.1, Mistral, Gemini Pro/Flash. Their "synthetic mirror prompts" all target instruction-tuned autoregressive models. Zero diffusion-generated text in their training set. [arXiv:2501.03437]
Key finding from their adversarial attack (Β§7):
- When they fine-tuned GPT-4o specifically against their detector: TPR dropped from 100% β 93.2%
- This was with a generic fine-tune API, not a custom architecture
- A purpose-built diffusion humanizer would exploit a fundamentally different statistical pathway they've never seen
1.3 Adversarial Paraphrasing Works (Cheng et al., 2025)
The closed-loop optimization is the secret: paraphrasing WITHOUT detector guidance increases detection by 8-15%. With detector feedback guiding the paraphrase, TPR drops by 87.88% average. The innovation is in the feedback loop, not the base model. [arXiv:2506.07001]
Our advantage: We can embed this feedback loop inside the diffusion denoising process, eliminating the need for an external autoregressive LLM that would leave AR-specific traces.
1.4 Diffusion Text Naturally Evades AR-Trained Detectors (TarΔ±m & Onan, 2025)
LLaDA-generated text "closely mimics human text in perplexity and burstiness, yielding high false-negative rates for AR-oriented detectors." Diffusion models generate through iterative denoising β left-to-right token probabilities don't exist, so perplexity-based methods break. [HF Papers:2507.10475]
2. Architecture Proposal
2.1 Base Model: DiffusionGemma 26B
| Property | Value |
|---|---|
| Total params | 25.2B |
| Active params | 3.8B (MoE, 8/128 experts + 1 shared) |
| Generation | Discrete diffusion, block-autoregressive, 256-token canvases |
| Attention | Bidirectional in decoder, cross-attention to AR encoder |
| Sampling | Entropy-Bounded Denoising, 1β48 steps, temperature 0.8β0.4 |
| License | Apache 2.0 |
| Release | June 9, 2026 |
Why this model:
- Generation pathway is fundamentally different from AR models β OOD for all current detectors
- Bidirectional canvas attention enables holistic text understanding for rewriting
- Iterative denoising (1-48 steps) provides a perfect mechanism for progressive humanization
- MoE (3.8B active) makes QLoRA fine-tuning feasible on A100
- 4-bit quantized versions already exist (NVFP4 from NVIDIA, AutoRound from Intel)
2.2 Training Strategy
Phase 1: Dataset Construction
- Base: HC3 (48.6K ChatGPT-human QA pairs)
- Augmentation: Synthetic pairs from heuristic humanization (sentence splitting, hedging, register variation)
- Detector scoring: Each pair annotated with pre/post scores from Fast-DetectGPT, burstiness, heuristics
Phase 2: QLoRA Fine-Tuning
- Method: QLoRA, rank 8-16, 4-bit base (NVIDIA NVFP4)
- Format: Chat template (system: "Rewrite to sound human", user: AI text, assistant: human text)
- Initial scale: 300-500 pairs, A100 80GB, ~$40
Phase 3: Detector-Guided Refinement
- At inference, use entropy-bounded denoising (48 steps max)
- Apply classifier-free guidance to push logits toward human-like tokens
- Target inversion of known detector signals:
- Reduce transition markers (furthermore, however, therefore)
- Increase sentence length variation
- Reduce passive voice constructions
- Increase hapax legomena ratio
2.3 Detector Signals to Invert
From the literature, detectors (especially Pangram) target:
| Signal | AI Text | Human Text | Inversion Strategy |
|---|---|---|---|
| Perplexity | Low (<20) | High (>30) | Add "surprising" tokens |
| Burstiness | Low (<0.15) | High (>0.3) | Vary sentence structure |
| Transition markers | Many | Few | Replace with informal alternatives |
| Passive voice | Many | Few | Convert to active voice |
| Sentence length Ο | Low (<5) | High (>10) | Mix short/long sentences |
| Hapax ratio | Low (<0.40) | High (>0.45) | Use more unique words |
| Lexical diversity | Moderate | Variable | Match domain-specific patterns |
3. Cost Breakdown
| Step | GPU | Est. Time | Cost |
|---|---|---|---|
| 1. Model download + baseline gen | A100 | 30 min | $1.25 |
| 2. Detector tests | A100 | 20 min | $0.85 |
| 3. Dataset construction | CPU (within A100) | 10 min | $0 |
| 4. QLoRA fine-tuning (300 pairs) | A100 | 1-2h | $5-10 |
| 5. Evaluation (5 prompts) | A100 | 15 min | $0.60 |
| 6. Export to HF | CPU | 5 min | $0 |
| Total per run | ~3h | ~$8-13 | |
| Available runs ($250) | 19-31 experiments |
Each experiment can vary: dataset size, LoRA rank, learning rate, prompt template, training epochs.
4. Expected Results (Hypothesis)
Based on cross-referencing all papers:
| Detector | Baseline (DiffusionGemma raw) | Fine-tuned Humanizer |
|---|---|---|
| GPTZero | 60-80% evasion (diffusion confuses perplexity) | 95%+ evasion |
| Binoculars | 50-70% evasion (cross-perplexity mismatch) | 90%+ evasion |
| Fast-DetectGPT | 40-60% evasion (curvature is AR-specific) | 85%+ evasion |
| Pangram | 30-50% evasion (untrained on diffusion) | 70-85% evasion initially |
Pangram will be the hardest because their deep classifier learns nuanced style patterns. But being completely OOD (out-of-distribution) for their training set gives a significant initial advantage. The window of high evasion will close once Pangram adds diffusion text to their training pipeline β estimated 1-3 months after public release.
5. Security: HF Token Setup
Required Permissions
- Read token: Download models (DiffusionGemma, GPT-2 for detectors)
- Write token: Upload to
simonlesaumon/diffusiongemma-humanizer
Setup on Modal (nothing stored locally)
# Create Modal secret (tokens stay encrypted on Modal, never on your PC)
modal secret create hf-secrets HF_TOKEN=hf_your_token_here
Then the app reads from os.environ["HF_TOKEN"] automatically.
Create your token at: https://huggingface.co/settings/tokens (needs "Write" permission)
6. File Structure
Deepseek_SOTA/
βββ modal_project/
β βββ app.py # Full Modal pipeline
βββ scripts/
β βββ run_pipeline.ps1 # PowerShell launcher
β βββ run.py # Python launcher
βββ research/
β βββ technical-diffusion-text-humanization-2026-06-29.md # Research report
β βββ architecture-strategy.md # This file
βββ .gitignore
Everything runs on Modal. Nothing downloaded locally except the scripts.
7. References
- Sadasivan et al. (2023) β Can AI-Generated Text be Reliably Detected? [arXiv:2303.11156]
- Masrour, Emi & Spero (2025) β DAMAGE: Detecting Adversarially Modified AI Generated Text [arXiv:2501.03437]
- Cheng et al. (2025) β Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text [arXiv:2506.07001]
- TarΔ±m & Onan (2025) β Can You Detect the Difference? Diffusion vs AR Text [HF:2507.10475]
- Ranganath & Ramesh (2026) β StealthRL: RL Paraphrase Attacks for Multi-Detector Evasion [arXiv:2602.08934]
- Xu et al. (2026) β Base Models Look Human To AI Detectors [arXiv:2605.19516]
- Schiff et al. (2024) β Simple Guidance Mechanisms for Discrete Diffusion Models [arXiv:2412.10193]
- DiffusionGemma β Hugging Face