Research Report: Fine-Tuning Diffusion Models for Text Humanization

Prepared: 2026-06-29 Scope: Gemma models, diffusion language models, Modal platform costs, text humanization feasibility

1. Google Gemma Models — Sizes and Fine-Tuning Options

Google's Gemma family spans three generations (Gemma 2, 3, and 4), with no diffusion-based variants in any generation. All Gemma models use standard autoregressive (causal) transformer architectures.

Gemma 2 (mid-2024) offers three text-only sizes: Gemma 2 2B, 9B, and 27B, all licensed under the Gemma license and available as both pre-trained (PT) and instruction-tuned (IT) variants google/gemma-2-2b-it, google/gemma-2-9b-it (accessed 2026-06-29, confidence: High).

Gemma 3 (March 2025) expanded to five sizes: 270M (text-only), 1B (text-only), 4B, 12B, and 27B (multimodal, supporting image inputs). Each size comes in both base (PT) and instruction-tuned (IT) variants. The models are gated under the Gemma license and use the gemma3 and gemma3_text architectures google/gemma-3-4b-it (4.3B params), google/gemma-3-12b-it (12.2B params), google/gemma-3-27b-it (27.4B params), google/gemma-3-1b-it (1B params) (accessed 2026-06-29, confidence: High). A Gemma 3n variant (Gemma 3n-E2B) was released in June 2025 adding native audio/video understanding google/gemma-3n-E2B-it (accessed 2026-06-29, confidence: High).

Gemma 4 (March–May 2026) introduced a unified any-to-any architecture with the largest model lineup: 12B, E2B, E4B, 26B-A4B (MoE with 26B total, 4B active), and 31B. These are licensed under Apache 2.0 google/gemma-4-12B-it (12B params), google/gemma-4-26B-A4B-it (MoE), google/gemma-4-31B-it (accessed 2026-06-29, confidence: High).

Fine-tuning options: All Gemma models support standard Hugging Face Transformers fine-tuning pipelines. They are compatible with parameter-efficient methods (LoRA, QLoRA) via the PEFT library, as well as full fine-tuning. The recommended fine-tuning pathway for a 7B-scale text humanization task would be LoRA/QLoRA on a Gemma 3 12B or Gemma 4 12B base model using 4-bit quantization, which requires approximately 8–16 GB of GPU VRAM — well within a single A100 or L40S. However, none of these models use diffusion-based generation, which is the core architectural interest for this project.

2. Diffusion Language Models — Available and Fine-Tunable

The diffusion language model ecosystem has matured rapidly since early 2025. The most relevant models are:

LLaDA 8B (GSAI-ML, February 2025). The foundational large language diffusion model, trained from scratch with masked diffusion and a vanilla Transformer backbone using bidirectional attention. Available in Base and Instruct variants (8.0B parameters, MIT license). It outperforms autoregressive baselines of comparable size on several benchmarks and notably addresses the "reversal curse" that plagues autoregressive models. The model is fully open-source with Hugging Face weights, custom inference code, and a documented SFT pipeline GSAI-ML/LLaDA-8B-Base, GSAI-ML/LLaDA-8B-Instruct (accessed 2026-06-29, confidence: High). The corresponding paper: Large Language Diffusion Models (Nie et al., 2025).

LLaDA 1.5 (May 2025). An improved version of LLaDA 8B with better training recipe and enhanced performance, also 8.0B params, MIT license GSAI-ML/LLaDA-1.5 (accessed 2026-06-29, confidence: High).

LLaDA-MoE 7B-A1B (September 2025). A Mixture-of-Experts variant trained on ~20T tokens. Maintains 7B total parameters while activating only 1.4B during inference, achieving state-of-the-art diffusion LM performance while rivaling Qwen2.5-3B-Instruct LLaDA-MoE paper (accessed 2026-06-29, confidence: High).

iLLaDA 8B (June 2026). The most recent and strongest LLaDA variant, pre-trained on 12T tokens with SFT on a 25B-token instruction corpus for 12 epochs. Improves significantly over LLaDA across benchmarks (e.g., +21.6 points on BBH, +14.5 on MATH) and is competitive with Qwen2.5 7B iLLaDA paper (accessed 2026-06-29, confidence: High). Model weights available at the same GitHub repository as LLaDA.

Dream 7B (DreamLM team, August 2025). The "most powerful open diffusion LLM to date" at its release. Uses discrete diffusion with AR-based LLM initialization and context-adaptive token-level noise rescheduling. Demonstrates superior planning, infilling, and tunable quality-speed trade-offs. Released as Dream-Base and Dream-Instruct Dream 7B paper (accessed 2026-06-29, confidence: Medium — HF model weights not found in search, possibly hosted under a different namespace).

MDLM — Masked Diffusion Language Models (Sahoo et al., June 2024). Demonstrated that simple masked discrete diffusion with modern training practices approaches autoregressive perplexity. The objective is a mixture of classical masked language modeling losses, enabling training of encoder-only models with efficient samplers MDLM paper (accessed 2026-06-29, confidence: High). Code and blog post at s-sahoo.com/mdlm.

Other notable models: GENIE (Microsoft, 2022) — a continuous paragraph denoising pre-training approach; DiffusionBERT (2022) — discrete diffusion initialized from BERT; Diffutron (March 2026) — a Turkish-specific masked diffusion LM demonstrating LoRA-based fine-tuning from a multilingual encoder.

Fine-tuning APIs and requirements: LLaDA and its variants use a standard SFT pipeline based on the Hugging Face Transformers library with custom modeling code. Since diffusion LMs use bidirectional attention rather than causal masking, they require more VRAM during training (full sequence attention). For an 8B model, full fine-tuning in bf16 requires approximately 64 GB VRAM (model weights ~16 GB + optimizer states ~32 GB + activations). With LoRA/QLoRA, this drops to ~16-24 GB, making fine-tuning feasible on a single A100 40GB or L40S. The LLaDA GitHub repository ML-GSAI/LLaDA provides fine-tuning scripts. The MDLM paper from June 2025 on machine unlearning for MDLMs MDU paper confirms that MDLMs like LLaDA and Dream support standard supervised fine-tuning where "MDLMs learn to recover responses from masked response states conditioned on a prompt" (accessed 2026-06-29, confidence: High).

3. Modal.com GPU Pricing

Modal offers a serverless GPU cloud with per-second billing and no idle costs. GPU pricing as of June 2026 Modal Pricing (accessed 2026-06-29, confidence: High):

GPU	Per Second	Per Hour
Nvidia B200	$0.001736	$6.25
Nvidia H200	$0.001261	$4.54
Nvidia H100	$0.001097	$3.95
Nvidia RTX PRO 6000	$0.000842	$3.03
Nvidia A100 80GB	$0.000694	$2.50
Nvidia A100 40GB	$0.000583	$2.10
Nvidia L40S	$0.000542	$1.95
Nvidia A10	$0.000306	$1.10
Nvidia L4	$0.000222	$0.80
Nvidia T4	$0.000164	$0.59

Additional costs: CPU cores at $0.0000131/core/sec, memory at $0.00000222/GiB/sec, storage at $0.09/GiB/month with 1 TiB free. Multi-GPU configurations are supported (up to 8 GPUs per container on H100, A100, L40S).

Plans: Starter ($0/mo + compute, $30/mo free credits, 10 GPU concurrency, 100 containers); Team ($250/mo + compute, $100/mo credits, 50 GPU concurrency, unlimited seats); Enterprise (custom, volume discounts). Academic grants of up to $10,000 in free compute credits are available Modal Academics (accessed 2026-06-29, confidence: High).

Important considerations: Region selection adds a 1.5-1.75x multiplier. Non-preemptible execution costs 3x base prices. Modal may auto-upgrade GPU requests (e.g., A100→A100-80GB, H100→H200) at no extra charge Modal GPU Docs (accessed 2026-06-29, confidence: High).

Comparison note: Modal's serverless model means you only pay for active training time. Unlike traditional cloud providers where you reserve a GPU instance 24/7, Modal containers scale to zero when not in use. For spiky or experimental workloads, this can reduce costs by 10-15% compared to reserved instances (Modal's own comparison shows $4,740 vs $5,400 for equivalent 24h workload).

4. Cost Estimate: Fine-Tuning a ~7-8B Parameter Diffusion Model

Estimating the cost to fine-tune a diffusion language model like LLaDA 8B or iLLaDA 8B on Modal requires considering the model architecture, training approach, and dataset size.

Model size and VRAM requirements: An 8B diffusion LM in bf16 uses approximately 16 GB for weights. With a bidirectional attention mechanism (full sequence self-attention), activation memory scales with O(n²) for sequence length, making it more VRAM-intensive than causal models. However, using LoRA with rank 16-64 and 4-bit QLoRA quantization reduces the trainable parameters to ~0.1-0.5% of the base model, requiring only ~16-24 GB total VRAM LoRA Land paper (accessed 2026-06-29, confidence: High). This fits comfortably on a single A100 40GB or L40S.

Training duration estimation: For a text humanization dataset (estimated 5,000-50,000 prompt-response pairs with ~512 token average length), LoRA fine-tuning on a single GPU typically requires:

1-3 epochs over the dataset
Training speed of approximately 500-1,000 tokens/second on an A100 (based on the QLoRA profiling study Profiling paper showing 500-628 tok/s on a consumer RTX 4060; an A100 should achieve 2-4x this throughput)
Total training tokens: ~2.5M-25M
Estimated wall time: 3-15 GPU hours

Cost scenarios on Modal:

Scenario	GPU	GPU Hours	GPU Cost	CPU + Mem	Storage (100 GiB/mo)	Total
LoRA (small dataset, 5K samples)	L40S	3h	$5.85	~$0.50	$0	~$6
LoRA (medium dataset, 20K samples)	A100 80GB	8h	$20.00	~$1.00	$0	~$21
LoRA (large dataset, 50K samples)	H100	15h	$59.25	~$2.00	$0	~$61
Full fine-tune (medium dataset)	A100 80GB ×2	24h	$120.00	~$5.00	$9.00	~$134
Full fine-tune (large dataset)	H100 ×4	48h	$758.40	~$10.00	$9.00	~$777

Note: Storage costs assume 1 TiB free tier. Memory and CPU are minimal overhead (~$1-5 total). Prices are at base region rates; non-preemptible or specific regions add 1.5-3x.

Recommended approach: A LoRA/QLoRA fine-tuning of LLaDA 8B (or iLLaDA 8B) on a single A100 80GB for ~10 GPU hours provides the best cost-quality trade-off. Total cost estimate: $25-35 for a complete fine-tuning run including all overhead. On the Starter plan with $30/mo free credits, the first several experiments would be effectively free. Even without credits, a $30-50 budget covers multiple training runs with hyperparameter exploration.

For comparison, training a diffusion model from scratch is orders of magnitude more expensive: Seaweed-7B (a video diffusion model) required 665,000 H100 GPU hours Seaweed-7B paper (accessed 2026-06-29, confidence: High). Fine-tuning leverages existing pre-trained weights and is 10,000-100,000x cheaper.

5. Existing Models for Text Humanization / AI Detection Evasion

No dedicated, publicly available Hugging Face model was found explicitly labeled for "text humanization" or "AI detection evasion" as of June 2026. Multiple searches using varied terminology (humanize text, paraphrase evade AI detection, undetectable writer, bypass AI detector) returned zero results in both models and datasets categories.

However, two private Hugging Face Spaces were discovered under the authenticated user's account:

simonlesaumon/antigravity-llada-humanizer (created June 5, 2026) — a private Gradio space
simonlesaumon/antigravity-llada-humanizer-2 (created June 5, 2026) — a private Gradio space

Both spaces are private and their contents could not be inspected, but their names strongly suggest an active experimentation effort using LLaDA (a diffusion language model) for text humanization — precisely the concept under investigation in this research.

Broader landscape: The "humanization" or "AI text undetectability" space is currently dominated by commercial API services (Undetectable.ai, WriteHuman, HumanizeAI, etc.) rather than open-source models. These services typically use proprietary fine-tuned models or ensembles. The absence of open-source models represents both a gap and an opportunity: a well-executed open-source diffusion-based humanizer could be the first of its kind publicly released on Hugging Face.

The closest relevant open-source work found is in adjacent spaces: paraphrasing models, style transfer models, and "uncensored" fine-tunes of models like Gemma (e.g., the various "Heretic-Uncensored" GGUF variants found in search). However, these are general-purpose creative writing models, not specifically optimized for evading AI detection.

6. Summary Assessment

Architectural recommendation: The diffusion language model approach (specifically LLaDA 8B or iLLaDA 8B via masked diffusion) offers a genuinely novel angle for text humanization. Unlike autoregressive models that generate left-to-right with causal attention, diffusion LMs use bidirectional attention and iterative denoising, which could produce text with different statistical fingerprints — potentially harder for AI detectors to identify. The "reversal curse" resistance demonstrated by LLaDA suggests fundamentally different generation dynamics that may evade detection patterns calibrated on autoregressive outputs.

Gemma is not ideal for this project: While Gemma models are excellent autoregressive LLMs with strong fine-tuning support, they do not provide diffusion-based generation. Using them for text humanization would require either (a) an autoregressive approach (well-trodden ground) or (b) architectural modification to add diffusion capabilities (research-grade complexity). The Gemma family's value here is as a comparison baseline, not as a diffusion foundation.

Cost feasibility: Fine-tuning a diffusion LM on Modal is highly affordable. A complete LoRA fine-tuning pipeline on LLaDA 8B can be executed for approximately $25-50, and the Starter plan's $30/mo free credits cover initial experimentation. Modal's academic grant program (up to $10K) could fund extensive research if eligible.

Key unknowns and risks:

LLaDA's custom inference code requires adaptation for standard fine-tuning pipelines (Unsloth, Axolotl, etc. may not support it out of the box)
Diffusion models are inherently slower at inference than autoregressive models (requiring multiple denoising steps)
No existing benchmark or evaluation framework exists specifically for "AI detection evasion" quality — evaluation methodology would need to be developed
The two private antigravity-llada-humanizer spaces suggest parallel concurrent work, but their approach and results are unknown

References

Modal Pricing — accessed 2026-06-29, confidence: High
Modal GPU Documentation — accessed 2026-06-29, confidence: High
Modal LLM Fine-Tuning Example — accessed 2026-06-29, confidence: High
google/gemma-2-2b-it — accessed 2026-06-29, confidence: High
google/gemma-3-4b-it — accessed 2026-06-29, confidence: High
google/gemma-3-12b-it — accessed 2026-06-29, confidence: High
google/gemma-3-27b-it — accessed 2026-06-29, confidence: High
google/gemma-4-12B-it — accessed 2026-06-29, confidence: High
google/gemma-4-26B-A4B-it — accessed 2026-06-29, confidence: High
GSAI-ML/LLaDA-8B-Base — accessed 2026-06-29, confidence: High
GSAI-ML/LLaDA-8B-Instruct — accessed 2026-06-29, confidence: High
GSAI-ML/LLaDA-1.5 — accessed 2026-06-29, confidence: High
LLaDA Paper — Large Language Diffusion Models (Nie et al., 2025) — accessed 2026-06-29, confidence: High
iLLaDA Paper — Improved Large Language Diffusion Models (Nie et al., 2026) — accessed 2026-06-29, confidence: High
LLaDA-MoE Paper (Zhu et al., 2025) — accessed 2026-06-29, confidence: High
Dream 7B Paper (Ye et al., 2025) — accessed 2026-06-29, confidence: High
MDLM Paper — Simple and Effective Masked Diffusion Language Models (Sahoo et al., 2024) — accessed 2026-06-29, confidence: High
MDU Paper — Machine Unlearning for MDLMs (Lee et al., 2026) — accessed 2026-06-29, confidence: High
LoRA Land Paper (Zhao et al., 2024) — accessed 2026-06-29, confidence: High
LoRA/QLoRA Profiling on Consumer GPUs (Avinash, 2025) — accessed 2026-06-29, confidence: Medium
Seaweed-7B Paper — Cost-Effective Training of Video Generation (Team Seawead, 2025) — accessed 2026-06-29, confidence: High
LLaDA GitHub Repository — accessed 2026-06-29, confidence: High
LLaDA-V Paper (You et al., 2025) — accessed 2026-06-29, confidence: High
GENIE Paper — Text Generation with Diffusion LMs (Lin et al., 2022) — accessed 2026-06-29, confidence: Medium
DiffusionBERT Paper (He et al., 2022) — accessed 2026-06-29, confidence: Medium