README.md

metadata

license: other
license_name: modilify-open-model-license-1.0
license_link: LICENSE
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - diffusion
  - multimodal
  - image-text-to-text
  - mixture-of-experts
  - trust-remote-code

Modilify Mk1 Preview

Modilify Mk1 Preview is a 26B-A5B multimodal block-diffusion model developed by Modilify. It combines the original DiffusionGemma text, image, and video context encoder with recurrent latent deliberation and a multiplicative confidence-and-entropy commit policy. Its central advance is elastic inference: the model can spend more heavy-denoise work and produce a longer response for a difficult problem, or commit more tokens per pass and finish sooner when the problem is easier.

Model Summary

Architecture	Mixture-of-Experts block diffusion
Total Parameters	26.139B
Activated Parameters	4.729B, including the vision encoder
Text Heavy-Denoise Activated Parameters	4.159B
FLOPs per Heavy Denoise	~2.12 TFLOPs at batch 1, 256-token canvas, empty KV prefix
Layers	30
Number of Experts	128
Selected Experts per Token	8
Number of Shared Experts	1
Vocabulary Size	262,144
Context Length	262,144 tokens
Activation Function	GELU, tanh approximation
Vision Encoder	Gemma 4 Vision
Vision Encoder Parameters	569.550M
Modality	Text, Image, Video
Sliding Window	1024 tokens
Canvas Length	256

The heavy-denoise FLOPs estimate counts multiply-adds as two FLOPs and covers decoder, expert, latent deliberation, and attention work only. It excludes the encoder pass, sampling/softmax, and elementwise ops. Batch size scales it roughly linearly: a 256-token prefix raises the estimate to ~2.16 TFLOPs, and a 4,096-token prefix to ~2.38 TFLOPs because some layers use full attention.

Benchmark Results

Benchmark	Modilify Mk1 Preview	Status
MMLU Pro	~78.0	Partial, one-shot estimate
AIME 2026, no tools	~70	Partial, one-shot estimate

These preliminary results use specialized inference settings: denoise_temperature=0.4, commit_failure_budget=0.05, fused_entropy_weight=0.6, and jump_on_no_progress_after=32. The defaults target substantially higher throughput and should not be expected to reproduce these estimates.

Only part of each dataset was evaluated, with one-shot prompting. Treat these values as unstable and non-comparable until the full benchmark release.

Core Capabilities

• Elastic reasoning length and throughput. Unlike a fixed reasoning-token budget, each heavy denoise can commit a variable-length prefix. Easier problems can converge in fewer passes with more tokens committed per pass; harder problems can keep deliberating and produce longer answers, up to the configured generation and ponder bounds. commit_failure_budget, fused_entropy_weight, and the progress limits let deployments move the throughput-quality operating point.
• Implicit context through latent deliberation. The model reasons over the complete noisy canvas inside each heavy denoise, then compresses the evolving trajectory into recurrent token latents and fixed memory slots. That compact state conditions the next heavy pass and survives rolling-canvas commits, providing an implicit context channel without copying the entire denoise history into visible output tokens.
• 120x faster trajectory training. The current memory-bounded training system delivered a measured 120x throughput improvement over the project's original trajectory-training implementation under the matched internal setup. This figure describes the training implementation comparison, not a 120x inference speedup or a comparison with unrelated systems.
• Text generation supports the native optional thinking-channel protocol, and input context extends to the configured 262,144-token limit.

Getting Started

Transformers 5.14.1 is the minimum supported version. To get started, install Transformers, PyTorch, and Accelerate in your environment:

pip install -U transformers torch accelerate

Text generation

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "modilify/Modilify-Mk1-preview"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain why the sky is blue."}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=256)
new_tokens = output.sequences[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(new_tokens, skip_special_tokens=False)[0])

Image input

from PIL import Image

image = Image.open("example.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Describe the image and identify uncertainty."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)

Video-frame input

The processor represents video as a sampled sequence of frames. The following example uses PyAV to decode a short local clip and samples at most 32 RGB frames.

import av
from PIL import Image

container = av.open("short_clip.mp4")
decoded = [Image.fromarray(frame.to_rgb().to_ndarray()) for frame in container.decode(video=0)]
stride = max(1, len(decoded) // 32)
frames = decoded[::stride][:32]

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": frames},
        {"type": "text", "text": "Summarize the main visual events in order."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)

Thinking mode

The Gemma chat template controls thinking markup. Set enable_thinking=True in apply_chat_template to request the native thinking/channel protocol, or set it to False for a direct answer. Applications should not assume hidden reasoning is complete, correct, or appropriate to expose to end users.

Configurable inference parameters

All model-owned values below are serialized in config.json and may be changed before loading or through a copied configuration object.

Parameter	Default	Meaning
canvas_length	256	Rolling diffusion canvas length
denoise_temperature	0.8	Sampling temperature
commit_failure_budget	0.2	Normal cumulative prefix risk limit
fused_entropy_weight	0.5	Entropy multiplier coefficient
jump_failure_budget	2.0	Forced-jump cumulative risk limit
vocab_chunk_size	65,536	Serialized projection planning size; standard tensor operations are used
jump_on_no_progress_after	12	Stagnation threshold
max_ponder_steps	64	Watchdog multiplier per requested token
min_trajectory_progress	0.005	Minimum fused-risk improvement
latent_dim	1,536	Latent state width
latent_memory_slots	64	Persistent memory slot count
latent_num_layers	4	Latent Transformer depth
latent_num_heads	16	Latent attention heads
latent_local_attention_window	128	Local token-attention window
latent_dropout	0.0	Inference dropout probability
turn_end_token_id	106	Gemma turn terminator

Example override:

from transformers import AutoConfig

config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
config.max_ponder_steps = 32
config.commit_failure_budget = 0.15
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    config=config,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

Generation supports left-padded batches with independent stopping and generated_lengths for every row. Batch prompts of similar lengths together for the best throughput; KV-cache and canvas memory grow with batch size. Streaming and caller-supplied KV caches remain limited to batch size 1.

Details

Trained on Apple silicon.

Developed on Mac by Modilify.

Evaluation status, limitations, and risks

The benchmark values above are partial one-shot estimates, not a complete evaluation. Export checks established checkpoint structure, exact adapter application, valid safetensors indexing, absence of residual adapters, and byte-level preservation of the vision tower and projection; they do not establish accuracy, robustness, calibration, fairness, safety, or fitness for use.

The model can hallucinate facts, citations, visual details, or temporal relationships; reproduce bias, unsafe content, personal information, or copyrighted material; and consume substantial time and memory during long iterative generation. Confidence-based commits are compute-control decisions, not guarantees of correctness. Visual performance can degrade with poor resolution, motion, occlusion, unusual aspect ratios, or domain shift.

Evaluate the exact deployment on representative, adversarial, and out-of-distribution inputs. Use layered safeguards, monitoring, incident response, and qualified human review, and never delegate autonomous high-risk medical, legal, financial, employment, housing, education, critical-infrastructure, or safety decisions to the model.

License

Released under the draft Modilify Open Model License 1.0, subject to its responsible-use and derivative-impact terms. Upstream rights, attribution, Apache-2.0 text, and the impact-statement template are retained in NOTICE.md. Obtain independent legal review before public release.

Citation

Until a formal paper is available, cite the model repository and its base:

@software{modilify_mk1_preview_2026,
  title = {Modilify Mk1 Preview},
  author = {Modilify},
  year = {2026},
  note = {A multimodal latent-deliberation derivative of DiffusionGemma}
}

Also cite the upstream DiffusionGemma release as requested by Google DeepMind.