Reza2kn

/

Cosmos3-Nano-INT4-AWQ

like 2

Modified from this space, which can generate video just fine. But can't seems to work with this repo weight

seed: 807131831
prompt: A cinematic wide shot of a vintage red convertible driving along a coastal highway at sunset. Waves crash against rocks below as the car speeds toward the horizon. Golden hour lighting, lens flare, smooth tracking shot.

Original repo weight

This repo weight

Modification from

pipe = Cosmos3OmniPipeline.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    enable_safety_checker=True,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
pipe.to("cuda")

to

from diffusers import Cosmos3OmniPipeline, Cosmos3OmniTransformer
from huggingface_hub import snapshot_download
import modelopt.torch.opt as mto

repo = snapshot_download("Reza2kn/Cosmos3-Nano-INT4-AWQ")
tf = Cosmos3OmniTransformer.from_config(
    Cosmos3OmniTransformer.load_config(f"{repo}/transformer/config.json")
).to(torch.bfloat16)

mto.restore(tf, f"{repo}/transformer/modelopt_quantized.pt")
pipe = Cosmos3OmniPipeline.from_pretrained(
    repo, transformer=tf, 
    torch_dtype=torch.bfloat16, enable_safety_checker=False,
).to("cuda")

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
pipe.to("cuda")

and

with torch.autocast("cuda", dtype=torch.bfloat16):
    result  = pipe(...)

It should be note that on Cosmos3OmniTransformer.from_config it give the following warning

The config attributes {'action_dim': 64, 'action_gen': True, 'dtype': 'bfloat16', 'freeze_und': False, 'hidden_act': 'silu', 'initializer_range': 0.02, 'joint_attn_implementation': 'two_way', 'max_action_dim': 64, 'max_position_embeddings': 262144, 'model_type': 'qwen3_vl_text', 'num_embodiment_domains': 32, 'position_embedding_type': 'unified_3d_mrope', 'qk_norm': False, 'qk_norm_for_diffusion': True, 'qk_norm_for_text': True, 'temporal_compression_factor_sound': 1, 'use_cache': True, 'use_moe': True, 'video_temporal_causal': False} were passed to Cosmos3OmniTransformer, but are not expected and will be ignored. Please verify your config.json configuration file.

I am not sure about its significant currently

The image generation seems to be okay (single frame), but for multiple frames, (and sound) the decoded latent becomes blocky-blobs