TransCity-VLM/Transcity-vlm-stageC

TransCity-VLM StageC

This repository contains the StageC-final SFT checkpoint of TransCity-VLM, a custom multimodal model for evidence-grounded smart-city reasoning and prediction. It combines a Qwen3-8B decoder, a RoBERTa-large chunk encoder, a SigLIP vision encoder, multimodal projectors, and task-routed MixLoRA experts.

This is a sharded checkpoint for the custom SmartCityLLMForCausalLM architecture. It is not directly loadable with AutoModelForCausalLM.from_pretrained(). Use it with the TransCity-VLM source code.

Loading

Install the project dependencies and run the following from the TransCity-VLM source directory:

from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

from model.builder import build_smartcity_llm

MODEL_ID = "TransCity-VLM/Transcity-vlm-stageC"
checkpoint_dir = snapshot_download(MODEL_ID)

tokenizer = AutoTokenizer.from_pretrained(
    checkpoint_dir,
    use_fast=True,
    padding_side="left",
    trust_remote_code=True,
)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token

mixlora_config = {
    "num_experts": 20,
    "lora_r": 8,
    "lora_alpha": 16.0,
    "enable_attention": False,
    "enable_ffn": True,
    "share_router_for_wi": True,
    "num_tasks": 3,
    "ensure_nonzero_gating": False,
    "num_universal": 4,
    "num_pred": 8,
    "num_reason": 8,
    "rank_universal_mul": 2.0,
    "target_modules_override": {
        "atte": "self_attn",
        "ffn": "mlp",
        "q": [], "k": [], "v": [], "o": [],
        "wi": ["mlp.gate_proj", "mlp.up_proj"],
        "wo": ["mlp.down_proj"],
    },
}

model = build_smartcity_llm(
    decoder_name="Qwen/Qwen3-8B",
    encoder_name="FacebookAI/roberta-large",
    num_chunk_tokens=4,
    adapter_heads=4,
    encoder_max_length=512,
    imagebind_variant="google/siglip-so400m-patch14-384",
    num_image_tokens=32,
    group_layers=2,
    group_heads=8,
    group_tau=1.0,
    open_roberta=False,
    open_vpma=False,
    open_imagebind=False,
    open_image_projector=False,
    freeze_decoder=True,
    use_mixlora=True,
    mixlora_cfg_overrides=mixlora_config,
)

model.load_full_state_dict(
    checkpoint_dir,
    cast_to_target_dtype=True,
    verbose=True,
)
model.decoder.config.pad_token_id = tokenizer.pad_token_id
model.decoder.config.eos_token_id = tokenizer.eos_token_id
model.eval().to("cuda")

For preprocessing and generation, use the project DataCollatorSmartCity pipeline and generate_with_modalities_cached() implementation.

Intended use and limitations

This checkpoint is intended for research on multimodal, evidence-grounded smart-city tasks. It expects the project-specific input format and preprocessing pipeline, and has not been validated for safety-critical or general-purpose deployment.

License

The released checkpoint is available under CC BY 4.0. Upstream backbones and source-code dependencies retain their respective licenses.