aniskhan25/densfilm-crowdfix

DensFiLM — Density-Conditioned Video Saliency for Crowd Scenes

DensFiLM is a Video Swin Transformer (Swin3D-S) augmented with a FiLM layer at the encoder bottleneck. A 64-dimensional learnable crowd-density embedding is projected to per-channel scale–shift parameters (γ, β) ∈ ℝ⁷⁶⁸ that recalibrate the most semantically abstract spatiotemporal feature map before decoding — at a cost of only ~100K additional parameters (<0.2% of backbone weights).

The motivation: as crowd density rises, human fixation strategy shifts categorically — from tracking individual pedestrians, to reading collective motion structure (flow lanes, merge boundaries), to localising scene-level landmarks against density gradients. Existing video saliency models treat all densities uniformly.

• Paper: arXiv:2607.25465
• Code: github.com/aniskhan25/crowdfix-saliency
• Author: Anis Ur Rahman, CSC – IT Center for Science Ltd., Espoo, Finland

Results on CrowdFix

5,356-clip CrowdFix test set. DensFiLM rows are mean ± std over 4 seeds; baselines are as published in the 2019 CrowdFix benchmark paper.

Model	AUC-J ↑	AUC-B ↑	NSS ↑	CC ↑	KL ↓	SIM ↑
SAM	0.777	–	0.894	0.325	–	–
DeepVS	0.788	–	0.985	0.401	–	–
ACL-Net (2019)	0.817	–	1.250	0.450	–	–
Swin3D-S, unconditioned	0.806	0.810	1.313	0.476	1.133	0.424
DensFiLM (oracle label)	0.820 ±0.001	0.825 ±0.001	1.434 ±0.007	0.517 ±0.002	1.068 ±0.004	0.449 ±0.003
DensFiLM (predicted label)	0.821	0.825	1.438	0.518	1.067	0.451

This surpasses ACL-Net — the sole published CrowdFix benchmark since 2019 — by +14.7% NSS and +14.9% CC.

Notably, replacing the ground-truth density label with the model's own density head prediction gives identical performance (ΔNSS ≤ 0.001), so the conditioning signal is self-derived at inference and no density annotation is needed.

Usage

import torch
from models.densfilm_hub import DensFiLM

model = DensFiLM.from_pretrained("aniskhan25/densfilm-crowdfix").eval()

# (B, 3, T, H, W) — 8 frames at 224x384, ImageNet-normalised
clip = torch.randn(1, 3, 8, 224, 384)

with torch.no_grad():
    saliency, density_logits = model(clip)      # density self-derived

# saliency: (1, 1, 224, 384) in [0, 1]
# density_logits: (1, 3) over SP / DF / DC
print(saliency.shape, density_logits.argmax(-1))

To condition on a known density category instead:

density = torch.tensor([2])                     # 0=SP, 1=DF, 2=DC
saliency, _ = model(clip, density)

Preprocessing

Frames must be resized to 224×384 and ImageNet-normalised (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]), stacked into clips of 8 consecutive frames as (B, 3, T, H, W).

import torch
import torchvision.transforms.functional as TF

MEAN = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)
STD  = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)

def preprocess(frames):                          # list of 8 PIL images
    t = torch.stack([TF.to_tensor(TF.resize(f, [224, 384], antialias=True))
                     for f in frames])           # (T, 3, H, W)
    t = (t - MEAN) / STD
    return t.permute(1, 0, 2, 3).unsqueeze(0)    # (1, 3, T, H, W)

Density categories

Label	Code	Meaning
0	SP	Sparse — individual pedestrian trajectories dominate fixation
1	DF	Dense Flow — collective motion structure, flow lanes, merge boundaries
2	DC	Dense Congested — scene-level semantic landmarks against density gradients

Training

Trained on CrowdFix (3,189 training clips) from a Kinetics-400-pretrained Swin3D-S backbone; FiLM layers, density embedding and decoder initialise fresh.

Setting	Value
Epochs	150, early stopping patience 20
Optimiser	AdamW, weight decay 1e-4, cosine annealing
LR	1e-4 (backbone 0.1×), 20 frozen-backbone warmup epochs
Batch size	2 per GPU × 8 GCDs
Clip	8 frames at 224×384
Hardware	8× AMD MI250X GCDs on LUMI

Reported figures are mean ± std over 4 seeds. This checkpoint is a single seed, so its individual metrics will sit within the quoted std rather than exactly on the mean.

Limitations

• Trained and evaluated only on CrowdFix; generalisation to non-crowd video saliency is untested. Domain shift to other crowd datasets is unmeasured.
• 3,189 training clips place the model firmly in the regularisation-dominated regime. The paper's ablations show that adding capacity (3D temporal decoder, learned social-force prior) collapses NSS to roughly the unconditioned baseline (~1.313).
• Explicit optical flow is redundant: a RAFT dual-stream variant lands within seed variance (NSS 1.437 vs 1.434 ± 0.007), as Swin3D-S's shifted-window attention already captures the motion signal.
• Debiased evaluation (subtracting mean training fixation density to remove shared centre-prior alignment) drops DensFiLM to AUC-J 0.765 / NSS 1.188. The unconditioned backbone loses 45% of its NSS while DensFiLM loses only 17% — density conditioning substitutes genuine structure for centre prior, but a centre-prior component remains.
• Fixed input geometry: 8 frames at 224×384. Other resolutions are untested.

Citation

@article{rahman2026densfilm,
  title   = {Density-Conditioned Video Saliency for Crowd Scenes via
             Feature-wise Linear Modulation},
  author  = {Rahman, Anis Ur},
  journal = {arXiv preprint arXiv:2607.25465},
  year    = {2026}
}