Upload folder using huggingface_hub

README.md

@@ -0,0 +1,87 @@
+---
+license: other
+license_name: polyform-small-business-1.0.0
+license_link: https://polyformproject.org/licenses/small-business/1.0.0/
+library_name: pytorch
+pipeline_tag: video-classification
+tags:
+  - bio-inspired
+  - neuroscience
+  - lightweight
+  - temporal
+  - video-understanding
+  - action-recognition
+  - retinal-ganglion-cells
+  - fibonacci-strides
+datasets:
+  - ucf101
+---
+
+# MPKNet V6.2 Temporal - Bio-Inspired Video Classification
+
+An extension of MPKNet V6 that adds temporal processing to the M (Magnocellular) pathway for video understanding and action recognition. The M pathway processes 8 consecutive frames and computes inter-frame deltas to capture motion, while the P pathway sees only the current frame for spatial detail.
+
+## Architecture
+
+Built on the three-pathway design with Fibonacci strides (2:3:5):
+
+- **P pathway** (stride 2): Current frame only - fine spatial detail
+- **K pathway** (stride 3): Current frame only - context and gating signals
+- **M pathway** (stride 5): 8 consecutive frames - computes 7 inter-frame deltas for motion
+
+The M pathway uses shared Conv2D weights across all frames, computing learned deltas between consecutive frame pairs. A temporal fusion module combines all 7 deltas into a single motion representation.
+
+For static images, the model generates pseudo-frames via progressive scale and blur augmentation, teaching M to detect change even without real motion. This transfers to real video at inference.
+
+## Results
+
+| Dataset | Classes | Accuracy | Parameters |
+|---------|---------|----------|------------|
+| UCF-101 | 101 | 77 percent | 0.58M |
+
+No pretraining. 0.58M parameters processing 8-frame sequences.
+
+## Usage
+
+```python
+import torch
+from mpknet_v6_2_temporal import BinocularMPKNetV6_2
+from mpknet_components import count_params
+
+model = BinocularMPKNetV6_2(num_classes=101, ch=48, use_stereo=True, num_frames=8)
+state_dict = torch.load("mpknet_v6_2_ucf101_best.pt", map_location="cpu", weights_only=True)
+model.load_state_dict(state_dict)
+model.eval()
+
+# Inference with video frames
+current_frame = torch.randn(1, 3, 224, 224)
+frame_sequence = torch.randn(1, 8, 3, 224, 224)
+
+with torch.no_grad():
+    logits = model(current_frame, frames=frame_sequence)
+    pred = logits.argmax(dim=1)
+
+# Or with a static image (auto-generates pseudo-frames)
+with torch.no_grad():
+    logits = model(current_frame)
+```
+
+## Files
+
+- `mpknet_v6_2_ucf101_best.pt` - Trained weights (UCF-101, 101 classes, 7.3MB)
+- `mpknet_v6_2_temporal.py` - Model architecture with SequentialTemporalMPathway
+- `mpknet_components.py` - Shared components
+
+## Citation
+
+```
+D.J. Lougen, "MPKNet: Bio-Inspired Visual Classification with Parallel LGN Pathways", 2025.
+```
+
+## License
+
+PolyForm Small Business License 1.0.0 - Free for organizations with less than 100K revenue, non-profits, and education.
+
+## Links
+
+- [GitHub](https://github.com/DJLougen/MPKnet)

mpknet_components.py

@@ -0,0 +1,397 @@
+"""
+Shared components for all MPKNet model variants.
+
+Contains building blocks used across V1, V2, V3, V4 and detection models:
+- RGCLayer: Biologically accurate retinal ganglion cell preprocessing
+- BinocularPreMPK: Legacy retinal preprocessing (deprecated, use RGCLayer)
+- StereoDisparity: Stereo disparity simulation
+- OcularDominanceConv: Convolution with ocular dominance channels
+- BinocularMPKPathway: Pathway with binocular processing
+- MonocularPathwayBlock: Pathway keeping eyes separate
+- StridedMonocularBlock: Strided pathway for V4
+"""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from typing import Tuple
+
+
+class RGCLayer(nn.Module):
+    """
+    Biologically accurate Retinal Ganglion Cell layer.
+
+    Based on Kim et al. 2021 "Retinal Ganglion Cells—Diversity of Cell Types
+    and Clinical Relevance" (Front. Neurol. 12:661938).
+
+    Models three main RGC types that feed the P/K/M pathways:
+
+    1. MIDGET RGCs (~70% of RGCs):
+       - Small receptive field (5-10 μm dendritic field)
+       - Center-surround via Difference of Gaussians (DoG)
+       - Red-Green color opponency (L-M or M-L)
+       - Feeds PARVOCELLULAR (P) pathway
+       - High spatial acuity, low temporal resolution
+
+    2. PARASOL RGCs (~10% of RGCs):
+       - Large receptive field (30-300 μm dendritic field)
+       - Center-surround DoG on luminance
+       - Achromatic (no color, L+M pooled)
+       - Feeds MAGNOCELLULAR (M) pathway
+       - Motion detection, high temporal resolution
+
+    3. SMALL BISTRATIFIED RGCs (~5-8% of RGCs):
+       - Medium receptive field
+       - S-cone ON center, (L+M) OFF surround
+       - Blue-Yellow opponency
+       - Feeds KONIOCELLULAR (K) pathway
+       - Color context, particularly blue
+
+    Key biological details implemented:
+    - DoG (Difference of Gaussians) for center-surround RF
+    - RF size ratios: Midget < Bistratified < Parasol
+    - Surround ~3-6x larger than center (we use 3x)
+    - ON-center and OFF-center populations (we use ON-center)
+    """
+
+    def __init__(
+        self,
+        midget_sigma: float = 0.8,    # Small RF for fine detail
+        parasol_sigma: float = 2.5,    # Large RF for motion/gist
+        bistrat_sigma: float = 1.2,    # Medium RF for color context
+        surround_ratio: float = 3.0,   # Surround is 3x center
+    ):
+        super().__init__()
+
+        self.midget_sigma = midget_sigma
+        self.parasol_sigma = parasol_sigma
+        self.bistrat_sigma = bistrat_sigma
+        self.surround_ratio = surround_ratio
+
+        # Create DoG kernels for each cell type
+        self.register_buffer('midget_center', self._make_gaussian(midget_sigma))
+        self.register_buffer('midget_surround', self._make_gaussian(midget_sigma * surround_ratio))
+
+        self.register_buffer('parasol_center', self._make_gaussian(parasol_sigma))
+        self.register_buffer('parasol_surround', self._make_gaussian(parasol_sigma * surround_ratio))
+
+        self.register_buffer('bistrat_center', self._make_gaussian(bistrat_sigma))
+        self.register_buffer('bistrat_surround', self._make_gaussian(bistrat_sigma * surround_ratio))
+
+        # Store kernel sizes for padding calculation
+        self.midget_ks = self.midget_surround.shape[-1]
+        self.parasol_ks = self.parasol_surround.shape[-1]
+        self.bistrat_ks = self.bistrat_surround.shape[-1]
+
+    def _make_gaussian(self, sigma: float) -> torch.Tensor:
+        """Create a normalized 2D Gaussian kernel."""
+        ks = int(6 * sigma + 1) | 1  # Ensure odd, cover 3 sigma each side
+        ax = torch.arange(ks, dtype=torch.float32) - ks // 2
+        xx, yy = torch.meshgrid(ax, ax, indexing='ij')
+        kernel = torch.exp(-(xx**2 + yy**2) / (2 * sigma**2))
+        kernel = kernel / kernel.sum()  # Normalize
+        return kernel.unsqueeze(0).unsqueeze(0)  # (1, 1, H, W)
+
+    def _apply_dog(
+        self,
+        x: torch.Tensor,
+        center_kernel: torch.Tensor,
+        surround_kernel: torch.Tensor,
+        kernel_size: int
+    ) -> torch.Tensor:
+        """Apply Difference of Gaussians (center - surround)."""
+        B, C, H, W = x.shape
+        padding = kernel_size // 2
+
+        # Expand kernels for all channels
+        center_k = center_kernel.expand(C, 1, -1, -1)
+        surround_k = surround_kernel.expand(C, 1, -1, -1)
+
+        # Pad surround kernel to match size if needed
+        c_size = center_k.shape[-1]
+        s_size = surround_k.shape[-1]
+        if c_size < s_size:
+            pad_amt = (s_size - c_size) // 2
+            center_k = F.pad(center_k, (pad_amt, pad_amt, pad_amt, pad_amt))
+
+        # Apply center and surround
+        center_response = F.conv2d(x, center_k, padding=padding, groups=C)
+        surround_response = F.conv2d(x, surround_k, padding=padding, groups=C)
+
+        # DoG: ON-center response (center - surround)
+        return center_response - surround_response
+
+    def forward(
+        self,
+        x_left: torch.Tensor,
+        x_right: torch.Tensor
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor,
+               torch.Tensor, torch.Tensor, torch.Tensor]:
+        """
+        Process left and right eye inputs through RGC populations.
+
+        Returns:
+            P_left, P_right: Midget RGC output (R-G opponency) -> P pathway
+            M_left, M_right: Parasol RGC output (luminance DoG) -> M pathway
+            K_left, K_right: Bistratified RGC output (S vs L+M) -> K pathway
+        """
+        # ========== MIDGET RGCs -> P pathway ==========
+        # Red-Green opponency: L-cone vs M-cone
+        # Approximate: R channel vs G channel
+        # DoG on the opponent signal
+
+        # Extract R and G channels (approximating L and M cones)
+        R_left, G_left = x_left[:, 0:1], x_left[:, 1:2]
+        R_right, G_right = x_right[:, 0:1], x_right[:, 1:2]
+
+        # L-M opponency (R-G) with small receptive field DoG
+        rg_left = R_left - G_left
+        rg_right = R_right - G_right
+
+        P_left = self._apply_dog(rg_left, self.midget_center, self.midget_surround, self.midget_ks)
+        P_right = self._apply_dog(rg_right, self.midget_center, self.midget_surround, self.midget_ks)
+
+        # Expand back to 3 channels for compatibility
+        P_left = P_left.expand(-1, 3, -1, -1)
+        P_right = P_right.expand(-1, 3, -1, -1)
+
+        # ========== PARASOL RGCs -> M pathway ==========
+        # Achromatic: pool L+M (approximate as luminance)
+        # Large RF DoG for motion sensitivity
+
+        lum_left = 0.299 * x_left[:, 0:1] + 0.587 * x_left[:, 1:2] + 0.114 * x_left[:, 2:3]
+        lum_right = 0.299 * x_right[:, 0:1] + 0.587 * x_right[:, 1:2] + 0.114 * x_right[:, 2:3]
+
+        M_left = self._apply_dog(lum_left, self.parasol_center, self.parasol_surround, self.parasol_ks)
+        M_right = self._apply_dog(lum_right, self.parasol_center, self.parasol_surround, self.parasol_ks)
+
+        # Expand to 3 channels
+        M_left = M_left.expand(-1, 3, -1, -1)
+        M_right = M_right.expand(-1, 3, -1, -1)
+
+        # ========== SMALL BISTRATIFIED RGCs -> K pathway ==========
+        # S-cone ON center, (L+M) OFF surround
+        # Blue-Yellow opponency: S vs (L+M)
+
+        # S-cone approximated by B channel
+        # (L+M) approximated by (R+G)/2
+        S_left = x_left[:, 2:3]  # Blue
+        S_right = x_right[:, 2:3]
+        LM_left = (x_left[:, 0:1] + x_left[:, 1:2]) / 2
+        LM_right = (x_right[:, 0:1] + x_right[:, 1:2]) / 2
+
+        # S - (L+M) opponency with medium RF
+        by_left = S_left - LM_left
+        by_right = S_right - LM_right
+
+        K_left = self._apply_dog(by_left, self.bistrat_center, self.bistrat_surround, self.bistrat_ks)
+        K_right = self._apply_dog(by_right, self.bistrat_center, self.bistrat_surround, self.bistrat_ks)
+
+        # Expand to 3 channels
+        K_left = K_left.expand(-1, 3, -1, -1)
+        K_right = K_right.expand(-1, 3, -1, -1)
+
+        return P_left, M_left, K_left, P_right, M_right, K_right
+
+
+class BinocularPreMPK(nn.Module):
+    """
+    Simulates retinal + LGN preprocessing for both eyes.
+    Each eye gets its own center-surround filtering.
+
+    Biological motivation:
+    - Retinal ganglion cells have center-surround receptive fields
+    - M cells respond to luminance changes (motion/gist)
+    - P cells respond to color/detail (high-pass filtered)
+    """
+    def __init__(self, sigma: float = 1.0):
+        super().__init__()
+        self.sigma = sigma
+        ks = int(4 * sigma + 1) | 1  # ensure odd
+        ax = torch.arange(ks, dtype=torch.float32) - ks // 2
+        xx, yy = torch.meshgrid(ax, ax, indexing='ij')
+        kernel = torch.exp(-(xx**2 + yy**2) / (2 * sigma**2))
+        kernel = kernel / kernel.sum()
+        self.register_buffer('gauss', kernel.unsqueeze(0).unsqueeze(0))
+        self.ks = ks
+
+    def _blur(self, x: torch.Tensor) -> torch.Tensor:
+        B, C, H, W = x.shape
+        kernel = self.gauss.expand(C, 1, self.ks, self.ks)
+        return F.conv2d(x, kernel, padding=self.ks // 2, groups=C)
+
+    def forward(self, x_left: torch.Tensor, x_right: torch.Tensor) -> Tuple[torch.Tensor, ...]:
+        """
+        Returns (P_left, M_left, P_right, M_right)
+        P = high-pass (center - surround) for detail
+        M = low-pass luminance for motion/gist
+        """
+        # Left eye
+        blur_L = self._blur(x_left)
+        P_left = x_left - blur_L  # high-pass (Parvo-like)
+        lum_L = x_left.mean(dim=1, keepdim=True)
+        M_left = self._blur(lum_L).expand(-1, 3, -1, -1)  # low-pass luminance (Magno-like)
+
+        # Right eye
+        blur_R = self._blur(x_right)
+        P_right = x_right - blur_R
+        lum_R = x_right.mean(dim=1, keepdim=True)
+        M_right = self._blur(lum_R).expand(-1, 3, -1, -1)
+
+        return P_left, M_left, P_right, M_right
+
+
+class StereoDisparity(nn.Module):
+    """
+    Creates stereo disparity by horizontally shifting left/right views.
+    Simulates the slight positional difference between two eyes.
+
+    disparity_range: maximum pixel shift (positive = crossed disparity)
+    """
+    def __init__(self, disparity_range: int = 2):
+        super().__init__()
+        self.disparity_range = disparity_range
+
+    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        """
+        Takes single image, returns (left_view, right_view) with disparity.
+        For training, uses random disparity. For inference, uses fixed small disparity.
+        """
+        B, C, H, W = x.shape
+
+        if self.training:
+            d = torch.randint(-self.disparity_range, self.disparity_range + 1, (1,)).item()
+        else:
+            d = 1
+
+        if d == 0:
+            return x, x
+
+        if d > 0:
+            x_left = F.pad(x[:, :, :, d:], (0, d, 0, 0), mode='replicate')
+            x_right = F.pad(x[:, :, :, :-d], (d, 0, 0, 0), mode='replicate')
+        else:
+            d = -d
+            x_left = F.pad(x[:, :, :, :-d], (d, 0, 0, 0), mode='replicate')
+            x_right = F.pad(x[:, :, :, d:], (0, d, 0, 0), mode='replicate')
+
+        return x_left, x_right
+
+
+class OcularDominanceConv(nn.Module):
+    """
+    Convolution with ocular dominance - channels are assigned to left/right eye
+    with graded mixing (some purely monocular, some binocular).
+
+    Inspired by V1 ocular dominance columns but applied at LGN stage
+    for computational efficiency.
+    """
+    def __init__(self, in_ch: int, out_ch: int, kernel_size: int,
+                 monocular_ratio: float = 0.5):
+        super().__init__()
+        self.out_ch = out_ch
+        self.monocular_ratio = monocular_ratio
+
+        n_mono = int(out_ch * monocular_ratio)
+        n_mono_per_eye = n_mono // 2
+        n_bino = out_ch - 2 * n_mono_per_eye
+
+        self.n_left = n_mono_per_eye
+        self.n_right = n_mono_per_eye
+        self.n_bino = n_bino
+
+        self.conv_left = nn.Conv2d(in_ch, n_mono_per_eye, kernel_size, padding=kernel_size//2)
+        self.conv_right = nn.Conv2d(in_ch, n_mono_per_eye, kernel_size, padding=kernel_size//2)
+        self.conv_bino_L = nn.Conv2d(in_ch, n_bino, kernel_size, padding=kernel_size//2)
+        self.conv_bino_R = nn.Conv2d(in_ch, n_bino, kernel_size, padding=kernel_size//2)
+        self.bn = nn.BatchNorm2d(out_ch)
+
+    def forward(self, x_left: torch.Tensor, x_right: torch.Tensor) -> torch.Tensor:
+        left_only = self.conv_left(x_left)
+        right_only = self.conv_right(x_right)
+        bino = self.conv_bino_L(x_left) + self.conv_bino_R(x_right)
+        out = torch.cat([left_only, right_only, bino], dim=1)
+        return F.relu(self.bn(out))
+
+
+class BinocularMPKPathway(nn.Module):
+    """
+    Single pathway (M, P, or K) with binocular processing.
+    Receives left and right eye inputs, produces fused output.
+    """
+    def __init__(self, in_ch: int, out_ch: int, kernel_sizes: list,
+                 monocular_ratio: float = 0.5):
+        super().__init__()
+
+        layers = []
+        ch = in_ch
+        for i, ks in enumerate(kernel_sizes):
+            is_first = (i == 0)
+            if is_first:
+                layers.append(OcularDominanceConv(ch, out_ch, ks, monocular_ratio))
+            else:
+                layers.append(nn.Sequential(
+                    nn.Conv2d(out_ch if i > 0 else ch, out_ch, ks, padding=ks//2),
+                    nn.BatchNorm2d(out_ch),
+                    nn.ReLU(inplace=True)
+                ))
+            ch = out_ch
+
+        self.first_layer = layers[0]
+        self.rest = nn.Sequential(*layers[1:]) if len(layers) > 1 else nn.Identity()
+
+    def forward(self, x_left: torch.Tensor, x_right: torch.Tensor) -> torch.Tensor:
+        x = self.first_layer(x_left, x_right)
+        return self.rest(x)
+
+
+class MonocularPathwayBlock(nn.Module):
+    """
+    Single pathway block that keeps left/right eyes separate.
+    Used for LGN processing where eye segregation persists.
+    """
+    def __init__(self, in_ch: int, out_ch: int, kernel_size: int):
+        super().__init__()
+        self.conv_left = nn.Sequential(
+            nn.Conv2d(in_ch, out_ch, kernel_size, padding=kernel_size//2),
+            nn.BatchNorm2d(out_ch),
+            nn.ReLU(inplace=True)
+        )
+        self.conv_right = nn.Sequential(
+            nn.Conv2d(in_ch, out_ch, kernel_size, padding=kernel_size//2),
+            nn.BatchNorm2d(out_ch),
+            nn.ReLU(inplace=True)
+        )
+
+    def forward(self, x_left: torch.Tensor, x_right: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        return self.conv_left(x_left), self.conv_right(x_right)
+
+
+class StridedMonocularBlock(nn.Module):
+    """
+    Monocular pathway block with configurable stride.
+    Keeps left/right eyes separate, uses stride to control spatial sampling.
+
+    Used in V4 for stride-based pathway differentiation.
+    """
+    def __init__(self, in_ch: int, out_ch: int, kernel_size: int = 3, stride: int = 1):
+        super().__init__()
+        padding = kernel_size // 2
+        self.conv_left = nn.Sequential(
+            nn.Conv2d(in_ch, out_ch, kernel_size, stride=stride, padding=padding),
+            nn.BatchNorm2d(out_ch),
+            nn.ReLU(inplace=True)
+        )
+        self.conv_right = nn.Sequential(
+            nn.Conv2d(in_ch, out_ch, kernel_size, stride=stride, padding=padding),
+            nn.BatchNorm2d(out_ch),
+            nn.ReLU(inplace=True)
+        )
+
+    def forward(self, x_left: torch.Tensor, x_right: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        return self.conv_left(x_left), self.conv_right(x_right)
+
+
+def count_params(model: nn.Module) -> int:
+    """Count total trainable parameters."""
+    return sum(p.numel() for p in model.parameters())

mpknet_v6_2_temporal.py

@@ -0,0 +1,379 @@
+"""
+BinocularMPKNet V6.2 - Sequential Temporal M-Pathway
+
+Key insight: M processes 8 frames sequentially, computing deltas between
+consecutive frames. P sees only the current frame for detail.
+
+M stream: f0→f1→f2→f3→f4→f5→f6→f7 (8 frames, 7 deltas)
+P stream: f7 only (current frame, full detail)
+
+This is Conv2D-based temporal processing:
+- Same M weights applied to each frame
+- Deltas capture motion between consecutive frames
+- Fusion learns motion patterns (acceleration, direction change, etc.)
+
+For static images: use augmented "pseudo-frames" (scales, shifts)
+For video: use actual consecutive frames
+"""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from mpknet_components import (
+    BinocularPreMPK,
+    StereoDisparity,
+    StridedMonocularBlock,
+    count_params,
+)
+
+
+class SequentialTemporalMPathway(nn.Module):
+    """
+    M-pathway that processes 8 frames sequentially.
+
+    Computes features for each frame, then deltas between consecutive frames.
+    Fuses all 7 deltas into a single motion representation.
+    """
+    def __init__(self, in_ch: int, out_ch: int, kernel_size: int, stride: int, num_frames: int = 8):
+        super().__init__()
+
+        self.num_frames = num_frames
+        self.num_deltas = num_frames - 1
+
+        # Shared M block (same weights for all frames)
+        self.m_block = StridedMonocularBlock(in_ch, out_ch, kernel_size, stride)
+
+        # Delta processing: learn how to combine consecutive features
+        # Instead of raw subtraction, learn the comparison
+        self.delta_conv = nn.Conv2d(out_ch * 2, out_ch, kernel_size=1, bias=False)
+
+        # Temporal fusion: combine all deltas into motion features
+        self.temporal_fuse = nn.Sequential(
+            nn.Conv2d(out_ch * self.num_deltas, out_ch * 2, kernel_size=1, bias=False),
+            nn.BatchNorm2d(out_ch * 2),
+            nn.ReLU(inplace=True),
+            nn.Conv2d(out_ch * 2, out_ch, kernel_size=1, bias=False),
+            nn.BatchNorm2d(out_ch),
+            nn.ReLU(inplace=True),
+        )
+
+    def forward(self, frames_left: torch.Tensor, frames_right: torch.Tensor):
+        """
+        Args:
+            frames_left: [B, num_frames, C, H, W] - left eye frame sequence
+            frames_right: [B, num_frames, C, H, W] - right eye frame sequence
+
+        Returns:
+            motion_left, motion_right: [B, out_ch, H', W'] - motion features
+        """
+        B = frames_left.shape[0]
+
+        # Process all frames through M (shared weights)
+        feats_left = []
+        feats_right = []
+
+        for t in range(self.num_frames):
+            fl, fr = self.m_block(frames_left[:, t], frames_right[:, t])
+            feats_left.append(fl)
+            feats_right.append(fr)
+
+        # Compute deltas between consecutive frames
+        deltas_left = []
+        deltas_right = []
+
+        for t in range(self.num_deltas):
+            # Concatenate consecutive features and learn the delta
+            pair_left = torch.cat([feats_left[t], feats_left[t+1]], dim=1)
+            pair_right = torch.cat([feats_right[t], feats_right[t+1]], dim=1)
+
+            delta_l = self.delta_conv(pair_left)
+            delta_r = self.delta_conv(pair_right)
+
+            deltas_left.append(delta_l)
+            deltas_right.append(delta_r)
+
+        # Fuse all deltas into motion representation
+        all_deltas_left = torch.cat(deltas_left, dim=1)   # [B, out_ch*7, H, W]
+        all_deltas_right = torch.cat(deltas_right, dim=1)
+
+        motion_left = self.temporal_fuse(all_deltas_left)
+        motion_right = self.temporal_fuse(all_deltas_right)
+
+        return motion_left, motion_right
+
+
+class BinocularMPKNetV6_2(nn.Module):
+    """
+    Binocular MPKNet V6.2 with sequential temporal M-pathway.
+
+    For video:
+        - M sees 8 consecutive frames, extracts motion
+        - P sees current frame only, extracts detail
+        - Fusion combines motion + detail
+
+    For static images (training):
+        - Generate pseudo-frames via augmentation (scale, shift, blur)
+        - M learns to detect "change" even without real motion
+        - Transfers to real video at test time
+    """
+    def __init__(self, num_classes: int = 10, ch: int = 48,
+                 use_stereo: bool = True, disparity_range: int = 2,
+                 kernel_size: int = 5, num_frames: int = 8):
+        super().__init__()
+
+        self.use_stereo = use_stereo
+        self.kernel_size = kernel_size
+        self.num_frames = num_frames
+
+        # Fibonacci strides
+        self.p_stride = 2
+        self.k_stride = 3
+        self.m_stride = 5
+
+        if use_stereo:
+            self.stereo = StereoDisparity(disparity_range)
+
+        self.pre_mpk = BinocularPreMPK(sigma=1.0)
+
+        # ========== BLOCK 1 ==========
+        # P pathway: single frame, high detail
+        self.P_block1_layer1 = StridedMonocularBlock(3, ch, kernel_size, stride=self.p_stride)
+        self.P_block1_layer2 = StridedMonocularBlock(ch, ch, kernel_size, stride=1)
+
+        # K pathway: single frame, context
+        self.K_block1 = StridedMonocularBlock(3, ch // 2, kernel_size, stride=self.k_stride)
+
+        # M pathway: TEMPORAL - processes num_frames frames
+        self.M_block1 = SequentialTemporalMPathway(3, ch, kernel_size, stride=self.m_stride, num_frames=num_frames)
+
+        # K gates for block 1
+        self.k_gate1_M_left = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+        self.k_gate1_M_right = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+        self.k_gate1_P_left = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+        self.k_gate1_P_right = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+
+        # ========== BLOCK 2 ==========
+        self.P_block2_layer1 = StridedMonocularBlock(ch, ch, kernel_size, stride=1)
+        self.P_block2_layer2 = StridedMonocularBlock(ch, ch, kernel_size, stride=1)
+
+        self.K_block2 = StridedMonocularBlock(ch // 2, ch // 2, kernel_size, stride=1)
+
+        # M block 2: single frame processing on motion features
+        self.M_block2 = StridedMonocularBlock(ch, ch, kernel_size, stride=1)
+
+        # K gates for block 2
+        self.k_gate2_M_left = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+        self.k_gate2_M_right = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+        self.k_gate2_P_left = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+        self.k_gate2_P_right = nn.Sequential(nn.Linear(ch // 2, ch), nn.Sigmoid())
+
+        # ========== V1 FUSION ==========
+        self.v1_fusion = nn.Sequential(
+            nn.Conv2d(ch * 4, ch * 2, 1),
+            nn.BatchNorm2d(ch * 2),
+            nn.ReLU(inplace=True),
+        )
+
+        # Classification head
+        self.gap = nn.AdaptiveAvgPool2d(1)
+        self.dropout = nn.Dropout(p=0.5)
+        self.fc = nn.Linear(ch * 2, num_classes)
+
+    def forward(self, x: torch.Tensor, frames: torch.Tensor = None) -> torch.Tensor:
+        """
+        Args:
+            x: [B, C, H, W] - current frame (for P pathway)
+            frames: [B, num_frames, C, H, W] - frame sequence (for M pathway)
+                    If None, generates pseudo-frames from x
+        """
+        # Generate pseudo-frames if not provided (for static image training)
+        if frames is None:
+            frames = self._generate_pseudo_frames(x)
+
+        # Create stereo views for current frame
+        if self.use_stereo:
+            x_left, x_right = self.stereo(x)
+            # Also create stereo for all frames
+            frames_left = torch.stack([self.stereo(frames[:, t])[0] for t in range(self.num_frames)], dim=1)
+            frames_right = torch.stack([self.stereo(frames[:, t])[1] for t in range(self.num_frames)], dim=1)
+        else:
+            x_left, x_right = x, x
+            frames_left = frames
+            frames_right = frames
+
+        # Retinal preprocessing for current frame (P pathway)
+        P_left, _, P_right, _ = self.pre_mpk(x_left, x_right)
+
+        # Retinal preprocessing for all frames (M pathway)
+        M_frames_left = []
+        M_frames_right = []
+        for t in range(self.num_frames):
+            _, ml, _, mr = self.pre_mpk(frames_left[:, t], frames_right[:, t])
+            M_frames_left.append(ml)
+            M_frames_right.append(mr)
+        M_frames_left = torch.stack(M_frames_left, dim=1)  # [B, num_frames, C, H, W]
+        M_frames_right = torch.stack(M_frames_right, dim=1)
+
+        # ========== BLOCK 1 ==========
+        # K pathway (from current frame)
+        K_left, K_right = self.K_block1(P_left, P_right)
+
+        # P pathway (current frame only)
+        P_left, P_right = self.P_block1_layer1(P_left, P_right)
+        P_left, P_right = self.P_block1_layer2(P_left, P_right)
+
+        # M pathway (all frames - TEMPORAL)
+        M_left, M_right = self.M_block1(M_frames_left, M_frames_right)
+
+        # K gating
+        k_ctx1_left = self.gap(K_left).flatten(1)
+        k_ctx1_right = self.gap(K_right).flatten(1)
+
+        gate1_M_left = self.k_gate1_M_left(k_ctx1_left).unsqueeze(-1).unsqueeze(-1)
+        gate1_M_right = self.k_gate1_M_right(k_ctx1_right).unsqueeze(-1).unsqueeze(-1)
+        gate1_P_left = self.k_gate1_P_left(k_ctx1_left).unsqueeze(-1).unsqueeze(-1)
+        gate1_P_right = self.k_gate1_P_right(k_ctx1_right).unsqueeze(-1).unsqueeze(-1)
+
+        M_left = M_left * gate1_M_left
+        M_right = M_right * gate1_M_right
+        P_left = P_left * gate1_P_left
+        P_right = P_right * gate1_P_right
+
+        # ========== BLOCK 2 ==========
+        P_left, P_right = self.P_block2_layer1(P_left, P_right)
+        P_left, P_right = self.P_block2_layer2(P_left, P_right)
+
+        K_left, K_right = self.K_block2(K_left, K_right)
+
+        M_left, M_right = self.M_block2(M_left, M_right)
+
+        # K gating
+        k_ctx2_left = self.gap(K_left).flatten(1)
+        k_ctx2_right = self.gap(K_right).flatten(1)
+
+        gate2_M_left = self.k_gate2_M_left(k_ctx2_left).unsqueeze(-1).unsqueeze(-1)
+        gate2_M_right = self.k_gate2_M_right(k_ctx2_right).unsqueeze(-1).unsqueeze(-1)
+        gate2_P_left = self.k_gate2_P_left(k_ctx2_left).unsqueeze(-1).unsqueeze(-1)
+        gate2_P_right = self.k_gate2_P_right(k_ctx2_right).unsqueeze(-1).unsqueeze(-1)
+
+        M_left = M_left * gate2_M_left
+        M_right = M_right * gate2_M_right
+        P_left = P_left * gate2_P_left
+        P_right = P_right * gate2_P_right
+
+        # ========== V1 FUSION ==========
+        target_size = M_left.shape[-1]
+        if P_left.shape[-1] != target_size:
+            P_left = F.adaptive_avg_pool2d(P_left, target_size)
+            P_right = F.adaptive_avg_pool2d(P_right, target_size)
+        if K_left.shape[-1] != target_size:
+            K_left = F.adaptive_avg_pool2d(K_left, target_size)
+            K_right = F.adaptive_avg_pool2d(K_right, target_size)
+
+        z = torch.cat([M_left, M_right, P_left, P_right], dim=1)
+        z = self.v1_fusion(z)
+
+        # Classification
+        z = self.gap(z).flatten(1)
+        z = self.dropout(z)
+        return self.fc(z)
+
+    def _generate_pseudo_frames(self, x: torch.Tensor) -> torch.Tensor:
+        """
+        Generate pseudo-frames from a single image for static image training.
+
+        Creates a sequence by applying progressive transformations:
+        - Scales (zoom in/out slightly)
+        - Small translations
+        - Blur levels
+
+        This teaches M to detect "change" even without real motion.
+        """
+        B, C, H, W = x.shape
+        frames = []
+
+        # Frame 0: most different (smaller scale, slight blur)
+        frames.append(self._augment_frame(x, scale=0.85, blur=1.5))
+
+        # Frames 1-6: gradual progression toward original
+        for i in range(1, self.num_frames - 1):
+            t = i / (self.num_frames - 1)  # 0 to 1
+            scale = 0.85 + 0.15 * t  # 0.85 to 1.0
+            blur = 1.5 * (1 - t)  # 1.5 to 0
+            frames.append(self._augment_frame(x, scale=scale, blur=blur))
+
+        # Frame 7: original (current frame)
+        frames.append(x)
+
+        return torch.stack(frames, dim=1)  # [B, num_frames, C, H, W]
+
+    def _augment_frame(self, x: torch.Tensor, scale: float = 1.0, blur: float = 0.0) -> torch.Tensor:
+        """Apply scale and blur augmentation to create pseudo-frame."""
+        B, C, H, W = x.shape
+
+        # Scale
+        if scale != 1.0:
+            new_H, new_W = int(H * scale), int(W * scale)
+            x = F.interpolate(x, size=(new_H, new_W), mode='bilinear', align_corners=False)
+            # Pad or crop back to original size
+            if scale < 1.0:
+                pad_h = (H - new_H) // 2
+                pad_w = (W - new_W) // 2
+                x = F.pad(x, (pad_w, W - new_W - pad_w, pad_h, H - new_H - pad_h), mode='reflect')
+            else:
+                start_h = (new_H - H) // 2
+                start_w = (new_W - W) // 2
+                x = x[:, :, start_h:start_h+H, start_w:start_w+W]
+
+        # Blur (simple box blur approximation)
+        if blur > 0:
+            kernel_size = max(3, int(blur) * 2 + 1)
+            if kernel_size % 2 == 0:
+                kernel_size += 1
+            x = F.avg_pool2d(x, kernel_size, stride=1, padding=kernel_size//2)
+
+        return x
+
+
+if __name__ == "__main__":
+    from mpknet_v6 import BinocularMPKNetV6
+    from mpknet_v6_1 import BinocularMPKNetV6_1
+
+    print("=" * 60)
+    print("V6 vs V6.1 vs V6.2 (Temporal) Comparison")
+    print("=" * 60)
+
+    v6 = BinocularMPKNetV6(num_classes=10, ch=48)
+    v6_1 = BinocularMPKNetV6_1(num_classes=10, ch=48)
+    v6_2 = BinocularMPKNetV6_2(num_classes=10, ch=48, num_frames=8)
+
+    print(f"V6   params: {count_params(v6)/1e6:.3f}M")
+    print(f"V6.1 params: {count_params(v6_1)/1e6:.3f}M (dual-pass M)")
+    print(f"V6.2 params: {count_params(v6_2)/1e6:.3f}M (8-frame temporal M)")
+    print()
+
+    # Test forward pass with static image (pseudo-frames)
+    x = torch.randn(2, 3, 32, 32)
+
+    print("Testing V6.2 with static image (generates pseudo-frames):")
+    y = v6_2(x)
+    print(f"  Input: {x.shape} → Output: {y.shape}")
+    print()
+
+    # Test forward pass with actual video frames
+    frames = torch.randn(2, 8, 3, 32, 32)  # 8 frames
+    print("Testing V6.2 with video frames:")
+    y = v6_2(x, frames=frames)
+    print(f"  Current frame: {x.shape}")
+    print(f"  Frame sequence: {frames.shape}")
+    print(f"  Output: {y.shape}")
+    print()
+
+    print("Architecture summary:")
+    print("  P pathway: sees current frame only (detail)")
+    print("  K pathway: sees current frame only (context/gating)")
+    print("  M pathway: sees 8 frames, computes 7 deltas (motion)")
+    print()
+    print("For static images: pseudo-frames via scale/blur progression")
+    print("For video: actual consecutive frames")

mpknet_v6_2_ucf101_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a0f8e24150057f717fbf8ab369dc0b060d9e63fa9f87d3f98c2a000ea6599ff
+size 7614431