RetViM — Deep Research & Clinical Intelligence

IEEE Conference Paper · 2026

RetViM: Sequential Hybrid Vision Transformer with MedMamba for Retinal Disease Classification

Khamir Desai, Mayuri A. Mehta, Sree Saicharan Vadapalli · Sarvajanik College of Engineering and Technology, Surat, India

Abstract Early detection of retinal diseases such as Diabetic Macular Edema (DME), Choroidal Neovascularization (CNV) and Drusen is crucial for preventing irreversible vision loss. This paper proposes RetViM, a novel sequential hybrid method combining Vision Transformers (ViT) and Modified MedMamba for retinal disease classification using OCT B-scan images. RetViM achieves 99.90% accuracy, 99.90% precision, 99.90% recall, 99.97% specificity, 99.90% F1-score, AUC-ROC of 1.00 on the test set and 0.9986 Cohen's kappa — with only 1 misclassification out of 968 test images.

Deep Learning Vision Transformer MedMamba State Space Model Optical Coherence Tomography Retinal Disease

Architecture

Sequential Hybrid
ViT-MedMamba

ViT-B/16 captures global long-range dependencies. Modified MedMamba refines with local Conv + selective State-Space modeling.

Fig 1 — RetViM Architecture: OCT B-scan → ViT-B/16 (6 frozen + 6 trainable blocks) → 2 Modified MedMamba blocks (Conv ∥ SSM → MLP fusion) → Multi-scale pooling → 4-class softmax

Architecture

Generated architecture visualization · 101.2M params · 12 ViT blocks (6 frozen) + 2 SS-Conv-SSM blocks · Multi-scale pooling fusion

Experimental Results · Kermany OCT 2017

Benchmark
Performance

All results on the test set (968 images, 242 per class), completely untouched during training and validation.

Table VIII — Full-Scale Model Comparison

Model	Acc %	Prec %	Recall %	Spec %	F1 %	AUC-ROC	Kappa	Errors
ViT-Base (Baseline)	99.38	99.38	99.38	99.79	99.39	1.0000	0.9917	6
Hybrid ViT-MedMamba v1	99.07	99.10	99.07	99.69	99.07	0.9999	0.9876	9
RetViM (Proposed) ★	99.90	99.90	99.90	99.97	99.90	1.0000	0.9986	1

Table IX — Per-Class Performance of RetViM

Class	Precision	Recall	Specificity	F1	AUC	Support
CNV	1.0000	1.0000	1.0000	1.0000	1.0000	242
DME	1.0000	1.0000	1.0000	1.0000	1.0000	242
DRUSEN	0.9959	1.0000	0.9986	0.9979	1.0000	242
NORMAL	1.0000	0.9959	1.0000	0.9979	1.0000	242
Macro Avg	0.9990	0.9990	0.9997	0.9990	1.0000	968

Confusion Matrix — 968 Test Images

Misclassification Analysis

1 Error out of 968 Images

Sample #862: NORMAL misclassified as DRUSEN
Decision margin: 1.61%

Why this is clinically acceptable:
1. Decision margin is only 1.61% — genuine ambiguity
2. Direction is conservative (false DRUSEN) — prompts monitoring rather than dismissal
3. The image exhibits subtle RPE irregularities creating real ambiguity

100%

CNV & DME Recall

99.90%

Overall Accuracy

0.9986

Cohen's Kappa

Table VI — Ablation Study (16,102 training images)

Model	Accuracy %	F1 %	AUC-ROC	Epochs	Key Finding
ViT-Small	98.97	98.97	0.9999	50	Self-attention highly effective for OCT
Hybrid (4 blk)	98.55	98.56	0.9998	80	Over-regularization from 4 blocks
MedMamba	97.11	97.11	0.9995	150	Slow convergence, needs 3× epochs

Table IV — Kermany OCT 2017 Dataset Distribution

CNV

33,509

30,258 train · 3,009 val · 242 test

DME

10,213

9,192 train · 779 val · 242 test

DRUSEN

7,768

6,992 train · 534 val · 242 test

NORMAL

33,006

28,731 train · 4,033 val · 242 test

Total: 84,496 OCT B-scan images · 75,173 train · 8,355 val · 968 test (balanced)

Feature Transformation · PyTorch

The Neural
Journey

Watch how a retinal OCT scan transforms through every layer of the hybrid ViT-Mamba pipeline. Row 1: smoothed feature heatmaps · Row 2: raw single-channel activations · Row 3: attention overlays (where the network looks). Compare Demo vs Real model weights side-by-side.

Demo vs Real

Model Architecture

12 ViT

Transformer Blocks

4 vs 2

Mamba Blocks

384 vs 768

Embedding Dim

96.68%

Real Model Val Acc

Demo · RetViMNet dim=384 · 12ViT + 4Mamba

Demo Weights

Real · ImprovedMedMamba dim=768 · 12ViT + 2Mamba

Real Weights · 96.68%

Rows:

Feature Heatmaps (smoothed spatial mean)

Raw Activations (high-variance channel)

Attention Overlays (jet blend + contours)

ViT blocks

Mamba blocks

Live Analysis

Two-mode
Intelligence

XAI Suite · CKA Matrix · Mamba Engine · Layer Inspector · Feature Space · Performance

OCT B-Scan InputSelect image

Drop OCT B-scan here

PNG, JPG · 224×224+

Sample scans:

Select a sample · click Analyze

Researcher: 6 panels · Doctor: 4 panels

RetViM Neural Engine v4.0

CPU

$ retvim --mode researcher --input scan.png

Initializing... 0%

PREDICTION

Class probabilities

Probability donut

Metric radar

Threshold curve

Full metrics

Explainability method:

Original OCT

GradCAM heatmap

Overlay (opacity: 50%)

0% 90%

Top-10% attribution

Red = positive · Blue = negative attribution

Attribution histogram

Precomputed reference · full validation set statistics

CKA Similarity Matrix — ViT backbone × SS-Conv-SSM Mamba blocks

Mamba Blk-1/2 CKA = 0.99: near-identical representations · ViT early layers are dissimilar from SSM · Dashed lines separate architectural boundaries

Representation entropy (bits)

Peak at ViT Blk-7 → collapse in SSM

Effective rank

Drusen drops to <1 after SSM

Feature energy

Explosion at ViT Blk-11

Patch attention rollout 14×14

Real attention rollout resampled to patch grid · teal = high attention

t-SNE feature space — last-layer embeddings

CNV

DME

Drusen

Normal

Current

Patch token L2 magnitude

L2 norm across 196 patch positions

CLS token PCA projection

First two principal components per class

MedMamba Block Analysis

Real computed outputs from each MedMamba block processing your uploaded image

Conv Branch (local)

DW-Conv 7x7/5x5/1x1 activation

SSM Branch (global)

Selective state-space activation

Fused Output

Combined conv + SSM features

Branch Dominance Map (warm=Conv, cool=SSM)

Which branch contributes more at each spatial location

Delta step size (per patch position)

Large delta = model allocates big state update at this position

Gate signal SiLU(z) (per patch position)

Controls information flow: 0=suppressed, 1=fully passed

Layer Inspector

12 layers × 12 heads — real attention patterns

Frozen L1–6 (gray) · Trainable L7–12 (teal) · CLS-to-patch attention from real forward pass

Precomputed from validation run · epoch 19 · 968 images

ROC curves — 4-class one-vs-rest

AUC computed on 968 balanced validation images (242/class)

Precision-Recall curves

Average Precision per class on validation set

Confusion matrix — 968 validation images (242 per class)

Overall accuracy: 967/968 = 99.90%

Ablation study — validation accuracy

Architecture comparison on Kermany OCT · same train/val split

Training history — 20 epochs

Best checkpoint at epoch 19 (val_acc = 96.68%)

Deep Analysis · Real Model Outputs

Research
Visualizations

Every figure generated from the actual trained RetViM model on real Kermany OCT data.

DEMO · RetViMNet 384d · 4×Mamba

Demo Weights

RetViMNet · 384-dim · ViT-Small transfer + OCT fine-tune · 12 ViT + 4 Mamba blocks

REAL · ImprovedMedMamba 768d · 2×Mamba

Real · 96.68% val acc

ImprovedMedMamba · 768-dim · ViT-Base/16 + 2 MedMamba blocks · val_acc = 96.68%

Rows:

Feature Heatmaps

Raw Activations

Attention Overlays

ViT Layers

Mamba Layers

All 4 Classes · Feature Journey

Input → Patch Embed → ViT Blk 3/7/11 → Mamba Blk 0/1/2/3 · Each row = one disease class · ViT stages: cool tones · Mamba stages: warm inferno

Feature Journey v2

Alternative feature evolution view · cleaner layout showing progression from raw pixel features to diagnostic representations

DME Feature Evolution

DME classification · attention focuses on INL/OPL cystoid spaces · Mamba blocks refine the central macular signal · attention progressively concentrates

How Network Distinguishes 4 Classes

PCA across all pipeline stages (Patch Embed → ViT Blk 3/11 → Mamba Blk 0/1/2/3) + t-SNE final layer + average final-layer activations per class · Classes become perfectly separable by Mamba stage

Disease-Specific Activation Patterns

Row = disease class · Columns = ViT Blk 3, ViT Blk 11, Mamba 0, Mamba 2, Mamba 3 · Each uses class-specific color: CNV=red, DME=blue, Drusen=gold, Normal=green

Class Visual Signatures

Visual signatures of each disease — Original OCT → ViT early/late features → Mamba features → final GradCAM overlay · Each class has distinct network fingerprint

Inside ViT Attention + Mamba State-Space

Top: ViT self-attention architecture (Q·Kᵀ/√d) + SS-Conv-SSM dual branch (h'=Ah+Bx, y=Ch+Dx) · Bottom rows: actual feature maps at Early ViT/Middle ViT/Late ViT/Early Mamba/Middle Mamba/Final Mamba

PCA Feature Space Across 8 Layers

8-panel PCA: Patch Embed → ViT Blk 3/7/11 → Mamba Blk 0/1/2/3 · Patch Embed: overlapping butterfly shape → ViT Blk 11: classes begin separating → Mamba: tight well-separated clusters · Var explained increases Blk-11 67.7% → Mamba 70.1%

Activation Heatmaps — All 4 Classes

What the network sees: Original → Patch Embed → ViT Blk 7 → Mamba Blk 0 → Mamba Blk 3 · Color-coded by disease: CNV=red, DME=cyan, Drusen=gold, Normal=teal · Brighter = higher activation

Feature Dictionary — Maximally Activating Patterns

8×8 grid · Rows = Patch Embed, ViT Blk 0/3/7/11 (blue), Mamba Blk 0/2/3 (red) · Columns = 8 channels · Each cell = synthetic input that maximally activates that channel · Early: color grids → Late: curved retinal structures

Hierarchical Feature Tree

Feature lineage across patch-embed → vit-3 → vit-7 → vit-11 → mamba-1 → mamba-3 · Lines show which features are preserved/transformed · Blue = ViT self-attention · Orange = Mamba state-space

DeepDream — Amplifying Network Perception

DeepDream — What Network Amplifies

Enhancing features the network detects in real OCT images · Original → ViT Block 5 (frozen, mid-level) → ViT Block 11 (trainable, semantic) → Combined · Network amplifies retinal layer boundaries and pathological textures

See inside
the machine
mind

RetViM: Sequential Hybrid Vision Transformer with MedMamba for Retinal Disease Classification

Sequential Hybrid
ViT-MedMamba

Benchmark
Performance

The Neural
Journey

Two-mode
Intelligence

Research
Visualizations

84,496 images trained.
99.90% test accuracy. 1 error in 968.

See insidethe machinemind

RetViM: Sequential Hybrid Vision Transformer with MedMamba for Retinal Disease Classification

Sequential HybridViT-MedMamba

BenchmarkPerformance

The NeuralJourney

Two-modeIntelligence

ResearchVisualizations

84,496 images trained.99.90% test accuracy. 1 error in 968.

See inside
the machine
mind

Sequential Hybrid
ViT-MedMamba

Benchmark
Performance

The Neural
Journey

Two-mode
Intelligence

Research
Visualizations

84,496 images trained.
99.90% test accuracy. 1 error in 968.