Implementation notes

Coding priorities

Do not break the baseline.
Add config flags for every new behavior.
Keep temporal memory modules isolated from detector-specific code.
Use small commits/patches.
Add shape assertions early.

Suggested file organization

Use the repository's existing module conventions. If no better location exists, create:

projects/mmdet3d_plugin/models/temporal_memory/
    __init__.py
    controller.py
    detector_adapters.py
    memory_packet.py
    updaters.py
    pe_handler.py
    distillation.py
    schedulers.py

Detector-specific adapters can live in:

projects/mmdet3d_plugin/models/temporal_memory/adapters/
    repdetr3d_adapter.py
    streampetr_adapter.py

Config flags

Add flags like:

temporal_memory = dict(
    enabled=False,
    detector_adapter="repdetr3d",
    keyframe_interval=3,
    updater="gated_residual",
    partial_layer="middle",
    memory_is_pre_pe=True,
    pe_mode="separate_recompute",
    use_ego_conditioning=True,
    use_distillation=True,
    distill_weight=1.0,
    train_with_teacher=True,
    inference_teacher_forbidden=True,
)

Debug flags

tm_debug_shapes=True
tm_debug_latency=True
tm_debug_compare_full_path=False
tm_assert_no_teacher_in_inference=True

Feature extraction

For EVA-style ViTs, implement intermediate feature collection carefully:

Avoid storing every block output by default.
Collect only requested layers.
Use hooks only if they are robust under distributed training and checkpointing.
Prefer explicit forward options if the backbone supports them.

Memory cache

The memory cache should reset on:

new scene
sequence discontinuity
missing previous frame
large timestamp gap
explicit batch boundary

The cache key should include scene/sample identity when batch size > 1.

Batch handling

Do not assume batch size 1 unless the config requires it. If batch sequences are interleaved, memory must be keyed per sequence.

Simpler first implementation may assert:

assert batch_size == 1

but this must be clearly documented.

Latency measurement

Use CUDA synchronization around measured regions:

torch.cuda.synchronize()
start = time.perf_counter()
...
torch.cuda.synchronize()
elapsed = time.perf_counter() - start

Report:

full backbone time
partial backbone time
updater time
detector head time
total per-frame time

Reproducibility

Log:

config path
checkpoint path
git commit
keyframe interval
updater type
partial layer
PE mode
distillation weight
whether backbone/head were frozen

Common failure modes

Shape mismatch

Fix using adapter conversions, not ad-hoc reshapes in the updater.

Stale PE

If outputs degrade badly on non-keyframes, verify whether memory includes PE. Try pre-PE memory and recomputed PE.

No speedup

Profile updater and partial feature extraction. Cross-attention may be too expensive.

Distillation improves but detection does not

The distilled feature may match globally but lose object-critical detail. Add foreground-weighted feature distillation only after the simple loss is tested.

Training collapse

Reduce update magnitude:

M_t = M_{t-1} + alpha * sigmoid(gate) * residual

Initialize alpha small, e.g. 0.1.