File size: 3,359 Bytes
6a176cb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | # Implementation notes
## Coding priorities
1. Do not break the baseline.
2. Add config flags for every new behavior.
3. Keep temporal memory modules isolated from detector-specific code.
4. Use small commits/patches.
5. Add shape assertions early.
## Suggested file organization
Use the repository's existing module conventions. If no better location exists, create:
```text
projects/mmdet3d_plugin/models/temporal_memory/
__init__.py
controller.py
detector_adapters.py
memory_packet.py
updaters.py
pe_handler.py
distillation.py
schedulers.py
```
Detector-specific adapters can live in:
```text
projects/mmdet3d_plugin/models/temporal_memory/adapters/
repdetr3d_adapter.py
streampetr_adapter.py
```
## Config flags
Add flags like:
```python
temporal_memory = dict(
enabled=False,
detector_adapter="repdetr3d",
keyframe_interval=3,
updater="gated_residual",
partial_layer="middle",
memory_is_pre_pe=True,
pe_mode="separate_recompute",
use_ego_conditioning=True,
use_distillation=True,
distill_weight=1.0,
train_with_teacher=True,
inference_teacher_forbidden=True,
)
```
## Debug flags
```python
tm_debug_shapes=True
tm_debug_latency=True
tm_debug_compare_full_path=False
tm_assert_no_teacher_in_inference=True
```
## Feature extraction
For EVA-style ViTs, implement intermediate feature collection carefully:
- Avoid storing every block output by default.
- Collect only requested layers.
- Use hooks only if they are robust under distributed training and checkpointing.
- Prefer explicit forward options if the backbone supports them.
## Memory cache
The memory cache should reset on:
- new scene
- sequence discontinuity
- missing previous frame
- large timestamp gap
- explicit batch boundary
The cache key should include scene/sample identity when batch size > 1.
## Batch handling
Do not assume batch size 1 unless the config requires it. If batch sequences are interleaved, memory must be keyed per sequence.
Simpler first implementation may assert:
```python
assert batch_size == 1
```
but this must be clearly documented.
## Latency measurement
Use CUDA synchronization around measured regions:
```python
torch.cuda.synchronize()
start = time.perf_counter()
...
torch.cuda.synchronize()
elapsed = time.perf_counter() - start
```
Report:
- full backbone time
- partial backbone time
- updater time
- detector head time
- total per-frame time
## Reproducibility
Log:
- config path
- checkpoint path
- git commit
- keyframe interval
- updater type
- partial layer
- PE mode
- distillation weight
- whether backbone/head were frozen
## Common failure modes
### Shape mismatch
Fix using adapter conversions, not ad-hoc reshapes in the updater.
### Stale PE
If outputs degrade badly on non-keyframes, verify whether memory includes PE. Try pre-PE memory and recomputed PE.
### No speedup
Profile updater and partial feature extraction. Cross-attention may be too expensive.
### Distillation improves but detection does not
The distilled feature may match globally but lose object-critical detail. Add foreground-weighted feature distillation only after the simple loss is tested.
### Training collapse
Reduce update magnitude:
```text
M_t = M_{t-1} + alpha * sigmoid(gate) * residual
```
Initialize `alpha` small, e.g. 0.1.
|