| # AGENTS.md |
|
|
| This repository implements temporal memory updating for efficient camera-only 3D detection. |
|
|
| The project should support both StreamPETR-style and RepDETR3D-style detectors. Do not hard-code the implementation to one detector unless the task explicitly asks for a detector-specific patch. |
|
|
| ## Context policy |
|
|
| Do not read every markdown file by default. Start with this file and `docs/PROJECT_BRIEF.md`. Then read only the design note relevant to the requested change: |
|
|
| - Architecture or module design: `docs/ARCHITECTURE.md` |
| - 3D position embedding, ego-motion, or coordinate-frame issues: `docs/PE_AND_EGOMOTION.md` |
| - Training losses, teacher features, or freezing strategy: `docs/DISTILLATION.md` |
| - StreamPETR vs RepDETR3D compatibility: `docs/DETECTOR_ADAPTERS.md` |
| - Experiment configs and ablations: `docs/EXPERIMENTS.md` |
| - Repo integration and coding rules: `docs/IMPLEMENTATION_NOTES.md` |
| - Literature framing: `docs/RELATED_WORK_NOTES.md` |
|
|
| When context is limited, prefer reading the most specific file rather than loading all docs. |
|
|
| ## Non-negotiable goals |
|
|
| 1. Preserve baseline behavior when temporal memory is disabled. |
| 2. Implement the memory updater behind a detector-agnostic adapter interface. |
| 3. Separate appearance memory from 3D positional encoding whenever possible. |
| 4. Include a naive reuse baseline before implementing complicated update modules. |
| 5. Include full-encoder teacher distillation for non-keyframe memory updates. |
| 6. Measure actual wall-clock latency, GPU memory, mAP, and NDS. FLOPs alone are insufficient. |
|
|
| ## Definitions |
|
|
| - Full memory: final image-token/image-feature representation produced by the full backbone/neck path on a keyframe. |
| - Partial feature: intermediate ViT/block feature from the current frame, computed with less depth or cheaper processing. |
| - Updated memory: feature passed to the detector head/decoder on non-keyframes after combining previous memory with current partial features. |
| - Keyframe: frame where the full encoder path is run. |
| - Non-keyframe: frame where the full final memory is approximated by the updater. |
|
|
| ## Expected implementation style |
|
|
| Prefer small, testable modules: |
|
|
| - `TemporalMemoryController`: decides keyframe vs non-keyframe and owns cached memory. |
| - `DetectorAdapter`: extracts/injects features for StreamPETR or RepDETR3D. |
| - `MemoryUpdater`: combines previous memory and current partial features. |
| - `PEHandler`: handles 3D PE recomputation or ego-motion-aware transformation. |
| - `DistillationLoss`: optional teacher loss between updated memory and full current memory. |
|
|
| Keep existing configs intact. Add new configs instead of mutating baseline configs. |
|
|
| ## Minimum tests before training |
|
|
| - With memory disabled, outputs should match baseline within numerical tolerance. |
| - With `keyframe_interval=1`, outputs should match the full path. |
| - With naive reuse, no crash across a sequence. |
| - With memory update enabled, tensor shapes must match the detector input exactly. |
| - Ego-motion transformation code must have an identity-transform test. |
|
|
| ## Naming |
|
|
| Use `tm_` or `temporal_memory_` prefixes for new modules, configs, and flags. |
