# AGENTS.md

This repository implements temporal memory updating for efficient camera-only 3D detection.

The project should support both StreamPETR-style and RepDETR3D-style detectors. Do not hard-code the implementation to one detector unless the task explicitly asks for a detector-specific patch.

## Context policy

Do not read every markdown file by default. Start with this file and `docs/PROJECT_BRIEF.md`. Then read only the design note relevant to the requested change:

- Architecture or module design: `docs/ARCHITECTURE.md`
- 3D position embedding, ego-motion, or coordinate-frame issues: `docs/PE_AND_EGOMOTION.md`
- Training losses, teacher features, or freezing strategy: `docs/DISTILLATION.md`
- StreamPETR vs RepDETR3D compatibility: `docs/DETECTOR_ADAPTERS.md`
- Experiment configs and ablations: `docs/EXPERIMENTS.md`
- Repo integration and coding rules: `docs/IMPLEMENTATION_NOTES.md`
- Literature framing: `docs/RELATED_WORK_NOTES.md`

When context is limited, prefer reading the most specific file rather than loading all docs.

## Non-negotiable goals

1. Preserve baseline behavior when temporal memory is disabled.
2. Implement the memory updater behind a detector-agnostic adapter interface.
3. Separate appearance memory from 3D positional encoding whenever possible.
4. Include a naive reuse baseline before implementing complicated update modules.
5. Include full-encoder teacher distillation for non-keyframe memory updates.
6. Measure actual wall-clock latency, GPU memory, mAP, and NDS. FLOPs alone are insufficient.

## Definitions

- Full memory: final image-token/image-feature representation produced by the full backbone/neck path on a keyframe.
- Partial feature: intermediate ViT/block feature from the current frame, computed with less depth or cheaper processing.
- Updated memory: feature passed to the detector head/decoder on non-keyframes after combining previous memory with current partial features.
- Keyframe: frame where the full encoder path is run.
- Non-keyframe: frame where the full final memory is approximated by the updater.

## Expected implementation style

Prefer small, testable modules:

- `TemporalMemoryController`: decides keyframe vs non-keyframe and owns cached memory.
- `DetectorAdapter`: extracts/injects features for StreamPETR or RepDETR3D.
- `MemoryUpdater`: combines previous memory and current partial features.
- `PEHandler`: handles 3D PE recomputation or ego-motion-aware transformation.
- `DistillationLoss`: optional teacher loss between updated memory and full current memory.

Keep existing configs intact. Add new configs instead of mutating baseline configs.

## Minimum tests before training

- With memory disabled, outputs should match baseline within numerical tolerance.
- With `keyframe_interval=1`, outputs should match the full path.
- With naive reuse, no crash across a sequence.
- With memory update enabled, tensor shapes must match the detector input exactly.
- Ego-motion transformation code must have an identity-transform test.

## Naming

Use `tm_` or `temporal_memory_` prefixes for new modules, configs, and flags.