System Design Proposal: Layered Cognitive Memory (LCM) for Bounded-VRAM Long-Horizon Generation
I am not an ML engineer or researcher. I am someone interested in AI video generation who has spent a lot of time thinking about why long-form generation tends to break down.
From the outside, many of the failures seem less like an intelligence problem and more like a memory problem.
Current systems often have to choose between two bad options:
- Keep more and more context alive, which increases compute and memory costs.
- Discard old context, which causes identity drift, object loss, forgotten instructions, and continuity failures.
That led me to a simple question:
What if model memory behaved more like human memory?
Humans do not remember every moment of experience with equal precision.
Recent events remain vivid.
Important events are remembered in summary form.
Distant memories become compressed concepts that can still be recalled later if needed.
Could generative models use a similar hierarchy?
Proposed Memory Layers
1. Active Memory
This is the current working context.
For LLMs:
- active tokens
- current reasoning state
- immediate conversation context
For video models:
- current generation chunk
- nearby latent frames
- current motion information
This receives full attention and full fidelity.
Its purpose is immediate reasoning and short-term continuity.
2. Reference Memory
This stores a small number of recent high-fidelity states.
Examples:
- recent keyframes
- character identity references
- active objects
- current environment state
- recent instructions
- current goals
This layer acts as a continuity anchor.
The model can forget distant details without forgetting who a character is or what task it is currently performing.
Conceptually:
Current Window
↓
Reference Cache
↓
Long-Term Memory
3. Episodic Memory
Older context is compressed into compact memory representations.
My original thought was a small spatial memory grid similar to a QR code.
Not an image thumbnail.
Not a blurry low-resolution frame.
A machine-readable latent representation that preserves the information needed for future continuity.
Examples of information that might survive compression:
- object relationships
- scene geometry
- camera movement
- character positions
- lighting state
- important actions
- story events
- environmental layout
Fine detail disappears.
Structural information remains.
The exact implementation could be:
- learned memory tokens
- VAE latents
- VQ codes
- latent grids
- compressed embeddings
The important idea is not the format.
The important idea is that older memories become cheaper instead of simply disappearing.
4. Retrieval
The model should not attend to every historical memory at every generation step.
Instead:
Current State
↓
Memory Query
↓
Retrieve Relevant Memories
↓
Inject Retrieved Context
↓
Continue Generation
Examples:
- a character returns after several minutes
- an object reappears
- the camera returns to a previous location
- an old conversation topic becomes relevant again
This turns memory into a retrieval problem rather than a quadratic attention problem.
5. Memory Consolidation
Not every memory deserves permanent storage.
Repeated information can be:
- merged
- compressed further
- archived
- removed
Important information survives.
Redundant information fades.
The goal is bounded active memory rather than unlimited active memory.
Two-Phase Generation
I have also wondered whether generation itself benefits from being split into separate jobs.
Phase 1: Structural Generation
Generate lower-resolution content focused on:
- motion
- identity
- object permanence
- scene layout
- camera movement
- temporal consistency
This is the expensive spatiotemporal problem.
Phase 2: Spatial Refinement
After structure exists, refine it in small chunks.
Focus on:
- textures
- facial detail
- hair
- clothing
- lighting quality
- fine edges
Since refinement only requires local temporal context, it could potentially process small frame batches, save results to disk, clear memory, and continue.
That means runtime increases with video length, but peak VRAM usage may remain relatively stable.
Why This Seems Interesting
Many current systems appear to behave like this:
High Detail
High Detail
High Detail
Context Limit
Forget Everything
The idea here is closer to:
High Detail
Reference Detail
Compressed Memory
Archived Memory
Retrievable History
The model does not remember everything equally.
It remembers different things at different resolutions.
Questions For People Building These Systems
Does this resemble existing memory architectures in video generation or LLM research?
Is hierarchical memory already being explored in current long-video work?
Would the hardest problem be:
- compression
- retrieval
- training
- memory selection
Does separating identity memory from scene memory make sense?
Are there obvious failure modes I am missing?
I am not presenting this as a solved problem or a breakthrough.
I am simply curious whether a layered memory hierarchy could help extend long-horizon generation without requiring models to keep every moment of history alive simultaneously.
If similar ideas already exist, I would love paper recommendations or examples of systems that explore related concepts.