docs/AGI_MEMORY_BLUEPRINT.md

MnemoCore AGI Memory Blueprint

Toward a True Cognitive Memory Substrate for Agentic Systems

This document defines the Phase 5 “AGI Memory” architecture for MnemoCore – transforming it from a high‑end hyperdimensional memory engine into a general cognitive substrate for autonomous AI agents.

---

0. Goals & Non‑Goals

0.1 Core Goals

• Provide a plug‑and‑play cognitive memory system that any agent framework can mount as its “mind”:
  • Solves context window limits by offloading long‑term structure and recall.
  • Solves memory management by autonomous consolidation, forgetting, and self‑repair.
  • Provides new thoughts, associations and suggestions rather than only retrieval.
• Implement an explicit, formal model of:
  • Working / Short‑Term Memory (WM/STM)
  • Episodic Memory
  • Semantic Memory
  • Procedural / Skill Memory
  • Meta‑Memory & Self‑Model
• Maintain:
  • pip install mnemocore zero‑infra dev mode (SQLite / in‑process vector store).
  • Full infra path (Redis, Qdrant, k8s, MCP, OpenClaw live memory integration).[cite:436][cite:437]
• Provide clean public APIs (Python + HTTP + MCP) that:
  • Give agents a minimal but powerful surface: observe / recall / reflect / propose_change.
  • Are stable enough to build higher‑level frameworks on (LangGraph, AutoGen, OpenAI Agents, OpenClaw, custom stacks).

0.2 Non‑Goals

• MnemoCore is not:
  • En LLM eller policy‑generator.
  • En komplett agentram – det är minnet + kognitiva processer.
• MnemoCore ska inte hårdkoda specifika LLM‑providers.
  • LLM används via abstraherad integration (SubconsciousAI, LLMIntegration) så att byte av motor är trivialt.

---

1. Cognitive Architecture Overview

1.1 High‑Level Mental Model

Systemet ska exponera en internt konsekvent kognitiv modell:

• Working Memory (WM)

  • Korttidsbuffert per agent / samtal / uppgift.
  • Håller aktuella mål, senaste steg, delresultat.
  • Living in RAM, med explicit API.

• Episodic Memory (EM)

  • Sekvens av episodes: “agent X gjorde Y i kontext Z och fick utfallet U”.
  • Tidsstämplad, med länkar mellan episoder (kedjor).
  • Riktad mot “vad hände när, i vilken ordning”.

• Semantic Memory (SM)

  • Abstraherade, konsoliderade representationer (concepts, prototypes).
  • Sammanfattningar av hundratals episoder → en “semantic anchor”.
  • Bra för svar på “vad vet jag generellt om X?”.

• Procedural Memory (PM)

  • Skills, planer, recept: “för att lösa typ‑X problem, gör följande steg …”.
  • Kan hålla både mänsklig läsbar text och exekverbar kod (snippets, tools).

• Meta‑Memory (MM)

  • Självmodell för MnemoCore själv: prestanda, reliability, konfiguration, kända svagheter.
  • Driver självförbättringsloopen.

Alla dessa lever ovanpå din befintliga HDV/VSA‑kärna, tier manager, synapse index, subconscious loop osv.[cite:436][cite:437]

1.2 New Core Services

Föreslagna nya Python‑moduler (under src/mnemocore/core):

• memory_model.py
  • Typed dataklasser för WM/EM/SM/PM/MM entities.
• working_memory.py
  • WM implementation per agent/task med snabb caching.
• episodic_store.py
  • Episodisk tidsserie, sekvens‑API.
• semantic_store.py
  • Wrapper ovanpå befintlig vektorstore (Qdrant/HDV/HNSW) + consolidation hooks.
• procedural_store.py
  • Lagret för skills, scripts, tool definitions.
• meta_memory.py
  • Självmodell, logik för self‑improvement proposals.
• pulse.py
  • “Heartbeat”‑loop: driver subtle thoughts, consolidation ticks, gap detection, self‑reflection.
• agent_profile.py
  • Persistent profil per agent: preferenser, styrkor/svagheter, quirks.

---

2. Data Model

2.1 Working Memory (WM)

# src/mnemocore/core/memory_model.py

@dataclass
class WorkingMemoryItem:
    id: str
    agent_id: str
    created_at: datetime
    ttl_seconds: int
    content: str
    kind: Literal["thought", "observation", "goal", "plan_step", "meta"]
    importance: float  # 
    tags: list[str]
    hdv: BinaryHDV | None

@dataclass
class WorkingMemoryState:
    agent_id: str
    items: list[WorkingMemoryItem]
    max_items: int

Invariantes:

• WM är liten (t.ex. 32–128 items per agent).
• WM ligger primärt i RAM; kan serialiseras till Redis/SQLite för persistens.
• Access är O(1)/O(log n); LRU + importance‑vägning vid evicering.

2.2 Episodic Memory (EM)

@dataclass
class EpisodeEvent:
    timestamp: datetime
    kind: Literal["observation", "action", "thought", "reward", "error"]
    content: str
    metadata: dict[str, Any]
    hdv: BinaryHDV

@dataclass
class Episode:
    id: str
    agent_id: str
    started_at: datetime
    ended_at: datetime | None
    goal: str | None
    context: str | None   # project / environment
    events: list[EpisodeEvent]
    outcome: Literal["success", "failure", "partial", "unknown"]
    reward: float | None
    links_prev: list[str]  # previous episode IDs
    links_next: list[str]  # next episode IDs
    ltp_strength: float
    reliability: float

2.3 Semantic Memory (SM)

@dataclass
class SemanticConcept:
    id: str
    label: str                     # "fastapi-request-validation"
    description: str
    tags: list[str]
    prototype_hdv: BinaryHDV
    support_episode_ids: list[str] # episodes som gav upphov
    reliability: float
    last_updated_at: datetime
    metadata: dict[str, Any]

Kopplas direkt mot consolidation/semantic_consolidation + codebook/immunology.[cite:436][cite:437]

2.4 Procedural Memory (PM)

@dataclass
class ProcedureStep:
    order: int
    instruction: str
    code_snippet: str | None
    tool_call: dict[str, Any] | None

@dataclass
class Procedure:
    id: str
    name: str
    description: str
    created_by_agent: str | None
    created_at: datetime
    updated_at: datetime
    steps: list[ProcedureStep]
    trigger_pattern: str           # "if user asks about X and Y"
    success_count: int
    failure_count: int
    reliability: float
    tags: list[str]

Procedurer kan genereras av LLM (SubconsciousAI), testas i episodiskt minne, och sedan promotas/demotas med reliability‑loop.

2.5 Meta‑Memory (MM)

@dataclass
class SelfMetric:
    name: str         # "hot_tier_hit_rate", "avg_query_latency_ms"
    value: float
    window: str       # "5m", "1h", "24h"
    updated_at: datetime

@dataclass
class SelfImprovementProposal:
    id: str
    created_at: datetime
    author: Literal["system", "agent", "human"]
    title: str
    description: str
    rationale: str
    expected_effect: str
    status: Literal["pending", "accepted", "rejected", "implemented"]
    metadata: dict[str, Any]

MM lagras delvis i vanlig storage (SM/PM) men har egen API‑yta.

---

3. Service Layer Design

3.1 Working Memory Service

Fil: src/mnemocore/core/working_memory.py

Ansvar:

• Hålla en per‑agent WM‑state.
• Explicita operationer:
  • push_item(agent_id, item: WorkingMemoryItem)
  • get_state(agent_id) -> WorkingMemoryState
  • clear(agent_id)
  • prune(agent_id) – enligt importance + LRU.
• Integrera med engine/query:
  • Vid varje query: WM får en snapshot av top‑K resultat som “context items”.
  • Vid svar: agent kan markera vilka items som var relevanta.

3.2 Episodic Store Service

Fil: src/mnemocore/core/episodic_store.py

Ansvar:

• Skapa och uppdatera Episodes:
  • start_episode(agent_id, goal, context) -> episode_id
  • append_event(episode_id, kind, content, metadata)
  • end_episode(episode_id, outcome, reward)
• Query:
  • get_episode(id)
  • get_recent(agent_id, limit, context)
  • find_similar_episodes(hdv, top_k)
• Koppling till befintlig HDV + tier manager:
  • Varje Episode får en “episode_hdv” (bundle över event‑HDVs).
  • LTP + reliabilitet följer samma formel som övrig LTP.

3.3 Semantic Store Service

Fil: src/mnemocore/core/semantic_store.py

Ansvar:

• Hålla SemanticConcepts + codebook.
• API:
  • upsert_concept(concept: SemanticConcept)
  • find_nearby_concepts(hdv, top_k)
  • get_concept(id)
• Hookar mot:
  • semantic_consolidation.py → abstraktioner / anchors.
  • immunology.py → attractor cleanup.
  • recursive_synthesizer.py → djup konceptsyntes.

3.4 Procedural Store Service

Fil: src/mnemocore/core/procedural_store.py

Ansvar:

• Lagra och hämta Procedures.
• API:
  • store_procedure(proc: Procedure)
  • get_procedure(id)
  • find_applicable_procedures(query, agent_id)
  • record_procedure_outcome(id, success: bool)
• Integrera med:
  • SubconsciousAI → generera nya procedurer från pattern i EM/SM.
  • Reliability‑loopen → promota “verified” skills.

3.5 Meta Memory Service

Fil: src/mnemocore/core/meta_memory.py

Ansvar:

• Hålla SelfMetrics + SelfImprovementProposals.
• API:
  • record_metric(metric: SelfMetric)
  • list_metrics(filter...)
  • create_proposal(...)
  • update_proposal_status(id, status)
• Integrera med:
  • Pulse → skanna metrics och föreslå ändringar.
  • LLM → generera förslagstexter (“self‑reflection reports”).

---

4. Pulse & Subtle Thoughts

4.1 Pulse Definition

Fil: src/mnemocore/core/pulse.py

“Pulsen” är en central loop (async task, cron, eller k8s CronJob) som:

• Kör med konfigurerbart intervall (t.ex. var 10:e sekund–var 5:e minut).
• Har ett definierat set “ticks”:

class PulseTick(Enum):
    WM_MAINTENANCE = "wm_maintenance"
    EPISODIC_CHAINING = "episodic_chaining"
    SEMANTIC_REFRESH = "semantic_refresh"
    GAP_DETECTION = "gap_detection"
    INSIGHT_GENERATION = "insight_generation"
    PROCEDURE_REFINEMENT = "procedure_refinement"
    META_SELF_REFLECTION = "meta_self_reflection"

Pulse orchestrerar:

• WM_MAINTENANCE

  • Prune WM per agent.
  • Lyfta nyligen viktiga items (“keep in focus”).

• EPISODIC_CHAINING

  • Skapa/länka episodiska sekvenser (prev/next).
  • “Temporala narrativ”.

• SEMANTIC_REFRESH

  • Uppdatera semantic concepts baserat på nya episoder.
  • Trigga immunology cleanup för drift.

• GAP_DETECTION

  • Kör GapDetector över EM/SM sista N minuter/timmar.
  • Producera strukturerade knowledge gaps.

• INSIGHT_GENERATION

  • Kör SubconsciousAI/LLM över utvalda kluster.
  • Skapar nya SemanticConcepts, Procedures, eller MetaProposals.

• PROCEDURE_REFINEMENT

  • Uppdatera reliability över PM.
  • Flagga outdated/farliga procedures.

• META_SELF_REFLECTION

  • Sammanfattar senaste metriker, gap, failures → SelfImprovementProposals.

4.2 Pulse Implementation Sketch

# src/mnemocore/core/pulse.py

class Pulse:
    def __init__(self, container, config):
        self.container = container
        self.config = config
        self._running = False

    async def start(self):
        self._running = True
        while self._running:
            start = datetime.utcnow()
            await self.tick()
            elapsed = (datetime.utcnow() - start).total_seconds()
            await asyncio.sleep(max(0, self.config.pulse_interval_seconds - elapsed))

    async def tick(self):
        await self._wm_maintenance()
        await self._episodic_chaining()
        await self._semantic_refresh()
        await self._gap_detection()
        await self._insight_generation()
        await self._procedure_refinement()
        await self._meta_self_reflection()

Konfiguration i config.yaml:

haim:
  pulse:
    enabled: true
    interval_seconds: 30
    max_agents_per_tick: 50
    max_episodes_per_tick: 200

---

5. Agent‑Facing APIs (Python & HTTP & MCP)

5.1 High‑Level Python API

Fil: src/mnemocore/agent_interface.py

Syfte: ge agent‑kod ett ENKELT API:

class CognitiveMemoryClient:
    def __init__(self, engine: HAIMEngine, wm, episodic, semantic, procedural, meta):
        ...

    # --- Observation & WM ---
    def observe(self, agent_id: str, content: str, **meta) -> str: ...
    def get_working_context(self, agent_id: str, limit: int = 16) -> list[WorkingMemoryItem]: ...

    # --- Episodic ---
    def start_episode(self, agent_id: str, goal: str, context: str | None = None) -> str: ...
    def append_event(self, episode_id: str, kind: str, content: str, **meta) -> None: ...
    def end_episode(self, episode_id: str, outcome: str, reward: float | None = None) -> None: ...

    # --- Semantic / Retrieval ---
    def recall(self, agent_id: str, query: str, context: str | None = None,
               top_k: int = 8, modes: tuple[str, ...] = ("episodic","semantic")) -> list[dict]: ...

    # --- Procedural ---
    def suggest_procedures(self, agent_id: str, query: str, top_k: int = 5) -> list[Procedure]: ...
    def record_procedure_outcome(self, proc_id: str, success: bool) -> None: ...

    # --- Meta / Self-awareness ---
    def get_knowledge_gaps(self, agent_id: str, lookback_hours: int = 24) -> list[dict]: ...
    def get_self_improvement_proposals(self) -> list[SelfImprovementProposal]: ...

5.2 HTTP Layer Additions

Utöver befintliga /store, /query, /feedback, osv.[cite:437]

Nya endpoints:

• POST /wm/observe
• GET /wm/{agent_id}
• POST /episodes/start
• POST /episodes/{id}/event
• POST /episodes/{id}/end
• GET /episodes/{id}
• GET /agents/{agent_id}/episodes
• GET /agents/{agent_id}/context
• GET /agents/{agent_id}/knowledge-gaps
• GET /procedures/search
• POST /procedures/{id}/feedback
• GET /meta/proposals
• POST /meta/proposals

5.3 MCP Tools

Utöka mnemocore.mcp.server med nya verktyg:

• store_observation
• recall_context
• start_episode, end_episode
• query_memory
• get_knowledge_gaps
• get_self_improvement_proposals

Så att Claude/GPT‑agenter kan:

• “Titta in” i agentens egen historik.
• Få WM + relevanta episoder + semantic concepts innan svar.
• Få gaps och self‑reflection prompts.

---

6. Self‑Improvement Loop

6.1 Loop Definition

Målet: MnemoCore ska ständigt förbättra sig:

1. Samlar metrics (performance + quality).
2. Upptäcker systematiska brister (höga felrates, gap‑clusters).
3. Genererar SelfImprovementProposals via LLM.
4. Låter människa eller meta‑agent granska & appliera.

6.2 Pipeline

1. Metrics Collection

   • Utnyttja befintlig metrics.py + Prometheus.[cite:436][cite:437]
   • Exempelmetriker:
     • query_hit_rate, retrieval_latency_ms
     • feedback_success_rate, feedback_failure_rate
     • hot_tier_size, tier_promotion_rate
     • gap_detection_count, gap_fill_count

2. Issue Detection (Rule‑Based)

   • Batchjobb (Pulse) kör enkla regler:
     • Om feedback_failure_rate > X för en viss tag (t.ex. “fastapi”) → skapa “knowledge area weak” flagg.
     • Om hot_tier_hit_rate < threshold → dålig context‑masking eller fel tuned thresholds.

3. Proposal Generation (LLM)

   • SubconsciousAI får inputs:
     • Metrics, knowledge gaps, failure cases, config snapshot.
   • Prompt genererar:
     • SelfImprovementProposal.title/description/rationale.

4. Review & Execution

   • API / UI för att lista proposals.
   • Människa/agent accepterar/rejectar.
   • Vid accept:
     • Kan trigga config ändringar (med patch PR).
     • Kan skapa GitHub issues/PR mallar.

6.3 API

• GET /meta/proposals
• POST /meta/proposals/{id}/status

---

7. Association & “Subtle Thoughts”

7.1 Association Engine

Målet: Systemet ska själv föreslå:

• Analogier (“det här liknar när vi gjorde X i annat projekt”).
• Relaterade koncept (“du pratar om Y, men Z har varit viktigt tidigare”).
• Långsiktiga teman och lärdomar.

Bygg vidare på:

• synapse_index.py (hebbian connections).[cite:436]
• ripple_context.py (kaskader).[cite:436]
• recursive_synthesizer.py (konceptsyntes).[cite:436]

Nya pattern:

• Vid varje Pulse:
  • Hämta senaste N episoder.
  • Kör k‑NN i semantic concept space.
  • Kör ripple over synapses.
  • Generera en uppsättning CandidateAssociations:

@dataclass
class CandidateAssociation:
    id: str
    agent_id: str
    created_at: datetime
    source_episode_ids: list[str]
    related_concept_ids: list[str]
    suggestion_text: str
    confidence: float

Lagra i SM/EM så att agent/LLM kan hämta “subtle thoughts” innan svar:

• GET /agents/{agent_id}/subtle-thoughts

---

8. Storage Backends & Profiles

8.1 Profiles

Behåll pip‑enkelheten via profiler:

• Lite Profile (default, no extra deps):
  • WM: in‑process dict
  • EM: SQLite
  • SM: in‑process HDV + mmap
  • PM/MM: SQLite/JSON
• Standard Profile:
  • WARM: Redis
  • COLD: filesystem
• Scale Profile:
  • WARM: Redis
  • COLD: Qdrant (eller annan vector DB)
  • Optionellt: S3 archive

Konfigurationsexempel:

haim:
  profile: "lite"  # "lite" | "standard" | "scale"

---

9. OpenClaw & External Agents

9.1 Designprincip för integration

För OpenClaw / liknande orchestrators:

• En agent definieras genom:
  • agent_id
  • capabilities (tools etc.)
• MnemoCore ska behandla agent_id som primär nyckel för:
  • WM
  • Episoder
  • Preferenser
  • Procedurer som agenten själv skapat

9.2 “Live Memory” Pattern

• När OpenClaw kör:
  • Varje observation → observe(agent_id, content, meta)
  • Varje tool call / action → episod event.
  • Före varje beslut:
    • Hämta:
      • WM
      • recent episodes
      • relevant semantic concepts
      • subtle thoughts / associations
      • knowledge gaps (om agenten vill använda dessa som frågor).

---

10. Testing & Evaluation Plan

10.1 Unit & Integration Tests

Nya testfiler:

• tests/test_working_memory.py
• tests/test_episodic_store.py
• tests/test_semantic_store.py
• tests/test_procedural_store.py
• tests/test_meta_memory.py
• tests/test_pulse.py
• tests/test_agent_interface.py

Fokus:

• Invariantes (max WM size, LTP thresholds, reliability‑update).
• Episodic chaining korrekt.
• Semantic consolidation integration med nya SM‑API:t.
• Pulse tick ordering & time budget.

10.2 Behavioural Benchmarks

Skapa benchmarks/AGI_MEMORY_SCENARIOS.md:

• Multi‑session tasks där agent måste:
  • Minnas user preferences över dagar.
  • Lära sig av failed attempts (feedback).
  • Använda analogier över domäner.

Mät:

• Context reuse rate.
• Time‑to‑solve vs “no memory” baseline.
• Antal genererade self‑improvement proposals som faktiskt förbättrar outcomes.

---

11. Implementation Roadmap

Phase 5.0 – Core Structure

1. Introduce memory_model.py, working_memory.py, episodic_store.py, semantic_store.py, procedural_store.py, meta_memory.py, pulse.py.
2. Wire everything in container.py (new providers).
3. Add CognitiveMemoryClient + minimal tests.

Phase 5.1 – WM/EM/SM in Engine

4. Integrate WM into engine query/store paths.
5. Integrate EM creation in API (store/query/feedback).
6. Adapt semantic_consolidation/immunology to new SM service.

Phase 5.2 – Procedural & Association

7. Implement procedural store + reliability integration.
8. Build association engine + subtle thoughts endpoints.

Phase 5.3 – Self‑Improvement

9. Wire metrics → meta_memory → proposals via SubconsciousAI.
10. Add endpoints & optional small UI for proposals.

Phase 5.4 – Hardening & Agents

11. Harden profiles (lite/standard/scale).
12. Build reference integrations (OpenClaw, LangGraph, AutoGen).

---

12. Developer Notes

• Håll backwards compatibility på API där det går:
  • Nya endpoints → prefix v2 om nödvändigt.
  • Python API kan vara “ny high‑level layer” ovanpå befintlig HAIMEngine.
• All ny funktionalitet feature‑flaggas i config:
  • haim.pulse.enabled
  • haim.episodic.enabled
  • haim.procedural.enabled
  • etc.
• Strikt logging / metrics för allt nytt:
  • haim_pulse_tick_duration_seconds
  • haim_wm_size
  • haim_episode_count
  • haim_procedure_success_rate
  • haim_self_proposals_pending

---

This blueprint is the contract between MnemoCore, its agents, and its contributors. The intention is to let autonomous AI agents, human developers, and MnemoCore itself co‑evolve toward a truly cognitive memory substrate – one that remembers, forgets, reflects, and grows.