Hugging Face's logo

MediaStreamAI
/
MOTHER_EXO

Image-Text-to-Text
PyTorch
world-model
multimodal
robotics
computer-vision
sovereign
mother-exo
Model card Files
xet
 Community
MOTHER_EXO / README.md
MediaStreamAI's picture
MediaStreamAI
Upload README.md with huggingface_hub
9a1923e verified
preview code
|
Raw
History Blame Contribute Delete
13.1 kB
metadata 
license: other
library_name: pytorch
pipeline_tag: image-text-to-text
tags:
  - world-model
  - multimodal
  - robotics
  - computer-vision
  - sovereign
  - mother-exo

MOTHER EXO — Model Card

Model: MOTHER EXO — one sovereign world model (world_model_exo_final_vN.pt) Architecture: frozen sovereign backbone ⊕ trainable head-only weights

  • CORE-7B (frozen) — sovereign text/reasoning backbone (48 layers, 3072 d)
  • MOTHER DeepVision (frozen) — sovereign SigLIP-SO400M ViT (27 layers, 1152 d, 896 px)
  • cross-attn adapters + per-head projections + task heads — the trained weights Posture: non-weapon · observe-and-advise · meaningful human control (HITL) · consent/GDPR/EU-AI-Act · sovereign (only MOTHER's own weights run in the stack; external models are data-only teachers; codecs/vocoders are renderers, not models). Build: training/build_final_model.py blends the frozen base + every head into one atomic checkpoint (base fingerprint preserved). Load/serve: training/load_exo.py (MotherExo) + mother_exo_sim/models/exo_node_server.py (:8000).

Modality cheat-sheet — what each weight consumes

Input modality Weights that consume it How it arrives
Video / image frames vision, detect, track, worldmodel, face, relations a JPEG/PNG frame (data-URL/base64 or fetchable URL)
Audio waveform audio (ASR), speech (as TTS output) a WAV (base64)
Text + world context (data points) reasoning, memory a query string + buildContext() (detections, relations, analytics, data_points)

So: HLS video → decode to frames → the six vision weights. Audio feeds → decode to WAV → the audio weight. All other data points → JSON summaries → the reasoning/memory context.

The weights

1. reasoning (adapters) · kind: language

  • Does: vision-grounded language reasoning, threat/HITL advice, tool-call planning, orchestration. CORE-7B ⊕ cross-attn adapters + projection.
  • Train: training/adapters/train_integrated.py (streams the reasoning manifest: mother_core_v31 + golden + mother_reason.jsonl + mother_cot.jsonl …). Format: Question:\n\n…\n\nAnswer:.
  • Checkpoint: ~/.mother-exo/heads/integrated_adapters.pt
  • Inference: exo.reason(prompt, image=None, use_memory=False) · node POST /reason {query,image?} · SIM /api/v1/exo/reason
  • Wire to VIDEO (HLS): pass a captured frame as image so reasoning is vision-grounded (reason(q, image=frame)).
  • Wire to AUDIO: transcribe first (audio weight) → feed the transcript as the prompt (reason(transcribe(wav))).
  • Wire to DATA POINTS: every feed ingested via ingestDataPoints() lands in buildContext().data_points; reason()/exoReason() pass that context to the model. Already wired by FeedAutoRegistrar.

2. detect · kind: vision

  • Does: 1233-class objects + per-class boxes + RED/GREEN/CAUTION safety tag (safety_lexicon).
  • Train: integrated_heads.py --head detect (COCO/LVIS/aircraft).
  • Checkpoint: integrated_detect.pt
  • Inference: exo.detect(img, topk) → [(label,score,box,safety)] · node POST /detect {image} · SIM /api/v1/exo/detect
  • Wire to VIDEO (HLS): an <video>+HLS.js tile registers a frame grabber → the world-model scheduler posts frames to /api/exo/detect (browser path), or server-side snapshot via {imageUrl}. See "Wiring HLS video" below.
  • Audio / data points: N/A (vision); its detections feed the reasoning/memory context.

3. track · kind: vision

  • Does: appearance embeddings → persistent, unlimited multi-object IDs across frames (re-ID, occlusion re-acquire) via tracker.py + entity registry OBJ- UINs.
  • Train: integrated_heads.py --head track (InfoNCE temporal pairs).
  • Checkpoint: integrated_track.pt
  • Inference: exo.track_frame(img, frame_idx) / track_video(frames) · node POST /track {image}
  • Wire to VIDEO (HLS): same frame path as detect; call /track per frame to keep IDs.

4. worldmodel · kind: vision/temporal (single-frame)

  • Does: next-latent prediction (CPC) from one frame's ViT tokens — a lightweight pre-act preview.
  • Train: integrated_heads.py --head worldmodel.
  • Checkpoint: integrated_worldmodel.pt (6.3M MLP)
  • Inference: exo.worldmodel_next(img) → latent · (no dedicated node route yet; used internally)
  • Wire to VIDEO (HLS): consumes frames like the other vision heads.

4b. worldmodel_latent · kind: video dynamics

  • Does: multi-step latent dynamics — a causal temporal transformer over the frozen MOTHER T2V 3D-VAE latents predicts future latent frames of a clip.
  • Train: training/mother_t2v/latent/train_worldmodel_latent.py (InfoNCE + L2 on real captioned clips). 19.7M params, d=512, depth=6.
  • Checkpoint: integrated_worldmodel_latent.pt
  • Eval: next-frame top-1 retrieval — SAME-CLIP (fair dynamics) 0.9983 on 400 held-out clips (the same-clip metric measures true temporal ordering; the prior single-frame head scored 0.167).
  • Inference: exo.worldmodel_latent_next(frames) → (embeds, predicted-next) — video-native (encodes the clip with the frozen T2V VAE first).

5. face · kind: vision (consent-gated)

  • Does: pseudonymous person UIN; name only for consent-enrolled people (Chris). GDPR/EU-AI-Act; persistent person re-ID off unless MOTHER_PERSON_UIN=1.
  • Train: integrated_heads.py --head face (reads the recognition store at /var/lib/mother-exo/training/recognition).
  • Checkpoint: integrated_face.pt
  • Inference: exo.identify(img) → {uin,name,score} · node POST /identify {image} · SIM /api/v1/exo/identify
  • Wire to VIDEO (HLS): frame → /identify. Enrolment is consent-only.

6. relations · kind: vision

  • Does: spatial relations (on / under / next_to) for scene-graph grounding.
  • Train: integrated_heads.py --head relations.
  • Checkpoint: integrated_relations.pt
  • Inference: exo.relation(img) → (rel,score)
  • Wire to VIDEO (HLS): frame → relation.

7. memory (MOTHERrag SPI) · kind: memory

  • Does: Semantic-Pyramid long-term memory — write/recall observations, reasoning, docs, by entity UIN; right-to-erasure.
  • Train: training/mother_rag.py --train.
  • Checkpoint: integrated_memory.pt
  • Inference: exo.remember/recall/recall_about/forget · node /api/v1/gb10-training/memory*
  • Wire to DATA POINTS: ingest documents/observations; recall feeds reasoning context. Text memory lazy-loads CORE.

8. speech · kind: speech (voice OUT)

  • Does: 20-language ID + Text-To-Voice — predicts neural-codec tokens → renderer (EnCodec, codec.py) decodes to a waveform. MOTHER speaks from her own weights.
  • Train: training/train_speech.py --train --audio <dir> (language-ID from text; acoustic from (text,audio) via build_speech_audio.py).
  • Checkpoint: integrated_speech.pt
  • Inference: exo.synthesize(text,lang) → wav · node POST /speak {text,lang} → {audio_b64} · SIM /api/v1/exo/speak
  • Wire to AUDIO (out): play audio_b64 in the browser, or pipe to a speaker. (This is output; pairs with the audio weight for two-way.)

9. vision (MOTHER DeepVision) · kind: vision foundation

  • Does: self-supervised visual foundation (SimCLR over all on-node imagery) — the shared embedding the vision heads build on. Exportable standalone (export_deepvision.py → mother_deepvision.pt).
  • Train: integrated_heads.py --head vision.
  • Checkpoint: integrated_vision.pt
  • Inference: exo.vision_embed(img) → 256-d · node POST /embed {image}
  • Wire to VIDEO (HLS): frame → embedding (recognition/search/Defence feeds).

10. audio (MOTHER's ears) · kind: audio perception (ASR, audio IN)

  • Does: radio signals + human speech → text (codec tokens → BiGRU encoder → text decoder over CORE vocab). The transcript feeds the reasoning weight. With the speech weight → two-way humanoid conversation (hear → reason → speak).
  • Train: training/train_audio.py --train --audio <dir> ((text,audio) pairs — build_speech_audio.py bootstrap, or CommonVoice/LibriSpeech + radio comms rendered to {text,audio} jsonl).
  • Checkpoint: integrated_audio.pt
  • Inference: exo.transcribe(wav) → text · exo.converse(wav,lang) → {heard,answer,audio_b64} · node POST /transcribe {audio_b64} · POST /converse {audio_b64,lang}
  • Wire to AUDIO (HLS / mic / radio): extract the audio track (HLS audio, mic capture, SDR/radio) → WAV → /transcribe or /converse.

Wiring guide

A. Wire a live HLS video feed → the vision weights

HLS playlists (.m3u8) are not single frames, so they must be decoded to frames first.

  1. Browser tile (preferred): mount an <video> with HLS.js, draw it to a <canvas>, and register a frame grabber with the world model: registerFrameProvider(id, () => canvas.toDataURL("image/jpeg")), then add the feed via registerDetectableSource(key, [{id,label,kind,lat,lon,streamType:"hls"}]). The scheduler posts grabbed frames to /api/exo/detect (+ /track//assess). This is exactly how OverWatch tiles + GB10ExoEyes already work.
  2. Server-side snapshot: for MJPEG/JPEG cameras, register with streamUrl + streamType:"jpeg"|"mjpeg"; the scheduler server-fetches frames ({imageUrl}) — no player needed. (FeedAutoRegistrar does this for all TfL JamCams.)
  3. TV broadcast: same as (1) — a broadcast HLS URL in an <video> tile. Add the URL to a tile/registrar; frames then flow to detect/track/assess/identify.

    Node-load: the scheduler caps 8 concurrent + 4 fps for URL feeds; HLS browser tiles target ~15 fps. Add feeds freely — it sweeps fairly.

B. Wire a live audio / radio feed → the audio weight

  1. Capture → WAV: mic (MediaRecorder), an HLS audio track, or an SDR/radio receiver → 16-bit PCM WAV (any sample rate; the codec resamples to 24 kHz).
  2. Base64 → endpoint: POST /api/v1/exo/transcribe {audio_b64} → {text} (ASR), or POST /api/v1/exo/converse {audio_b64,lang} → {heard,answer,audio_b64} (hear→reason→speak).
  3. Two-way humanoid: loop converse — mic in, play the returned audio_b64 out.

    Radio: feed demodulated audio (voice nets) for transcription; for non-voice RF, feed any metadata as data points (below). Quality scales with the ASR training corpus (bootstrap = espeak; production = CommonVoice/LibriSpeech + radio comms).

C. Wire all data points → the reasoning/memory weights

  1. Catalog the feed in lib/exo/feedCatalog.ts (DATA_FEEDS for OSINT/sensor JSON; AUDIO_FEEDS for audio metadata/transcripts).
  2. Ingest on an interval: FeedAutoRegistrar pulls each api and calls ingestDataPoints(id, summary) → merged into buildContext().data_points.
  3. Reason over it: reason() / exoReason() ship buildContext() to the model, so every datapoint is in scope. Persist important items via remember().

    Already wired: ACLED, FIRMS, USGS, CISA, GDELT, ADS-B, AIS, OpenSky, IODA, CelesTrak. Add more by appending to feedCatalog.ts.

Endpoint reference

Capability load_exo Node (:8000) SIM (/api/v1/exo/*) MOTHER_AI (/api/exo/*)
detect detect /detect /detect /detect
assess (scene RED/GREEN) assess /assess /assess /assess
track (unlimited IDs) track_frame /track /track /track
identify (consent) identify /identify /identify /identify
reason reason /reason /reason /reason
speak (voice out) synthesize /speak /speak /speak
transcribe (audio in) transcribe /transcribe /transcribe (add route)
converse (2-way) converse /converse /converse (add route)
embed (DeepVision) vision_embed /embed /embed —

Environment

Var Where Purpose
MOTHER_EXO_FINAL_MODEL node pin the blended vN checkpoint
MOTHER_EXO_API_TOKEN node /etc/default/mother-exo gates /api/v1/* (must equal MOTHER_AI MOTHER_ROBOTICS_TOKEN)
MOTHER_EXO_VISION_URL node SIM → node inference (http://127.0.0.1:8000)
MOTHER_ROBOTICS_URL / _TOKEN Netlify MOTHER_AI → node (funnel + token)
MOTHER_PERSON_UIN node enable persistent person re-ID (lawful basis required)
NEXT_PUBLIC_EXO_AUDIO_FEEDS Netlify extra audio-metadata feeds for the catalog

Training all weights

bash training/adapters/run_all_weights.sh --full (vision heads), plus train_integrated.py (reasoning), mother_rag.py (memory), train_speech.py (voice), train_audio.py (ears) — then build_final_model.py to blend. Validate any blend with python training/test_all_weights.py --model <vN.pt> --image <frame> and corpora with python training/check_dataset.py.