You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

⚡ Each donation funds the next large quant.

I host free GGUF or MoE quants as independent research.
Local hardware: Mechrevo Kuangshi GM7AG0M — RTX 3060 Laptop 6GB GDDR6, 64GB DDR5, i7-12700H (14C/20T, 4.7GHz), Windows 11, Samsung 990 Pro.
Good for imatrix and 0.6–35B-class work in RAM. 9B+ and searches need rented H200/Blackwell, typically $100 per quant.

🎉 Boosty🦄 | ☕ Buy Me a Coffee🦄 | ⭐ DonationAlerts🦄

💚 Thanks to Hugging Face for extra storage.🦄

NOESIS / AMAImedia

Released as part of the NOESIS Professional Multilingual Dubbing Automation Platform (framework: DHCF-FNO — Deterministic Hybrid Control Framework for Frozen Neural Operators).

Founder: Ilia Bolotnikov
Organization: AMAImedia.com
X (Twitter): @AMAImediacom
LinkedIn: Ilia Bolotnikov
Telegram: @AMAImediacom
Release date: 2026-09-27

NOESIS-Omni-RealTime

A portable omni model: one language-model core that takes video frames, images, audio, speech and documents, and answers in text. Eight components, one self-contained folder, plain safetensors.

This repository is private.

Everything runs in 4-bit (NF4) on a single 6 GB GPU. Measured, not estimated: peak 4.47 GiB, load in 20.8 s, generation ~29 tokens/s.

At a glance


parameters	9.396 B across 8 components
tensors	4054
weights	19.00 GB (17.70 GiB) in `components/` + 0.75 GiB speech vocoder
repository	167 files, 18.53 GiB, private
precision	bf16/fp32 on disk, quantised to NF4 4-bit at load time
peak VRAM	4.47 GiB (core: LLM + vision + audio + bridges)
text speed	~29 tokens/s
hardware	one 6 GB card, 32 GB RAM, 30 GB disk, Python 3.12
code	Apache 2.0

Nothing here is encrypted and nothing needs a password: the components are ordinary safetensors, one file per component, with that component's own config next to it.

What is inside

component	file	size	params
`llm`	`components/llm.safetensors`	9.02 GiB	4.841 B
`audio`	`components/audio.safetensors`	2.88 GiB	1.543 B
`aligner`	`components/aligner.safetensors`	1.71 GiB	0.918 B
`ocr`	`components/ocr.safetensors`	1.59 GiB	0.853 B
`tts`	`components/tts.safetensors`	1.14 GiB	0.613 B
`vision`	`components/vision.safetensors`	0.97 GiB	0.522 B
`diarization`	`components/diarization.safetensors`	0.37 GiB	0.099 B
`bridges`	`components/bridges.safetensors`	0.02 GiB	0.006 B
vocoder	`4bit/tools/tts_audio_tokenizer/`	0.75 GiB	—

Tensor counts and exact byte sizes are in components.json.

Original names

llm — agentscope-ai/QwenPaw-Flash-4B — AgentScope
vision — AMAImedia/NOESIS-MiniCPM-V-4.6-SigLIP2-BF16 — project
audio — AMAImedia/NOESIS-Whisper3-1.6B-Large-Turbo-Darwin-99LANG-BF16 — project
aligner — AMAImedia/NOESIS-Qwen3-Forced-Aligner-0.6B-112LANG-BF16 — project
tts — AMAImedia/NOESIS-OmniVoice-118-646-LANG-BF16 — project
ocr — ATH-MaaS/OvisOCR2 — ATH-MaaS
diarization — nvidia/Nemotron-3-Diarization — NVIDIA
bridges — trained in this project
vocoder — bosonai/higgs-audio-v2-tokenizer — bosonai

Five of the eight components are the project's own trained models from the AMAImedia Hugging Face organisation. The exceptions are the language model (QwenPaw-Flash-4B by AgentScope), OCR (OvisOCR2 by ATH-MaaS) and diarization (Nemotron-3 by NVIDIA).

The eight components

1. `llm` — the language model

Architecture: Qwen3_5ForConditionalGeneration, qwen3_5
Hidden size 2560, 32 layers, vocabulary 248320, 262144 max positions
Hybrid attention: 24 linear-attention layers + 8 full-attention layers (full_attention_interval = 4)
9.02 GiB, 427 tensors

An agentic model — it can call tools, which is exactly how OCR, TTS, alignment and diarization are wired in. It was chosen over the automotive ("Drive") variant because NOESIS Omni is a general-purpose model.

Its own visual tower is excluded: the SigLIP2 tower in vision is used instead.

2. `vision` — image and video tower

Architecture: SiglipVisionModel (siglip_vision_model)
Hidden size 1152, 27 layers, output 1152
0.97 GiB, 453 tensors

A SigLIP2 tower extracted from MiniCPM-V-4.6 and rebuilt as a standalone vision model. Upstream basis: openbmb/MiniCPM-V-4.6 and Google SigLIP2 design.

This is the only component whose bridge had to be trained: the language model has no vision of its own, and the tower output (1152) does not match the LLM input (2560).

3. `audio` — audio encoder and ASR decoder

Architecture: WhisperForConditionalGeneration (whisper), bfloat16
99 languages, including Vietnamese; hidden size 1280, 32 encoder layers, 128 mel bins
2.88 GiB, 1259 tensors

The project's own work on top of openai/whisper-large-v3 with the turbo branch added. It covers more languages than stock Whisper.

4. `aligner` — forced word alignment

Architecture: Qwen3ASRForConditionalGeneration (qwen3_asr)
112 languages (Vietnamese at TIER B); basis Qwen/Qwen3-ForcedAligner-0.6B
1.71 GiB, 708 tensors

Takes audio plus text and returns per-word timestamps — for subtitles and dubbing synchronisation.

Loading notes. In the released layout the weights are named thinker.* and the timestamp classifier is thinker.lm_head with 5000 classes (classify_num). The model is therefore loaded as a token-classification variant with the mapping thinker.audio_tower → model.audio_tower, thinker.model → model.language_model, thinker.lm_head → score. Three specifics:

Qwen3ASRConfig in transformers 5.17 does not register qwen3_asr_audio_encoder in CONFIG_MAPPING, so sub-configs are built explicitly;
the release does not contain multi_modal_projector weights (and output_dim == d_model == 1024), so it is replaced with Identity — with random weights the timestamps ran past the end of the audio (7.84 s for 6 s of audio), with identity they are monotonic;
the aligner chat template discards the transcript, so the token layout (audio placeholders + word + two <timestamp>) is assembled by hand.

Runs in 4-bit: 3.51 GiB, word timestamps at 80 ms granularity.

5. `ocr` — text from documents

Original name: ATH-MaaS/OvisOCR2
Base model: Qwen/Qwen3.5-0.8B
Architecture: Qwen3_5ForConditionalGeneration, 0.853 B params
bfloat16, hidden 1024, 24 layers
1.59 GiB, 473 tensors
Measured in 4-bit: 1430 input tokens, 12.26 s, 1.02 GiB

OvisOCR2 is a compact end-to-end model for page-level document parsing. Given a document page image it generates a Markdown representation in natural reading order, covering text, formulas, tables and visual regions. It is Qwen3.5-0.8B post-trained with a data engine combining real-world and synthetic data, using a multi-stage recipe of SFT, RL and OPD.

Benchmarks from the model card:

benchmark	score
OmniDocBench v1.6, overall	96.58
PureDocBench, Avg3	75.06

On OmniDocBench v1.6 it is the first end-to-end model to top a leaderboard previously dominated by pipeline methods.

The qwen3_5 entry in components/ocr.config.json is the base-family architecture, not a plain Qwen3-VL: the released weights are the OvisOCR2 document parser.

Reads text from scans, screenshots, forms, receipts and video frames. It outputs text, so it needs no bridge — it is wired in as a tool.

6. `tts` — speech synthesis

Original name: AMAImedia/NOESIS-OmniVoice-118-646-LANG-BF16
Architecture: OmniVoice (omnivoice), bfloat16; base Qwen3-0.6B-Omni-Voice
118 languages on a 646-language base — the name encodes exactly that range: 112 languages in the merged model plus per-language fine-tunes (VN, TWI, BE, TH, HOKKIEN, BRPT, SINGING)
1.14 GiB, 313 tensors

Clones a voice from a short reference clip. The adapters nt142_duration, nt168_emotion and nt169_per_lang are already merged into the weights — nothing extra to download.

components/tts.safetensors is only the audio-token language model: 313 tensors, of which 310 are llm.* plus audio_embeddings, audio_heads and codebook_layer_offsets — no decoder, no vocoder. Turning its 8-codebook output into a waveform requires a speech vocoder, which is included in this repository at 4bit/tools/tts_audio_tokenizer/ (768 MiB, architecture HiggsAudioV2TokenizerModel, 9 codebooks, 24 kHz, hop_length 960).

Measured: 4.24 s of speech at 24 kHz with a cloned voice, 3.0 GiB peak.

7. `diarization` — who is speaking

Architecture: Nemotron3DiarizationForAudioFrameClassification, 99.2 M params
0.37 GiB, 417 tensors

Splits audio by speaker (up to 8 voices, with overlap). The architecture only appeared in transformers >= 5.18, so transformers 5.19 is kept alongside in venv/extras_tf519 and invoked as a subprocess — two transformers versions are never mixed in one process.

8. `bridges` — vision and audio projections

What: Linear(1152 → 2560) for vision, Linear(1280 → 2560) for audio
6.24 M params, 4 tensors, 0.02 GiB
Trained in this project

Deliberately not a Perceiver resampler: its trainable queries are bound to the encoder they were trained with and become meaningless when the encoder is swapped.

Only the vision bridge was trained. The audio bridge is still at random initialisation, and the numbers show it: audio_projection_layer has a maximum weight of 0.0277 against the random-init boundary 1/√1280 = 0.0279, and scale is 0.01985 in all 2560 cells, while the trained vision bridge goes beyond that boundary (0.0475) and has a varying scale (std 0.005). The cause is in the training script: it accepted only records with an image field (if rec.get("caption") and rec.get("image")), so audio pairs never entered training.

Practical consequence: the core does not hear audio directly. Audio questions are answered by a chain of already-trained modules — Whisper3 (words) → forced aligner (word timings) → Nemotron-3 (who spoke when) → LLM. Intonation and emotion are not carried through this chain; no module in it emits them.

Two decisions determined the vision result:

scale — a random linear layer produces values an order of magnitude larger than LLM embeddings (std 1.30 vs 0.0204), and image positions become outliers. The scale is initialised to the embedding std;
centring — SigLIP features are dominated by a component common to the whole image, with the difference between images a small remainder. Averaging 1036 rows into 64 kept the common part and washed out the remainder: two different images gave cosine 0.9932 and the model ignored the channel. Centring along the token axis and selecting positions instead of averaging gives 0.0655.

How it is wired

Core — a single forward pass: language model + both towers + bridges.

frames    -> SigLIP2   -> Linear(1152 -> 2560) -+
text      ------------------------------------- +-> LLM -> text
document  -> OvisOCR2  -> text ----------------+
audio     -> Whisper3  -> Linear(1280 -> 2560) -+    (bridge untrained)

audio question -> Whisper3 (words)
               -> forced aligner (word timings)
               -> Nemotron-3 (who spoke, when) -> LLM -> text
                        text -> OmniVoice -> HiggsAudioV2 -> speech

Tools do not share the core's process. They are wired directly in 4bit/omni_tools.py and loaded one at a time, then freed, so the whole set fits into 6 GB. The aligner needs explicit sub-config assembly; diarization needs transformers 5.19 in venv/extras_tf519 via a subprocess. 4bit/omni_tools.py wraps all five (OCR, aligner, diarization, TTS, ASR).

Measured results

All numbers below are real runs on a single 6 GB GPU. The raw JSON behind every figure is in 4bit/out/ (11 files).

Core, 4-bit NF4

task	input	time	peak VRAM
load model	—	20.8 s	4.47 GiB
text generation	—	~29 tok/s	4.47 GiB
image question	94 tokens	1.58 s	4.47 GiB
video, 8 frames	629 tokens	7.03 s	4.47 GiB
video, 5 s clip	322 tokens	8.05 s	4.47 GiB

A picture becomes 64 vision tokens.

Tools

tool	load	run	peak VRAM	result
OCR	7.3 s	12.26 s	1.02 GiB	1430 input tokens
Aligner	11.5 s	0.09 s	3.51 GiB	15 words, 80 ms steps
Diarization	—	22.4 s / 33.6 s	0.38 GiB	2 segments on 6 s audio
TTS	26.3 s	4.24 s of speech	3.00 GiB	24 kHz, cloned voice
ASR (audio core)	—	1.01 s	1.10 GiB	language detected: russian

Audio question-and-answer chain

The full chain 4bit/demo_audio_qa.py — ASR → aligner → diarization → LLM — takes ~104 s end to end, of which the LLM answer is 5.8 s at 4.48 GiB.

Keep --max-new-token at 200: at higher values the model starts inventing the continuation of the dialogue.

Figures

4bit/figs/ holds 12 files (PNG + SVG), all generated from the JSON above: case-4bit-demo, tools-demo, 4bit-metrics, tts-demo, duplex, understanding. Every label is English, every number comes from a run. Three more diagrams sit at the root: omni-architecture, omni-capabilities, omni-vs-rtvenus (PNG + SVG each).

What does not work

Stated plainly, because it matters:

the core does not hear audio — the audio bridge was never trained (see bridges above); audio reaches the LLM only as text from the module chain;
intonation and emotion are never passed on — no module in the chain emits them;
no real-time duplex — the build is an offline pipeline;
no external benchmarks (StreamingBench, MMAU) have been run — none of the numbers above are benchmark scores;
no training of the language model — only the two bridges were ever trained, and only the vision one converged.

Running it

Environment

bash setup.sh                       # builds .venv from requirements.lock.txt
./.venv/bin/python run.py --list    # what the build contains
./.venv/bin/python run.py --verify  # every component readable: 8 of 8 ok

setup.sh builds a fresh environment rather than trusting a copied venv/. A venv is not portable: its bin/python is a symlink to the interpreter that created it and every console script carries that absolute path, so on another machine it falls back to the system python and cannot import torch even though the packages are all there.

Requirements: Python 3.12, NVIDIA driver >= 550, 32 GB RAM (64 preferred), 30 GB free disk. requirements.lock.txt pins 110 packages including torch-2.6.0+cu124 and transformers-5.17.0.

4-bit core

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
python 4bit/make_tool_dirs.py     # builds 4bit/tools/* from components/
python 4bit/load_4bit.py --text-test
python 4bit/demo_media.py --video clip.mp4 --frames 8 --prompt "Describe the video."
python 4bit/demo_media.py --image photo.jpg --prompt "What is in the picture?"

Tools, one at a time

python 4bit/demo_tools.py --only ocr,aligner,diarization,tts

Audio question and answer, full chain

python 4bit/demo_audio_qa.py --audio speech.wav \
    --question "What was said?" --max-new-token 200

Regenerate the figures

python 4bit/make_figs.py          # reads 4bit/out/*.json, writes 4bit/figs/

4bit/figs/ is recreated on the spot by the script. scripts/make_svg_figs.py does the same for the three root-level diagrams.

Full pipeline through `run.py`

./.venv/bin/python run.py --video clip.mp4 --question "Describe what you see."

examples/run_example.sh is the same three commands in one script.

Verifying that vision actually works

Loss is not a valid test here. An earlier gate scored the loss against a stored caption and reported only 0.05 nat of movement while the model was in fact describing each image correctly — the model writes its own description, so it never matches the stored one.

scripts/gate_vision.py compares generations instead: the same question is asked twice with two different pictures, and the answers are compared. Text cannot be right for both, so the comparison cannot be gamed.

./.venv/bin/python scripts/gate_vision.py --samples 8

Pass means at least 80% of samples produce different answers.

Bridges and training

BRIDGE_TRAINING.md covers the state of the bridges, the evidence, what needs improving, and how to run training.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
python scripts/train_projections.py \
    --pairs work/audio_pairs_v1.jsonl \
    --epochs 5 --lr 2e-5 \
    --out work/projections_audio

Pairs are {"image", "caption"} for vision and {"audio", "transcript"} for audio. --4bit is the default and takes weights from components/ through 4bit/load_4bit.py, so the backbone takes ~4.5 GiB and only the bridges train. --shards restores the original fp32/bf16 shard path, which needs ~32 GB.

Repository layout

Layout of this repository; the GitHub mirror differs, see Two mirrors below.

README.md                 this file
RUN.md                    how to run, requirements, troubleshooting
MODELS.md                 every component: source, licence, size
README-plan.md            project plan and actual state
BRIDGE_TRAINING.md        bridges: state, evidence, what to improve
components.json           manifest: file, tensor count, bytes per component
config.json               core config (hybrid attention, mrope, vision tower)
run.py                    entry point: --list, --verify, --video
setup.sh                  builds .venv from requirements.lock.txt
requirements.lock.txt     110 exact package versions
LICENSE                   licence text
run-from-external.sh      runs from an NTFS/exFAT external disk
examples/run_example.sh   three commands: list, verify, describe a video

omni-architecture.*       architecture diagram, PNG + SVG
omni-capabilities.*       capabilities diagram, PNG + SVG
omni-vs-rtvenus.*         comparison diagram, PNG + SVG
paper-case_gnsL.png       input image for the OCR / image demo
paper-duplex_gnsL.svg     duplex-style figure
paper-understanding_gnsL.svg  understanding-style figure

.github/ACCESS.md         who has access to the GitHub mirror

components/               44 files — 8 × safetensors plus each component's own
                          config, tokenizer, processor and chat template files

code/drive_omni/          model code: configuration, modeling, processing,
                          tokenizer (json + vocab + chat template), navit_siglip

4bit/                     everything runs in 4-bit on one 6 GB card
  load_4bit.py            4-bit NF4 loader (streaming quantisation, rope)
  omni_tools.py           OCR / aligner / diarization / TTS / ASR wrappers
  demo_media.py           image and video runs
  demo_tools.py           per-tool runs
  demo_audio_qa.py        audio question-and-answer chain
  inspect_model.py        structure inspection
  make_tool_dirs.py       builds 4bit/tools/* from components/
  make_figs.py            figures from real measurements
  make_paper_style.py     paper-style figures
  _diar_worker.py         diarization subprocess (transformers 5.19)
  README.md               4-bit notes, constraints, per-module measurements
  assets/                 test audio and the piper voice used to record it
  out/                    11 JSON files: the raw numbers behind every figure
  figs/                   12 figures from real runs, PNG + SVG
  tools/tts_audio_tokenizer/   speech vocoder (768 MiB) for `tts`
  third_party/omnivoice/  vendored vocoder model code

scripts/                  10 files: bridge training, the bridges themselves,
                          shard averaging, vision checks, SVG figure generator

Documentation

document	contents
`RUN.md`	how to run, requirements, troubleshooting table
`MODELS.md`	every component: source, licence, size, how they connect
`README-plan.md`	project plan and actual state
`4bit/README.md`	4-bit notes, constraints, per-module measurements
`BRIDGE_TRAINING.md`	bridges: state, evidence, what to improve

The GitHub mirror carries the same five documents plus .github/ACCESS.md, which records who has access to it.

Weights

This repository holds the weights as well as the code. The GitHub mirror holds the code only — 19 GB of safetensors does not fit there, and its .gitignore blocks them. Everything else is identical in both places.

Hardware note

requirements.lock.txt pins torch-2.6.0+cu124 on purpose: the build machine has Tesla V100 cards (compute capability 7.0, sm_70), and Volta support was removed in torch 2.11 and in every CUDA 13.x build. On a machine without those cards any recent torch works, but the pin keeps one environment valid for both.

License

Preview is released under the Apache License 2.0. See LICENSE for details.

Downloads last month: 2

Inference Providers NEW

Any-to-Any

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including AMAImedia/NOESIS-Omni-RealTime

NOESIS-Original-Trained-Models

Collection

9 items • Updated about 5 hours ago