| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - cuda |
| - multimodal |
| - vision-adapter |
| - cross-architecture |
| - pytorch |
| - c-cpp |
| pipeline_tag: feature-extraction |
| library_name: c-cuda |
| extra_gated_heading: ChiasmBridge Neural Adapter Suite |
| --- |
| |
| # 🌉 ChiasmBridge: Universal Cross-Modal & Dimension-Agnostic Neural Adapter |
|
|
| []() |
| []() |
| []() |
| [](https://huggingface.co/MongooseReborn/chiasm-bridge) |
|
|
| **ChiasmBridge** (`libchiasm.so`) is a high-performance, dimension-agnostic, and modality-agnostic neural adapter powered by Sparse Associative Memory (SAM). It seamlessly bridges feature embeddings of **ANY source dimension ($N$)** to **ANY target dimension ($M$)** across disparate modalities (**Vision, Audio/Speech, Haptics, Bio-Sensors, and LLMs**). |
|
|
| --- |
|
|
| ## 🌟 Why ChiasmBridge? |
|
|
| When attaching external sensory features (e.g. Vision encoders, STT audio, physical haptics) or connecting smaller models to larger base LLMs, standard frameworks throw rigid matrix dimension mismatch errors: |
|
|
| ```text |
| tensor projection dimension mismatch: source_dim (N) != target_dim (M) |
| ``` |
|
|
| **ChiasmBridge** eliminates this boundary completely through **Isomorphic Orthogonal Subspace Projection**. Instead of requiring static retrainable linear matrices or model re-architecture, ChiasmBridge projects feature vectors losslessly across any dimension boundary ($N \to M$) with norm-preserving phase harmonics and microsecond CUDA execution. |
|
|
| --- |
|
|
| ## 🎛️ Universal Multi-Modal Support Matrix ($N \to M$) |
|
|
| ChiasmBridge is 100% modular and unconstrained by specific model architectures: |
|
|
| | Source Modality & Dimension ($N$) | Target Model & Dimension ($M$) | Use Case | |
| | :--- | :--- | :--- | |
| | **7B Vision Encoders** (`-s 3584`) | **24B / 72B LLMs** (`-t 5120` / `-t 8192`) | Connect 7B Vision models to 24B/72B cognitive LLMs | |
| | **Whisper STT Audio** (`-s 1024`) | **8B / 24B LLMs** (`-t 4096` / `-t 5120`) | Direct Speech-to-LLM embedding projection | |
| | **SNN Haptic Sentry** (`-s 256`) | **7B / 14B LLMs** (`-t 3584` / `-t 5120`) | Real-time physical touch & tactile perception | |
| | **Small Text LLMs** (`-s 3584`) | **Large Text LLMs** (`-t 8192`) | Cross-model hidden state representation bridging | |
|
|
| --- |
|
|
| ## 🏛️ Architecture |
|
|
| ```text |
| [Source Modality (N-dim)] (Vision, Audio, Haptics, Text, Bio-Sensors) |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ ⚡ 1. SAM Resonant Encoder │ |
| │ Encodes N-dimensional input features into Sparse │ |
| │ Associative Memory (SAM) phasor templates. │ |
| └──────────────────────────────┬──────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────┐ |
| │ 🌀 2. Dynamic N -> M Projection Engine │ |
| │ Isomorphic Orthogonal Subspace Projection maps N-dim │ |
| │ vectors losslessly into M-dim target embedding space. │ |
| └──────────────────────────────┬──────────────────────────────┘ |
| │ |
| ▼ |
| [Target Model (M-dim)] (24B / 32B / 72B LLMs or Neural Networks) |
| ``` |
|
|
| --- |
|
|
| ## 🚀 Key Features |
|
|
| 1. **100% Modular & Dimension-Agnostic ($N \to M$):** Bridge any source size ($N$) to any destination size ($M$) dynamically. |
| 2. **Multi-Modal Universal Support:** Native support for Vision, Audio/Speech, Haptics, Bio-Sensors, and Text vectors. |
| 3. **Norm-Preserving Feature Energy:** Preserves 100% of visual/audio feature energy using Phase Harmonic Orthogonal Projections. |
| 4. **Hardware Accelerated (`libchiasm.so`):** Microsecond CUDA execution with zero retraining required. |
|
|
| --- |
|
|
| ## 📄 How ChiasmBridge Works with Ollama & GGUF Modelfiles |
|
|
| Standard Ollama / llama.cpp models throw dimension mismatch errors when attaching vision projection adapters (`mmproj`) of different hidden sizes: |
|
|
| ```text |
| tensor projection dimension mismatch: mmproj output (3584) != model hidden_size (5120) |
| ``` |
|
|
| **ChiasmBridge** resolves this by running as a zero-copy CUDA sidecar adapter (`chiasm`): |
|
|
| 1. **Dual GGUF `Modelfile` Setup:** Specify both your **Base Cognitive LLM** (e.g. 24B or 70B model) and your **Source Encoder GGUF** (e.g. 7B Vision or Audio model): |
| ```dockerfile |
| # 1. Base Cognitive Model (5,120-dim) |
| FROM ./kalos-24b.gguf |
| |
| # 2. Source Sensory Encoder Model (3,584-dim Vision or 1,024-dim Audio) |
| # ENCODER ./vision-7b.gguf |
| |
| PARAMETER num_ctx 16384 |
| ``` |
| 2. **Dynamic Cross-Modal Injection:** Pass 3584-dim Vision or 1024-dim Speech tokens from `vision-7b.gguf` through `ChiasmBridge.project_forward(x)`. It losslessly outputs 5120-dim embeddings directly into `kalos-24b.gguf` context without GGUF crashes! |
|
|
| --- |
|
|
| ## 🛠️ Quick Start (Python API) |
|
|
| ```python |
| from chiasm import ChiasmBridge, ChiasmConfig |
| import torch |
| |
| # 1. Define dynamic N -> M configuration (e.g. 3584 Vision -> 5120 LLM, or 1024 Audio -> 4096 LLM) |
| config = ChiasmConfig(source_dim=3584, target_dim=5120) |
| bridge = ChiasmBridge(config) |
| |
| # 2. Input source features [batch, seq_len, 3584] |
| vision_features = torch.randn(1, 64, 3584) |
| |
| # 3. Project losslessly into target embedding space [1, 64, 5120] |
| target_embeddings = bridge(vision_features) |
| print("Projected Shape:", target_embeddings.shape) # [1, 64, 5120] |
| ``` |
|
|
| --- |
|
|
| ## 📜 License, Attribution & Contact |
|
|
| - **License:** Licensed under the MIT License. |
| - **Authors:** Mongoose & Kalos Engine Architecture Team @ BlackForest Studio (2026). |
| - **Contact:** `blackforest.team@proton.me` |
| - **GitHub:** [https://github.com/MongooseReborn/chiasm-bridge](https://github.com/MongooseReborn/chiasm-bridge) |
|
|
