Sheikh Mohammad Rakib commited on
Commit ·
a9ffeba
1
Parent(s): 86244a6
docs: update README with live link, social media URL, and refined technical documentation
Browse files
README.md
CHANGED
|
@@ -19,13 +19,15 @@ tags:
|
|
| 19 |
|
| 20 |
# Memory Bridge (Memory Keeper)
|
| 21 |
|
|
|
|
|
|
|
| 22 |
## 🏆 Hackathon Submission: Build Small Hackathon
|
| 23 |
**Value Proposition:** Memory Bridge offers a comforting way to preserve the legacy and essence of loved ones who have passed away. By uploading fragments of their life—letters, photos, and voice recordings—users can interact with an AI persona that embodies their personality, memories, and voice.
|
| 24 |
|
| 25 |
**Intended Users:** Individuals grieving the loss of a loved one or families wishing to preserve the digital memory of their ancestors for future generations.
|
| 26 |
|
| 27 |
- **Demo Video URL:** [Placeholder: Add Video URL]
|
| 28 |
-
- **Social Post URL:** [
|
| 29 |
|
| 30 |
---
|
| 31 |
|
|
@@ -42,26 +44,63 @@ When we lose someone, their voice, their stories, and their unique way of speaki
|
|
| 42 |
- **Multilingual Support:** Chat in English, Bengali, Hindi, Chinese, Japanese, Korean, and Thai.
|
| 43 |
- **Persistent Memories:** Save personas and retrieve them later using a unique Persona ID.
|
| 44 |
|
| 45 |
-
## ⚙️ How It Works
|
| 46 |
-
1. **Data Upload:** Users upload different artifacts of their loved one via the Gradio interface.
|
| 47 |
-
2. **AI Analysis:**
|
| 48 |
-
- Voice notes are transcribed using an ASR model.
|
| 49 |
-
- Photos are analyzed and described using a Vision-Language Model.
|
| 50 |
-
- Scanned documents undergo OCR to extract handwritten or typed text.
|
| 51 |
-
3. **Persona Building:** All extracted texts, captions, and user-provided inputs are fed into an LLM to generate a structured persona with personality traits, key memories, and a voice description.
|
| 52 |
-
4. **Chat & Voice:** The user chats with the persona via another LLM endpoint. The responses can be optionally converted to audio using a Text-to-Speech (TTS) model.
|
| 53 |
-
|
| 54 |
## 🛠️ Tech Stack
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
## 📁 Repository Structure
|
| 67 |
```
|
|
@@ -73,41 +112,16 @@ When we lose someone, their voice, their stories, and their unique way of speaki
|
|
| 73 |
```
|
| 74 |
*(Note: The backend model inference code in `modal_app.py` runs externally on Modal endpoints.)*
|
| 75 |
|
| 76 |
-
## 🚀 Installation
|
| 77 |
-
|
| 78 |
-
1. Clone the repository:
|
| 79 |
-
```bash
|
| 80 |
-
git clone <repository-url>
|
| 81 |
-
cd memory-bridge
|
| 82 |
-
```
|
| 83 |
-
|
| 84 |
-
2. Create and activate a virtual environment (optional but recommended):
|
| 85 |
-
```bash
|
| 86 |
-
python -m venv .venv
|
| 87 |
-
# On Windows:
|
| 88 |
-
.venv\Scripts\activate
|
| 89 |
-
# On macOS/Linux:
|
| 90 |
-
source .venv/bin/activate
|
| 91 |
-
```
|
| 92 |
|
| 93 |
-
3. Install the required dependencies:
|
| 94 |
-
```bash
|
| 95 |
-
pip install -r requirements.txt
|
| 96 |
-
pip install gradio spaces
|
| 97 |
-
```
|
| 98 |
|
| 99 |
## 🔧 Configuration
|
| 100 |
The application connects to external Modal endpoints. The URLs are hardcoded in `app.py`. No additional local configuration or API keys are required to run the frontend if the Modal endpoints are active and public.
|
| 101 |
|
| 102 |
## 💻 Usage Instructions
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
2. Open your browser and navigate to the local URL provided by Gradio (usually `http://127.0.0.1:7860`).
|
| 108 |
-
3. **Preserve a Memory (Tab 1):** Enter the person's name, your relationship, and upload any available texts, photos, or voice notes. Click "Preserve Their Memory" and wait for the Persona ID to be generated.
|
| 109 |
-
4. **Talk to Them (Tab 2):** Paste the generated Persona ID, select a language, and enable "Voice Response" if desired. Start chatting!
|
| 110 |
-
5. **Saved Memories (Tab 3):** View a list of previously created personas and their IDs.
|
| 111 |
|
| 112 |
## ☁️ Deployment Instructions
|
| 113 |
This application consists of two parts: a backend hosted on Modal and a frontend hosted on Hugging Face Spaces.
|
|
@@ -138,5 +152,5 @@ This application consists of two parts: a backend hosted on Modal and a frontend
|
|
| 138 |
|
| 139 |
## 🙏 Credits
|
| 140 |
- Backend Modal endpoints developed by Sheikh Md Rakib.
|
| 141 |
-
- Powered by open-source models from Hugging Face, Cohere, and
|
| 142 |
- Hosted on Hugging Face Spaces and Modal.
|
|
|
|
| 19 |
|
| 20 |
# Memory Bridge (Memory Keeper)
|
| 21 |
|
| 22 |
+
🚀 **Try it live here:** [https://huggingface.co/spaces/build-small-hackathon/memory-bridge](https://huggingface.co/spaces/build-small-hackathon/memory-bridge)
|
| 23 |
+
|
| 24 |
## 🏆 Hackathon Submission: Build Small Hackathon
|
| 25 |
**Value Proposition:** Memory Bridge offers a comforting way to preserve the legacy and essence of loved ones who have passed away. By uploading fragments of their life—letters, photos, and voice recordings—users can interact with an AI persona that embodies their personality, memories, and voice.
|
| 26 |
|
| 27 |
**Intended Users:** Individuals grieving the loss of a loved one or families wishing to preserve the digital memory of their ancestors for future generations.
|
| 28 |
|
| 29 |
- **Demo Video URL:** [Placeholder: Add Video URL]
|
| 30 |
+
- **Social Post URL:** [https://x.com/sheikhMdRakib23/status/2066510302041235609](https://x.com/sheikhMdRakib23/status/2066510302041235609)
|
| 31 |
|
| 32 |
---
|
| 33 |
|
|
|
|
| 44 |
- **Multilingual Support:** Chat in English, Bengali, Hindi, Chinese, Japanese, Korean, and Thai.
|
| 45 |
- **Persistent Memories:** Save personas and retrieve them later using a unique Persona ID.
|
| 46 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
## 🛠️ Tech Stack
|
| 48 |
+
|
| 49 |
+
* **Frontend / UI:** Gradio
|
| 50 |
+
* **Backend Infrastructure:** Modal
|
| 51 |
+
* **Hosting:** Hugging Face Spaces
|
| 52 |
+
* **Storage:** Modal Volumes
|
| 53 |
+
* **Inference:** Open-source AI models running on Modal and MiniCPM endpoints
|
| 54 |
+
|
| 55 |
+
## 🤖 Models Used
|
| 56 |
+
|
| 57 |
+
All models used in Memory Bridge are individually below the 32B parameter limit required by the Build Small Hackathon.
|
| 58 |
+
|
| 59 |
+
| Model | Purpose |
|
| 60 |
+
| ------------------------------------ | ---------------------------------------------------------------- |
|
| 61 |
+
| **MiniCPM4.1-8B** | Persona generation and conversational AI |
|
| 62 |
+
| **MiniCPM-V 4.6 (8B)** | Photo understanding and visual memory extraction |
|
| 63 |
+
| **Cohere Transcribe 03-2026 (~2B)** | Speech-to-text transcription of voice recordings |
|
| 64 |
+
| **NVIDIA Nemotron Parse v1.2 (<1B)** | OCR and document understanding for scanned letters and documents |
|
| 65 |
+
| **VoxCPM2 (~1B)** | Text-to-speech voice synthesis |
|
| 66 |
+
| **Tiny Aya Fire (3.35B)** | South Asian multilingual language support |
|
| 67 |
+
| **Tiny Aya Water (3.35B)** | Asia-Pacific multilingual language support |
|
| 68 |
+
|
| 69 |
+
### AI Pipeline
|
| 70 |
+
|
| 71 |
+
1. **Voice Notes → Cohere Transcribe**
|
| 72 |
+
|
| 73 |
+
* Converts uploaded audio into text.
|
| 74 |
+
|
| 75 |
+
2. **Photos → MiniCPM-V 4.6**
|
| 76 |
+
|
| 77 |
+
* Generates detailed descriptions of people, scenes, and emotional context.
|
| 78 |
+
|
| 79 |
+
3. **Scanned Documents → Nemotron Parse**
|
| 80 |
+
|
| 81 |
+
* Extracts text from handwritten or printed documents.
|
| 82 |
+
|
| 83 |
+
4. **Persona Creation → MiniCPM4.1-8B**
|
| 84 |
+
|
| 85 |
+
* Builds a structured memory profile including:
|
| 86 |
+
|
| 87 |
+
* Personality traits
|
| 88 |
+
* Speech style
|
| 89 |
+
* Key memories
|
| 90 |
+
* Values
|
| 91 |
+
* Voice characteristics
|
| 92 |
+
|
| 93 |
+
5. **Conversation → MiniCPM4.1-8B**
|
| 94 |
+
|
| 95 |
+
* Powers real-time conversations with the preserved persona.
|
| 96 |
+
|
| 97 |
+
6. **Voice Response → VoxCPM2**
|
| 98 |
+
|
| 99 |
+
* Converts generated responses into speech.
|
| 100 |
+
|
| 101 |
+
7. **Multilingual Support → Tiny Aya Fire & Tiny Aya Water**
|
| 102 |
+
|
| 103 |
+
* Supports Bengali, Hindi, Chinese, Japanese, Korean, Thai, and other languages.
|
| 104 |
|
| 105 |
## 📁 Repository Structure
|
| 106 |
```
|
|
|
|
| 112 |
```
|
| 113 |
*(Note: The backend model inference code in `modal_app.py` runs externally on Modal endpoints.)*
|
| 114 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
## 🔧 Configuration
|
| 118 |
The application connects to external Modal endpoints. The URLs are hardcoded in `app.py`. No additional local configuration or API keys are required to run the frontend if the Modal endpoints are active and public.
|
| 119 |
|
| 120 |
## 💻 Usage Instructions
|
| 121 |
+
First, visit the live application at [Memory Bridge](https://huggingface.co/spaces/build-small-hackathon/memory-bridge).
|
| 122 |
+
1. **Preserve a Memory (Tab 1):** Enter the person's name, your relationship, and upload any available texts, photos, or voice notes. Click "Preserve Their Memory" and wait for the Persona ID to be generated.
|
| 123 |
+
2. **Talk to Them (Tab 2):** Paste the generated Persona ID, select a language, and enable "Voice Response" if desired. Start chatting!
|
| 124 |
+
3. **Saved Memories (Tab 3):** View a list of previously created personas and their IDs.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
|
| 126 |
## ☁️ Deployment Instructions
|
| 127 |
This application consists of two parts: a backend hosted on Modal and a frontend hosted on Hugging Face Spaces.
|
|
|
|
| 152 |
|
| 153 |
## 🙏 Credits
|
| 154 |
- Backend Modal endpoints developed by Sheikh Md Rakib.
|
| 155 |
+
- Powered by open-source models from Hugging Face, Cohere, OpenBMB, and NVIDIA.
|
| 156 |
- Hosted on Hugging Face Spaces and Modal.
|