Sheikh Mohammad Rakib commited on
Commit
a9ffeba
·
1 Parent(s): 86244a6

docs: update README with live link, social media URL, and refined technical documentation

Browse files
Files changed (1) hide show
  1. README.md +64 -50
README.md CHANGED
@@ -19,13 +19,15 @@ tags:
19
 
20
  # Memory Bridge (Memory Keeper)
21
 
 
 
22
  ## 🏆 Hackathon Submission: Build Small Hackathon
23
  **Value Proposition:** Memory Bridge offers a comforting way to preserve the legacy and essence of loved ones who have passed away. By uploading fragments of their life—letters, photos, and voice recordings—users can interact with an AI persona that embodies their personality, memories, and voice.
24
 
25
  **Intended Users:** Individuals grieving the loss of a loved one or families wishing to preserve the digital memory of their ancestors for future generations.
26
 
27
  - **Demo Video URL:** [Placeholder: Add Video URL]
28
- - **Social Post URL:** [Placeholder: Add Social Post URL]
29
 
30
  ---
31
 
@@ -42,26 +44,63 @@ When we lose someone, their voice, their stories, and their unique way of speaki
42
  - **Multilingual Support:** Chat in English, Bengali, Hindi, Chinese, Japanese, Korean, and Thai.
43
  - **Persistent Memories:** Save personas and retrieve them later using a unique Persona ID.
44
 
45
- ## ⚙️ How It Works
46
- 1. **Data Upload:** Users upload different artifacts of their loved one via the Gradio interface.
47
- 2. **AI Analysis:**
48
- - Voice notes are transcribed using an ASR model.
49
- - Photos are analyzed and described using a Vision-Language Model.
50
- - Scanned documents undergo OCR to extract handwritten or typed text.
51
- 3. **Persona Building:** All extracted texts, captions, and user-provided inputs are fed into an LLM to generate a structured persona with personality traits, key memories, and a voice description.
52
- 4. **Chat & Voice:** The user chats with the persona via another LLM endpoint. The responses can be optionally converted to audio using a Text-to-Speech (TTS) model.
53
-
54
  ## 🛠️ Tech Stack
55
- - **Frontend / UI:** Gradio (Python)
56
- - **Backend Infrastructure:** Serverless architecture hosted on [Modal](https://modal.com)
57
- - **Hosting:** Hugging Face Spaces
58
- - **AI Models (~26.7B total parameters):**
59
- - **MiniCPM4.1-8B (8B)** & **Qwen 32B** (Persona Building and Chat)
60
- - **MiniCPM-V 4.6 (8B)** (Vision/Photo Description)
61
- - **Cohere Transcribe (2B)** (Audio Transcription)
62
- - **Nemotron Parse v1.2 (<1B)** (OCR for Scans)
63
- - **Tiny Aya Fire / Water (3.35B)** (Language Models)
64
- - **VoxCPM2 (~1B)** (Text-to-Speech)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
  ## 📁 Repository Structure
67
  ```
@@ -73,41 +112,16 @@ When we lose someone, their voice, their stories, and their unique way of speaki
73
  ```
74
  *(Note: The backend model inference code in `modal_app.py` runs externally on Modal endpoints.)*
75
 
76
- ## 🚀 Installation
77
-
78
- 1. Clone the repository:
79
- ```bash
80
- git clone <repository-url>
81
- cd memory-bridge
82
- ```
83
-
84
- 2. Create and activate a virtual environment (optional but recommended):
85
- ```bash
86
- python -m venv .venv
87
- # On Windows:
88
- .venv\Scripts\activate
89
- # On macOS/Linux:
90
- source .venv/bin/activate
91
- ```
92
 
93
- 3. Install the required dependencies:
94
- ```bash
95
- pip install -r requirements.txt
96
- pip install gradio spaces
97
- ```
98
 
99
  ## 🔧 Configuration
100
  The application connects to external Modal endpoints. The URLs are hardcoded in `app.py`. No additional local configuration or API keys are required to run the frontend if the Modal endpoints are active and public.
101
 
102
  ## 💻 Usage Instructions
103
- 1. Start the Gradio application:
104
- ```bash
105
- python app.py
106
- ```
107
- 2. Open your browser and navigate to the local URL provided by Gradio (usually `http://127.0.0.1:7860`).
108
- 3. **Preserve a Memory (Tab 1):** Enter the person's name, your relationship, and upload any available texts, photos, or voice notes. Click "Preserve Their Memory" and wait for the Persona ID to be generated.
109
- 4. **Talk to Them (Tab 2):** Paste the generated Persona ID, select a language, and enable "Voice Response" if desired. Start chatting!
110
- 5. **Saved Memories (Tab 3):** View a list of previously created personas and their IDs.
111
 
112
  ## ☁️ Deployment Instructions
113
  This application consists of two parts: a backend hosted on Modal and a frontend hosted on Hugging Face Spaces.
@@ -138,5 +152,5 @@ This application consists of two parts: a backend hosted on Modal and a frontend
138
 
139
  ## 🙏 Credits
140
  - Backend Modal endpoints developed by Sheikh Md Rakib.
141
- - Powered by open-source models from Hugging Face, Cohere, and others.
142
  - Hosted on Hugging Face Spaces and Modal.
 
19
 
20
  # Memory Bridge (Memory Keeper)
21
 
22
+ 🚀 **Try it live here:** [https://huggingface.co/spaces/build-small-hackathon/memory-bridge](https://huggingface.co/spaces/build-small-hackathon/memory-bridge)
23
+
24
  ## 🏆 Hackathon Submission: Build Small Hackathon
25
  **Value Proposition:** Memory Bridge offers a comforting way to preserve the legacy and essence of loved ones who have passed away. By uploading fragments of their life—letters, photos, and voice recordings—users can interact with an AI persona that embodies their personality, memories, and voice.
26
 
27
  **Intended Users:** Individuals grieving the loss of a loved one or families wishing to preserve the digital memory of their ancestors for future generations.
28
 
29
  - **Demo Video URL:** [Placeholder: Add Video URL]
30
+ - **Social Post URL:** [https://x.com/sheikhMdRakib23/status/2066510302041235609](https://x.com/sheikhMdRakib23/status/2066510302041235609)
31
 
32
  ---
33
 
 
44
  - **Multilingual Support:** Chat in English, Bengali, Hindi, Chinese, Japanese, Korean, and Thai.
45
  - **Persistent Memories:** Save personas and retrieve them later using a unique Persona ID.
46
 
 
 
 
 
 
 
 
 
 
47
  ## 🛠️ Tech Stack
48
+
49
+ * **Frontend / UI:** Gradio
50
+ * **Backend Infrastructure:** Modal
51
+ * **Hosting:** Hugging Face Spaces
52
+ * **Storage:** Modal Volumes
53
+ * **Inference:** Open-source AI models running on Modal and MiniCPM endpoints
54
+
55
+ ## 🤖 Models Used
56
+
57
+ All models used in Memory Bridge are individually below the 32B parameter limit required by the Build Small Hackathon.
58
+
59
+ | Model | Purpose |
60
+ | ------------------------------------ | ---------------------------------------------------------------- |
61
+ | **MiniCPM4.1-8B** | Persona generation and conversational AI |
62
+ | **MiniCPM-V 4.6 (8B)** | Photo understanding and visual memory extraction |
63
+ | **Cohere Transcribe 03-2026 (~2B)** | Speech-to-text transcription of voice recordings |
64
+ | **NVIDIA Nemotron Parse v1.2 (<1B)** | OCR and document understanding for scanned letters and documents |
65
+ | **VoxCPM2 (~1B)** | Text-to-speech voice synthesis |
66
+ | **Tiny Aya Fire (3.35B)** | South Asian multilingual language support |
67
+ | **Tiny Aya Water (3.35B)** | Asia-Pacific multilingual language support |
68
+
69
+ ### AI Pipeline
70
+
71
+ 1. **Voice Notes → Cohere Transcribe**
72
+
73
+ * Converts uploaded audio into text.
74
+
75
+ 2. **Photos → MiniCPM-V 4.6**
76
+
77
+ * Generates detailed descriptions of people, scenes, and emotional context.
78
+
79
+ 3. **Scanned Documents → Nemotron Parse**
80
+
81
+ * Extracts text from handwritten or printed documents.
82
+
83
+ 4. **Persona Creation → MiniCPM4.1-8B**
84
+
85
+ * Builds a structured memory profile including:
86
+
87
+ * Personality traits
88
+ * Speech style
89
+ * Key memories
90
+ * Values
91
+ * Voice characteristics
92
+
93
+ 5. **Conversation → MiniCPM4.1-8B**
94
+
95
+ * Powers real-time conversations with the preserved persona.
96
+
97
+ 6. **Voice Response → VoxCPM2**
98
+
99
+ * Converts generated responses into speech.
100
+
101
+ 7. **Multilingual Support → Tiny Aya Fire & Tiny Aya Water**
102
+
103
+ * Supports Bengali, Hindi, Chinese, Japanese, Korean, Thai, and other languages.
104
 
105
  ## 📁 Repository Structure
106
  ```
 
112
  ```
113
  *(Note: The backend model inference code in `modal_app.py` runs externally on Modal endpoints.)*
114
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
 
 
 
 
 
 
116
 
117
  ## 🔧 Configuration
118
  The application connects to external Modal endpoints. The URLs are hardcoded in `app.py`. No additional local configuration or API keys are required to run the frontend if the Modal endpoints are active and public.
119
 
120
  ## 💻 Usage Instructions
121
+ First, visit the live application at [Memory Bridge](https://huggingface.co/spaces/build-small-hackathon/memory-bridge).
122
+ 1. **Preserve a Memory (Tab 1):** Enter the person's name, your relationship, and upload any available texts, photos, or voice notes. Click "Preserve Their Memory" and wait for the Persona ID to be generated.
123
+ 2. **Talk to Them (Tab 2):** Paste the generated Persona ID, select a language, and enable "Voice Response" if desired. Start chatting!
124
+ 3. **Saved Memories (Tab 3):** View a list of previously created personas and their IDs.
 
 
 
 
125
 
126
  ## ☁️ Deployment Instructions
127
  This application consists of two parts: a backend hosted on Modal and a frontend hosted on Hugging Face Spaces.
 
152
 
153
  ## 🙏 Credits
154
  - Backend Modal endpoints developed by Sheikh Md Rakib.
155
+ - Powered by open-source models from Hugging Face, Cohere, OpenBMB, and NVIDIA.
156
  - Hosted on Hugging Face Spaces and Modal.