Upload 4 files

README.md

@@ -1,25 +1,126 @@
 ---
-license: apache-2.0
 language:
-- ko
-metrics:
-- accuracy
-- recall
+  - ko
+  - en
 tags:
-- multimodal
-- audio
-- video
-- homecam
-- transformer
-datasets:
-- SilverAvocado/SilverDataset
+  - transformer
+  - video
+  - audio
+  - homecam 
+  - multimodal
+  - senior
+  - yolo
+  - mediapipe
 ---
-## Load Model For Inference
 
+# Model Card for `Silver-Multimodal`
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+## Model Details
+
+- The Silver-Multimodal model integrates both audio and video modalities for real-time situation classification. 
+- This architecture allows it to process diverse inputs simultaneously and identify scenarios like daily activities, violence, and fall events with high precision. 
+- The model leverages a Transformer-based architecture to combine features extracted from audio (MFCC) and video (MediaPipe keypoints), enabling robust multimodal learning.
+
+- Key Highlights:
+	- Multimodal Integration: Combines YOLO, MediaPipe, and MFCC features for comprehensive situation understanding.
+	- Middle Fusion: The extracted features are fused and passed through the Transformer model for context-aware classification.
+	- Output Classes:
+	    - 0 Daily Activities: Normal indoor movements like walking or sitting.
+	    - 1 Violence: Aggressive behaviors or physical conflicts.
+	    - 2 Fall Down: Sudden fall or collapse.
+
+![Multimodal Model](./pics/multimodal-overview.png)
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Activity with:** NIPA-Google(2024.10.23-20224.11.08), Kosa Hackathon(2024.12.9)
+- **Model type:** Multimodal Transformer Model
+- **API used:** Keras
+- **Dataset:** [HuggingFace Silver-Multimodal-Dataset](https://huggingface.co/datasets/SilverAvocado/Silver-Multimodal-Dataset)
+- **Code:** [GitHub Silver Model Code](https://github.com/silverAvocado/silver-model-code)
+- **Language(s) (NLP):** Korean, English
+
+## Training Details
+### Dataset Preperation
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+- **HuggingFace:** [HuggingFace Silver-Multimodal-Dataset](https://huggingface.co/datasets/SilverAvocado/Silver-Multimodal-Dataset)
+- **Description:**
+   - The dataset is designed to support the development of machine learning models for detecting daily activities, violence, and fall down scenarios from combined audio and video sources.
+    - The preprocessing pipeline leverages audio feature extraction, human keypoint detection, and relative positional encoding to generate a unified representation for training and inference.
+	- Classes:
+	    - 0: Daily - Normal indoor activities
+	    - 1: Violence - Aggressive behaviors
+	    - 2: Fall Down - Sudden falls or collapses
+
+### Model Description
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+- **Model Structure:**
+    ![Multimodal Model Structure](./pics/model-structure.png)
+
+    - Input Shape and Division
+	    1.	Input Shape:
+            - The input shape for each branch is (N, 100, 750), where:
+	            - N: Batch size (number of sequences in a batch).
+	            - 100: Temporal dimension (time steps).
+	            - 750: Feature dimension, representing extracted features for each input modality.
+        2.	Why Four Inputs?:
+            - The model processes four distinct inputs, each corresponding to a specific set of features derived from video keypoints. Here’s how they are divided:
+            - Input 1, Input 2, Input 3:
+                - For each detected individual (up to 3 people), the model extracts 30 keypoints using MediaPipe.
+                - Each keypoint contains 3 features (x, y, z), resulting in 30 \times 3 = 90 features per frame.
+            - Input 4:
+                - Represents relative positional coordinates calculated from the 10 most important key joints (e.g., shoulders, elbows, knees) for all 3 individuals.
+                - These relative coordinates capture spatial relationships among individuals, crucial for contextual understanding.
+
+    - Detailed Explanation of Architecture
+	    1.	Positional Encoding:
+	        - Adds temporal position information to the input embeddings, allowing the transformer to consider the sequence order.
+	    2.	Multi-Head Attention:
+	        - Captures interdependencies and relationships across the temporal dimension within each input.
+	        - Ensures the model focuses on the most relevant frames or segments of the sequence.
+	    3.	Dropout:
+	        - Applies dropout regularization to prevent overfitting and improve generalization.
+	    4.	LayerNormalization:
+	        - Normalizes the output of each layer to stabilize training and accelerate convergence.
+	    5.	Dense Layers:
+	        - Extracts higher-level features after the attention mechanism.
+	        - The first dense layer processes features from attention, followed by another dropout and dense layer to refine features further.
+	    6.	AttentionPooling1D:
+	        - Combines outputs from all four inputs into a unified representation.
+	        - Aggregates temporal features using an attention mechanism, emphasizing the most important segments across modalities.
+	    7.	Final Dense Layers:
+	        - The combined representation is passed through dense layers and a softmax activation function for final classification into target classes:
+	            - 0: Daily Activities
+	            - 1: Violence
+	            - 2: Fall Down
+
+- **Model Performance:**
+    ![Confusion Matrix](./pics/confusion-matrix.png)
+    
+    - Confusion Matrix Insights:
+	    - Class 0 (Daily): 100% accuracy with no misclassifications.
+	    - Class 1 (Violence): 96.96% accuracy with minimal false positives or false negatives.
+	    - Class 2 (Fall Down): 98.67% accuracy, highlighting the model’s robustness in detecting falls.
+	    - The overall accuracy is 98.37%, indicating the model’s reliability for real-time applications.
+    
+
+## Model Usage
+- `Silver Assistant` Project
+    - [GitHub SilverAvocado](https://github.com/silverAvocado)
+
+## Load Model For Inference
 ```python
 # Hugging Face Hub에서 모델 다운로드
 MODEL_PATH="silver_assistant_transformer.keras"
-model_path = hf_hub_download(repo_id="SilverAvocado/silverAssistant", filename=MODEL_PATH)
+model_path = hf_hub_download(repo_id="SilverAvocado/Silver-Multimodal", filename=MODEL_PATH)
 
 # 사용자 정의 클래스 로드
 model = load_model(
@@ -32,4 +133,14 @@ model = load_model(
 
 y_pred = np.argmax(model.predict([X_test1, X_test2, X_test3, X_test4]), axis=1)
 accuracy = accuracy_score(y_test, y_pred)
-print(f"Test Accuracy: {accuracy:.4f}")
+print(f"Test Accuracy: {accuracy:.4f}")
+```
+
+## Conclusion
+- The Silver-Multimodal model demonstrates exceptional capabilities in multimodal learning for situation classification. 
+- Its ability to effectively integrate audio and video modalities ensures:
+	1.	High Accuracy: Consistent performance across all classes.
+	2.	Real-World Applicability: Suitable for applications like healthcare monitoring, safety systems, and smart homes.
+	3.	Scalable Architecture: Transformer-based design allows future enhancements and additional modality integration.
+
+- This model sets a new benchmark for multimodal AI systems, empowering safety-critical projects like `Silver Assistant` with state-of-the-art situation awareness.