bilalfaye commited on
Commit
f79703b
Β·
verified Β·
1 Parent(s): 71e38de

OneEncoder for text, image and audio with temperature = 2.5 during training.

Browse files
Files changed (2) hide show
  1. README.md +3 -32
  2. model.safetensors +2 -2
README.md CHANGED
@@ -2,37 +2,8 @@
2
  tags:
3
  - model_hub_mixin
4
  - pytorch_model_hub_mixin
5
- license: mit
6
- language:
7
- - en
8
- base_model:
9
- - openai/clip-vit-large-patch14
10
- - google-bert/bert-base-uncased
11
- - facebook/wav2vec2-base-960h
12
  ---
13
 
14
- # πŸ–ΌοΈπŸ“ OneEncoder: A Unified Text & Image & Audio Model
15
-
16
- **OneEncoder** is a lightweight framework for cross-modal alignment, focusing on efficiently integrating **text and images** (with future extensions to other modalities). Unlike traditional methods relying on massive modality-specific encoders, OneEncoder progressively aligns different data types, making it cost-effective and performant even on small paired datasets.
17
-
18
- ## πŸš€ Key Features
19
- βœ… **Multimodal Alignment**: Initially supports **text & image**, with extension to other modalities.
20
- βœ… **Lightweight & Efficient**: Avoids full retraining when adding new modalities.
21
- βœ… **Superior Performance**: Outperforms models that require large specialized datasets.
22
-
23
- ## 🎯 Applications
24
- - **Visual Question Answering (VQA)**
25
- - **Image-Text Retrieval**
26
- - **Multimodal Content Understanding**
27
-
28
- ## πŸ“ Authors
29
- πŸ“Œ **Bilal FAYE**, Hanane AZZAG, Mustapha LEBBAH, Djamel BOUCHAFFRA
30
-
31
- ## πŸ“„ Research Paper
32
- πŸ“œ **arXiv**: [OneEncoder: Progressive Cross-Modal Alignment](https://arxiv.org/abs/2409.11059)
33
-
34
- ## πŸ“Œ Resources
35
- πŸ”— **GitHub Repo**: [OneEncoder](https://github.com/b-faye/OneEncoder)
36
- πŸš€ **Hugging Face Demo**: [OneEncoder Retriever](https://huggingface.co/spaces/bilalfaye/OneEncoder-retriever)
37
- πŸ““ **Demo Notebook**: [OneEncoder Demos](https://github.com/b-faye/OneEncoder/tree/main/demo)
38
- πŸ”Š **OneEncoder for Text, Image**: [HF Model](https://huggingface.co/bilalfaye/OneEncoder-text-image)
 
2
  tags:
3
  - model_hub_mixin
4
  - pytorch_model_hub_mixin
 
 
 
 
 
 
 
5
  ---
6
 
7
+ This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
8
+ - Library: [More Information Needed]
9
+ - Docs: [More Information Needed]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7a8b6e88f2ff9007007ad99ae8359ddd6045e35c27d1e3cff39dc2be7eb7b1af
3
- size 1217021568
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:01a06ebec4744b25977481615b0524d2dc335db0d1513f55b11b3083a42cef29
3
+ size 1172721276