wip

README.md

@@ -15,7 +15,6 @@ Some cool model...
 
 - [Model Card for m4-80b](#model-card-for--model_id-)
 - [Table of Contents](#table-of-contents)
-- [Table of Contents](#table-of-contents-1)
 - [Model Details](#model-details)
   - [Model Description](#model-description)
 - [Uses](#uses)
@@ -57,15 +56,14 @@ Some cool model...
 <!-- Provide a longer summary of what this model is/does. -->
 Some cool model...
 
-- **Developed by:** More information needed
-- **Shared by [Optional]:** More information needed
-- **Model type:** Language model
+- **Developed by:** HuggingFace
+- **Model type:** Multi-modal model (text+image)
 - **Language(s) (NLP):** en
 - **License:** apache-2.0
-- **Parent Model:** More information needed
+- **Parent Model:** [laion/CLIP-ViT-H-14-laion2B-s32B-b79K](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) and  [huggingface/llama-65b](https://huggingface.co/huggingface/llama-65b)
 - **Resources for more information:** More information needed
     - [GitHub Repo](https://github.com/huggingface/m4/)
-    - [Associated Paper](Flamingo)
+    - Associated Paper: [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198)
 
 # Uses
 
@@ -172,10 +170,9 @@ More information needed
 
 Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-- **Hardware Type:** More information needed
-- **Hours used:** More information needed
-- **Cloud Provider:** More information needed
-- **Compute Region:** More information needed
+- **Hardware Type:** 64 nodes of 8x 80GB A100 gpus, EFA network
+- **Hours used:** ~672 node hours
+- **Cloud Provider:** AWS Sagemaker
 - **Carbon Emitted:** unknown
 
 # Technical Specifications [optional]
@@ -190,11 +187,15 @@ More information needed
 
 ### Hardware
 
-More information needed
+The training was performed on AWS SageMaker cluster with 64 nodes of 8x80GB A100 GPUs (512 GPUs total). The cluster uses the current EFA network which provides about 340GBps throughput.
+
+As the network is quite slow for the needs of DeepSpeed ZeRO-3 we were only able to clock ~90 TFLOPs.
+
 
 ### Software
 
-More information needed
+The training software is built on top of HuggingFace Transformers + Accelerate, and DeepSpeed ZeRO-3. Plus [WebDataset](https://github.com/webdataset/webdataset) for data loading.
+
 
 # Citation