Update README.md

README.md

@@ -11,24 +11,28 @@ inference:
     temperature: 0.7
 ---
 
-## Model sheet for AstraQuasar-4.5B-v.0.1
+## Model sheet for AstraQuasar-4B
 
-**AstraQuasar-4.5B-v.0.1** is our first pre-trained Large Language Model (LLM) for text generation. It is a model with **4.5B parameters**.
-AstraQuasar-4.5B-v.0.1 is built upon the foundation of the Phi-2 architecture, with **significant enhancements including an increased number of layers and the innovative introduction of a novel technique known as the duplicate trick.**
+**AstraQuasar-4B** is our first pre-trained Large Language Model (LLM) for text generation. It is a model with **4B parameters**, whithout embeddings.
+AstraQuasar-4B-v.0.1 is built upon the foundation of the Phi-2 architecture, with **significant enhancements including an increased number of layers and the innovative introduction of a novel technique known as the duplicate trick.**
 
-AstraQuasar-4.5B-v.0.1 at the moment is an under trained model. Serving as a demonstration of the potential of the duplication trick and its implications for future advancements in language modeling. Despite its nascent status, our model has already demonstrated superior performance compared to both the base Phi-2 model and earlier iterations of AstraQuasar-4.5B that do not utilize the duplication trick.
+<p align="center">
+  <img src="https://cdn.discordapp.com/attachments/1196455895559839774/1207665552387080293/image.png?ex=65e07931&is=65ce0431&hm=cf9dcfada83cff4b70eda8302108b3a4267f066bac57455348b09f4cbb92e702&" width="800"/>
+  </p>
 
-One of the key milestones achieved by AstraQuasar-4.5B-v.0.1 is its successful application of backpropagation on the duplication trick, setting a precedent for future research and development in this area.
+
+AstraQuasar-4B-v.0.1 at the moment is an under trained model. Serving as a demonstration of the potential of the duplication trick and its implications for future advancements in language modeling. Despite its nascent status, our model has already demonstrated superior performance compared to both the base Phi-2 model and earlier iterations of AstraQuasar-4B that do not utilize the duplication trick.
+
+One of the key milestones achieved by AstraQuasar-4B is its successful application of backpropagation on the duplication trick, setting a precedent for future research and development in this area.
 
 Our model's architecture is fully compatible with leading training frameworks such as Axolotl and LLaMA Factory, ensuring seamless integration into existing workflows leveraging the standard Hugging Face Transformers library.
 
 ## Example:
-AstraQuasar-4.5B-v.0.1 can be easily instantiated using the Hugging Face Transformers library:
-
+AstraQuasar-4B can be easily instantiated using the Hugging Face Transformers library:
     from transformers import AutoTokenizer, AutoModelForCausalLM
     
-    model = AutoModelForCausalLM.from_pretrained("AstraMindAI/AstraQuasar-4.5B", trust_remote_code=True)
-    tokenizer = AutoTokenizer.from_pretrained("AstraMindAI/AstraQuasar-4.5B")
+    model = AutoModelForCausalLM.from_pretrained("AstraMindAI/AstraQuasar-4B", trust_remote_code=True)
+    tokenizer = AutoTokenizer.from_pretrained("AstraMindAI/AstraQuasar-4B")
     
     # you can optionally disable the duplicate trick
     # model.model.duplicate_trick = False
@@ -46,12 +50,15 @@ AstraQuasar-4.5B-v.0.1 can be easily instantiated using the Hugging Face Transfo
     generate_ids = model.generate(inputs.input_ids, max_length=30)
     tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
 
-Pre-training and fine-tuning can be performed using **accelerate**.
+Pre-training and fine-tuning can be performed using **accelerate** or **deepspeed**.
 
 ## Notice
 
-It's important to note that AstraQuasar-4.5B-v.0.1 is a pre-trained base model and does not incorporate any moderation mechanisms.
+It's important to note that AstraQuasar-4B is a pre-trained base model and does not incorporate any moderation mechanisms.
 
 ## NEWS
 
 Stay tuned for exciting developments! A new architecture, **AstraPulsar**, is on the horizon, promising further advancements in language modeling.
+
+## Credits: 
+- (Undi95)[https://huggingface.co/Undi95] for helping us to understanding the process self-calling layers.