Merge branch 'main' of https://huggingface.co/GeoV/GeoV-9b into main

README.md

@@ -8,13 +8,13 @@ license: bigscience-openrail-m
 ---
 
 
-[GeoV](https://huggingface.co/docs/transformers/model_doc/geov)-9B is a 9 billion parameter autoregressive language model.
+[GeoV](https://github.com/geov-ai/geov)-9B is a 9 billion parameter causal language model.
 
 The GeoV model was designed by Georges Harik and uses 
-[Rotary Positional Embeddings with Relative distances (RoPER)](http://research.labml.ai/RoPER.html)
-by [Georges Hark](https://twitter.com/ghark) and [Varuna Jayasiri](https://twitter.com/vpj).
+[Rotary Positional Embeddings with Relative distances (RoPER)](https://research.labml.ai/RoPER.html)
+by [Georges Harik](https://twitter.com/gharik) and [Varuna Jayasiri](https://twitter.com/vpj).
 
-[RoPER]((http://research.labml.ai/RoPER.html), 
+[RoPER](https://research.labml.ai/RoPER.html), 
 in addition to using relative positions in the attention score calculation by RoPE embeddings,
 adds relative positional information explicitly to value embeddings.
 Specifically, it incorporates the relative positions of the tokens paid attention to.
@@ -36,29 +36,38 @@ RoPER has given better performance in some algorithmic tasks, and seems comparab
 | n<sub>heads</sub>      | 40          |
 | d<sub>head</sub>       | 128         |
 | n<sub>vocab</sub>      | 65500       |
-| Sequence Length        | 2049        |
+| Sequence Length        | 2048        |
 </figure>
 
+The released weights were trained on ~70 billion tokens.
+We plan to continue training up to 300 billion tokens and update the weights at every 20b tokens.
+This training run is monolingual and uses c4en and english wikipedia datasets.
+
+## Installation
+
+```shell
+pip install geov
+```
 
 ## Generation
 
-The `generate()` method can be used to generate text using GeoV model.
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/geov-ai/geov/blob/master/notebooks/generate.ipynb)
 
 ```python
->>> from transformers import GeoVForCausalLM, GeoVTokenizer
+from geov import GeoVForCausalLM, GeoVTokenizer
 
->>> model = GeoVForCausalLM.from_pretrained("GeoV/GeoV-9b")
->>> tokenizer = GeoVTokenizer.from_pretrained("GeoV/GeoV-9b")
+model = GeoVForCausalLM.from_pretrained("GeoV/GeoV-9b")
+tokenizer = GeoVTokenizer.from_pretrained("GeoV/GeoV-9b")
 
->>> prompt = "In mathematics, topology is the study of"
+prompt = "In mathematics, topology is the study of"
 
->>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids
 
->>> gen_tokens = model.generate(
-...     input_ids,
-...     do_sample=True,
-...     temperature=0.9,
-...     max_length=100,
-... )
->>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
+gen_tokens = model.generate(
+    input_ids,
+    do_sample=True,
+    temperature=0.9,
+    max_length=100,
+)
+gen_text = tokenizer.batch_decode(gen_tokens)[0]
 ```