Update README.md

README.md

@@ -1,3 +1,88 @@
----
-license: llama3
----
+---
+license: llama3
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- nlp
+---
+
+# LLama-3-8B-Tele Model Card
+
+## Model Summary
+
+The language model LLama-3-8B-Tele is a Transformer with **8 billion** parameters, specialized in telecommunications. It is based on Meta [LLama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) and was continutally pretrained on [Tele-Data](https://huggingface.co/datasets/AliMaatouk/Tele-Data), a large-scale dataset of approximately 2.5 billion tokens of telecommunications material, including articles, standards, and general web content related to the telecommunications domain.
+
+When assessed against telecommunications benchmarks such as [Tele-Eval](https://huggingface.co/datasets/AliMaatouk/Tele-Eval), LLama-3-8B-Tele outperforms [LLama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) by several percentage points. Additionally, LLama-3-8B-Tele matches [LLama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) across benchmarks related to common sense, language understanding, and logical reasoning. Thus, this adaptation was achieved with minimal compromise in performance on the original version.
+
+### Context Length
+
+The model was trained on a context length of 8192 tokens.
+
+## Usage
+
+LLama-3-8B-Tele is a base model best suited for fine-tuning on applications related to telecommunications. It has not been fine-tuned to follow instructions and operates solely within a text completion framework. An example of this completion can be found below:
+
+```markdown
+Prompt: Shannon capacity is
+
+Model: the maximum rate at which information can be transmitted over a communication channel. It was named after Claude Shannon, who introduced it in his 1948 paper "A Mathematical Theory of Communication". Shannon capacity is also known as channel capacity, channel transmission capacity, or simply capacity.
+```
+
+The instruct version of this model can be found by following the link [LLama-3-8B-Tele-it](https://huggingface.co/AliMaatouk/LLama-3-8B-Tele-it). 
+
+## Sample Code
+
+Below we share some code snippets on how to get quickly started with running the model. First, make sure to `pip install transformers`, then copy the snippet corresponding to your hardware and adapt it to your usecase.
+
+#### Running the model on a CPU
+
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained("AliMaatouk/LLama-3-8B-Tele", torch_dtype="auto")
+tokenizer = AutoTokenizer.from_pretrained("AliMaatouk/LLama-3-8B-Tele")
+
+prompt = "Shannon capacity is"
+input_ids = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**input_ids, max_new_tokens=100)
+
+generated_tokens = outputs[0, len(input_ids['input_ids'][0]):]
+response = tokenizer.decode(generated_tokens, skip_special_tokens=True)
+print(response)
+```
+
+#### Running the model on a single / multi GPU
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("AliMaatouk/LLama-3-8B-Tele", torch_dtype="auto", device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("AliMaatouk/LLama-3-8B-Tele")
+
+prompt = "Shannon capacity is"
+input_ids = tokenizer(prompt, return_tensors="pt").to("cuda")
+outputs = model.generate(**input_ids, max_new_tokens=100)
+
+generated_tokens = outputs[0, len(input_ids['input_ids'][0]):]
+response = tokenizer.decode(generated_tokens, skip_special_tokens=True)
+print(response)
+```
+
+## Citation
+
+You can find the paper with all details about the model at https://arxiv.org/abs/2409.05314. Please cite it as follows:
+
+```bib
+@misc{maatouk2024telellmsseriesspecializedlarge,
+      title={Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications}, 
+      author={Ali Maatouk and Kenny Chirino Ampudia and Rex Ying and Leandros Tassiulas},
+      year={2024},
+      eprint={2409.05314},
+      archivePrefix={arXiv},
+      primaryClass={cs.IT},
+      url={https://arxiv.org/abs/2409.05314}, 
+}
+```