Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,93 @@
+---
+license: other
+language:
+- en
+pipeline_tag: text-generation
+inference: false
+tags:
+- transformers
+- gguf
+- imatrix
+- starcoder2-7b
+---
+Quantizations of https://huggingface.co/bigcode/starcoder2-7b
+
+
+# From original readme
+
+### Generation
+Here are some examples to get started with the model. You can find a script for fine-tuning in StarCoder2's [GitHub repository](https://github.com/bigcode-project/starcoder2).
+
+First, make sure to install `transformers` from source:
+```bash
+pip install git+https://github.com/huggingface/transformers.git
+```
+
+#### Running the model on CPU/GPU/multi GPU
+* _Using full precision_
+```python
+# pip install git+https://github.com/huggingface/transformers.git # TODO: merge PR to main
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+checkpoint = "bigcode/starcoder2-7b"
+device = "cuda" # for GPU usage or "cpu" for CPU usage
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
+model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
+
+inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
+outputs = model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
+```bash
+>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
+Memory footprint: 29232.57 MB
+```
+* _Using `torch.bfloat16`_
+```python
+# pip install accelerate
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+checkpoint = "bigcode/starcoder2-7b"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+
+# for fp16 use `torch_dtype=torch.float16` instead
+model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", torch_dtype=torch.bfloat16)
+
+inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
+outputs = model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
+```bash
+>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
+Memory footprint: 14616.29 MB
+```
+
+#### Quantized Versions through `bitsandbytes`
+* _Using 8-bit precision (int8)_
+
+```python
+# pip install bitsandbytes accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+
+# to use 4bit use `load_in_4bit=True` instead
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+
+checkpoint = "bigcode/starcoder2-7b"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config)
+
+inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
+outputs = model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
+```bash
+>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
+# load_in_8bit
+Memory footprint: 7670.52 MB
+# load_in_4bit
+>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
+Memory footprint: 4197.64 MB
+```