Instructions to use abacusai/Llama-3-Giraffe-70B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use abacusai/Llama-3-Giraffe-70B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="abacusai/Llama-3-Giraffe-70B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("abacusai/Llama-3-Giraffe-70B-Instruct")
model = AutoModelForCausalLM.from_pretrained("abacusai/Llama-3-Giraffe-70B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use abacusai/Llama-3-Giraffe-70B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "abacusai/Llama-3-Giraffe-70B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abacusai/Llama-3-Giraffe-70B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/abacusai/Llama-3-Giraffe-70B-Instruct

SGLang

How to use abacusai/Llama-3-Giraffe-70B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "abacusai/Llama-3-Giraffe-70B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abacusai/Llama-3-Giraffe-70B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "abacusai/Llama-3-Giraffe-70B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abacusai/Llama-3-Giraffe-70B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use abacusai/Llama-3-Giraffe-70B-Instruct with Docker Model Runner:
```
docker model run hf.co/abacusai/Llama-3-Giraffe-70B-Instruct
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Llama-3-Giraffe-70B-Instruct

Abacus.AI presents our longer-necked variant of Llama 3 70B - now with the instruct variant!

This model has an effective context length of approximately 128k.

We have currently trained on ~1.5B tokens.

There are our Needle-in-a-Haystack heatmap results. We are conducting further evals of model efficacy and will update our model card as these come in:

MT-Bench Evaluation

We also measured performance on MT-Bench to verify that the context extension did not significantly impact performance on instruct tasks:

####### 1st turn:
Meta-Llama-3-70B-Instruct      9.21
Llama-3-Giraffe-70B-Instruct 9.19

####### 2nd turn:
Meta-Llama-3-70B-Instruct     2   8.80
Llama-3-Giraffe-70B-Instruct 2   8.54

####### average:
Meta-Llama-3-70B-Instruct      9.00
Llama-3-Giraffe-70B-Instruct 8.87

Training Methodology

The methodology for training uses PoSE and dynamic-NTK interpolation.

NTK-scaling

The scale factor for NTK is 4. Note that we also tried theta-scaling but this did not work as well as NTK scaling in our experiments.

PoSE

We utilise Positional Skip-wise Training (PoSE) with the following parameters:

Number of Chunks: 5
Max position ID: 32768

Data

We use on average ~8K long samples from RedPajama.

Hardware

We train on 8xH100 GPUs with Deepspeed Zero Stage 3.

Evaluation Methodology

We use the EasyContext implementation of Needle-in-a-Haystack to evaluate Llama-3-Giraffe-70B.

We evaluate with the following parameters:

Min context length: 2000
Max context length: 128000
Context interval: 4000
Depth interval: 0.1
Num samples: 2
Rnd number digits: 7
Haystack dir: PaulGrahamEssays

Adapter Transfer

We apply the above techniques first to Llama-3-70B-Base, using LoRA on the Q and K weights only. This adapter is then applied to Llama-3-70B-Instruct, and we release the merged version here.

Downloads last month: 30

Safetensors

Model size

71B params

Tensor type

BF16

Model tree for abacusai/Llama-3-Giraffe-70B-Instruct

Quantizations

2 models

Paper for abacusai/Llama-3-Giraffe-70B-Instruct

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

Paper • 2309.10400 • Published Sep 19, 2023 • 26