Instructions to use tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ")
model = AutoModelForCausalLM.from_pretrained("tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ

SGLang

How to use tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ with Docker Model Runner:
```
docker model run hf.co/tmpupload/superhot-7b-8k-no-rlhf-test-GPTQ
```

superhot-7b-8k-4bit--1g-safetensors

Note: Maximum sequence length (max_seq_len) and compression factor (compress_pos_emb) need to be set to 8192 (or lower) and 4.

Merged base LLaMA and LoRA with this: https://github.com/tloen/alpaca-lora

Base LLaMA 7B: https://huggingface.co/huggyllama/llama-7b

SuperHOT 7B 8k no-rlhf-test LoRA: https://huggingface.co/kaiokendev/superhot-7b-8k-no-rlhf-test

BASE_MODEL=huggyllama_llama-7b LORA=kaiokendev_superhot-7b-8k-no-rlhf-test python export_hf_checkpoint.py

Quantized with AutoGPTQ: https://github.com/PanQiWei/AutoGPTQ

python quant_with_alpaca.py --pretrained_model_dir superhot-7b-8k-safetensors --quantized_model_dir superhot-7b-8k-no-rlhf-test-GPTQ --bits 4 --group_size -1 --desc_act --num_samples 256 --save_and_reload

Perplexity:

CUDA_VISIBLE_DEVICES=0 python test_benchmark_inference.py \
         -d /workspace/models/superhot-7b-8k-no-rlhf-test-GPTQ \
         -ppl \
         -ppl_ds datasets/wikitext2.txt \
         -l 8192 \
         -cpe 4 \
         -ppl_cn 40 \
         -ppl_cs 8192 \
         -ppl_ct 8192
 -- Perplexity:
 -- - Dataset: datasets/wikitext2.txt
 -- - Chunks: 40
 -- - Chunk size: 8192 -> 8192
 -- - Chunk overlap: 0
 -- - Min. chunk size: 50
 -- - Key: text
 -- Tokenizer: /workspace/models/superhot-7b-8k-no-rlhf-test-GPTQ/tokenizer.model
 -- Model config: /workspace/models/superhot-7b-8k-no-rlhf-test-GPTQ/config.json
 -- Model: /workspace/models/superhot-7b-8k-no-rlhf-test-GPTQ/4bit.safetensors
 -- Sequence length: 8192
 -- RoPE compression factor: 4.0
 -- Tuning:
 -- --matmul_recons_thd: 8
 -- --fused_mlp_thd: 2
 -- --sdp_thd: 8
 -- Options: ['perplexity']
 ** Time, Load model: 2.74 seconds
 ** Time, Load tokenizer: 0.01 seconds
 -- Groupsize (inferred): None
 -- Act-order (inferred): no
 !! Model has empty group index (discarded)
 ** VRAM, Model: [cuda:0] 3,652.09 MB
 -- Loading dataset...
 -- Testing 40 chunks....
 ** Perplexity: 7.0522

Downloads last month: 6