NorthernTribe-Research/UMSR-Reasoner-7B-NF4

UMSR-Reasoner-7B (NF4)

4-bit NF4 bitsandbytes runtime quantization profile.

Model Tree

• Source model: https://huggingface.co/NorthernTribe-Research/UMSR-Reasoner-7B
• Variant repo: https://huggingface.co/NorthernTribe-Research/UMSR-Reasoner-7B-NF4
• Dataset: https://huggingface.co/datasets/NorthernTribe-Research/UMSR-v1

Quantization Profile

• Variant: nf4
• Scheme: bitsandbytes nf4 (4-bit)
• Materialized weights in this repo: False

Usage

This variant repository is a profile card. Load weights from the source model id.

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

model_id = "NorthernTribe-Research/UMSR-Reasoner-7B"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    quantization_config=bnb_config,
)

Output Contract

<final_answer>...</final_answer>