Instructions to use arunvenkat17/Param-1-7B-GodMode-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use arunvenkat17/Param-1-7B-GodMode-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="arunvenkat17/Param-1-7B-GodMode-4bit", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("arunvenkat17/Param-1-7B-GodMode-4bit", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use arunvenkat17/Param-1-7B-GodMode-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "arunvenkat17/Param-1-7B-GodMode-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arunvenkat17/Param-1-7B-GodMode-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/arunvenkat17/Param-1-7B-GodMode-4bit

SGLang

How to use arunvenkat17/Param-1-7B-GodMode-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "arunvenkat17/Param-1-7B-GodMode-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arunvenkat17/Param-1-7B-GodMode-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "arunvenkat17/Param-1-7B-GodMode-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arunvenkat17/Param-1-7B-GodMode-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use arunvenkat17/Param-1-7B-GodMode-4bit with Docker Model Runner:
```
docker model run hf.co/arunvenkat17/Param-1-7B-GodMode-4bit
```

Param-1-7B God Mode Hybrid (4-bit / 16-bit)

A highly optimized, hybrid-quantized version of the bharatgenai/Param-1-7B model. This model utilizes a custom 4-bit/16-bit "God Mode" configuration to drastically reduce VRAM usage while preserving high mathematical accuracy by selectively keeping critical attention and routing mechanisms in 16-bit precision.

Model Details

Model Description

This model was created to enable consumer-hardware inference for the Param-1-7B architecture without suffering the severe degradation often associated with pure 4-bit quantization. It uses bitsandbytes NF4 quantization with a highly sensitive skip-list, keeping the vocabulary projector, context engine, attention aggregators, and Mixture-of-Experts (MoE) routers in bfloat16.

Developed by: Arun Venkat Jonna
Affiliation: Sustainable, Advanced, Robust Computing Systems (SARCS) Lab
Model type: Causal Language Model (Quantized / Mixture of Experts)
Language(s) (NLP): English
License: Inherited from bharatgenai/Param-1-7B
Quantized from base model: bharatgenai/Param-1-7B

Model Sources

Base Model Repository: bharatgenai/Param-1-7B

Uses

Direct Use

This model is intended for developers, researchers, and hobbyists who want to run the Param-1-7B model on GPUs with extremely limited VRAM (5 GB or more). Because the quantization configuration is baked directly into the config.json, it can be loaded natively with the transformers library without explicitly defining a BitsAndBytesConfig.

How to Get Started with the Model

Use the code below to get started with the model. Note that you do not need to define the skip-list; it is handled automatically.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "arunvenkat17/Param-1-7B-GodMode-4bit"

# Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Load Optimized Model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="sdpa"
)

print("Model loaded successfully!")
print(f"VRAM Used: {model.get_memory_footprint() / (1024 ** 3):.2f} GB")

Downloads last month: -

Safetensors

Model size

7B params

Tensor type

F16

Model tree for arunvenkat17/Param-1-7B-GodMode-4bit

Base model

bharatgenai/Param-1-7B

Quantized

(1)

this model