Instructions to use Joseph717171/Modelz with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Joseph717171/Modelz with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Joseph717171/Modelz",
	filename="Hermes-4-14B-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use Joseph717171/Modelz with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Joseph717171/Modelz:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Joseph717171/Modelz:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Joseph717171/Modelz:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Joseph717171/Modelz:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Joseph717171/Modelz:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Joseph717171/Modelz:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Joseph717171/Modelz:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Joseph717171/Modelz:Q4_K_M

Use Docker

docker model run hf.co/Joseph717171/Modelz:Q4_K_M

LM Studio
Jan
Ollama
How to use Joseph717171/Modelz with Ollama:
```
ollama run hf.co/Joseph717171/Modelz:Q4_K_M
```

Unsloth Studio new

How to use Joseph717171/Modelz with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Joseph717171/Modelz to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Joseph717171/Modelz to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Joseph717171/Modelz to start chatting

Pi new

How to use Joseph717171/Modelz with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Joseph717171/Modelz:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Joseph717171/Modelz:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Joseph717171/Modelz with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Joseph717171/Modelz:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Joseph717171/Modelz:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use Joseph717171/Modelz with Docker Model Runner:
```
docker model run hf.co/Joseph717171/Modelz:Q4_K_M
```

Lemonade

How to use Joseph717171/Modelz with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Joseph717171/Modelz:Q4_K_M

Run and chat with the model

lemonade run user.Modelz-Q4_K_M

List all available models

lemonade list

Modelz / .gitattributes

Joseph717171

Upload Hermes-4-14B-Q4_K_M.gguf with huggingface_hub

12e3255 verified 9 months ago

raw

history blame contribute delete

7.06 kB

	*.7z filter=lfs diff=lfs merge=lfs -text
	*.arrow filter=lfs diff=lfs merge=lfs -text
	*.bin filter=lfs diff=lfs merge=lfs -text
	*.bz2 filter=lfs diff=lfs merge=lfs -text
	*.ckpt filter=lfs diff=lfs merge=lfs -text
	*.ftz filter=lfs diff=lfs merge=lfs -text
	*.gz filter=lfs diff=lfs merge=lfs -text
	*.h5 filter=lfs diff=lfs merge=lfs -text
	*.joblib filter=lfs diff=lfs merge=lfs -text
	.lfs. filter=lfs diff=lfs merge=lfs -text
	*.mlmodel filter=lfs diff=lfs merge=lfs -text
	*.model filter=lfs diff=lfs merge=lfs -text
	*.msgpack filter=lfs diff=lfs merge=lfs -text
	*.npy filter=lfs diff=lfs merge=lfs -text
	*.npz filter=lfs diff=lfs merge=lfs -text
	*.onnx filter=lfs diff=lfs merge=lfs -text
	*.ot filter=lfs diff=lfs merge=lfs -text
	*.parquet filter=lfs diff=lfs merge=lfs -text
	*.pb filter=lfs diff=lfs merge=lfs -text
	*.pickle filter=lfs diff=lfs merge=lfs -text
	*.pkl filter=lfs diff=lfs merge=lfs -text
	*.pt filter=lfs diff=lfs merge=lfs -text
	*.pth filter=lfs diff=lfs merge=lfs -text
	*.rar filter=lfs diff=lfs merge=lfs -text
	*.safetensors filter=lfs diff=lfs merge=lfs -text
	saved_model/*/ filter=lfs diff=lfs merge=lfs -text
	.tar. filter=lfs diff=lfs merge=lfs -text
	*.tar filter=lfs diff=lfs merge=lfs -text
	*.tflite filter=lfs diff=lfs merge=lfs -text
	*.tgz filter=lfs diff=lfs merge=lfs -text
	*.wasm filter=lfs diff=lfs merge=lfs -text
	*.xz filter=lfs diff=lfs merge=lfs -text
	*.zip filter=lfs diff=lfs merge=lfs -text
	*.zst filter=lfs diff=lfs merge=lfs -text
	tfevents filter=lfs diff=lfs merge=lfs -text
	Orca-2-X-17B-f16.gguf filter=lfs diff=lfs merge=lfs -text
	N0rcaMaid-7B-f16.gguf filter=lfs diff=lfs merge=lfs -text
	N0rcaMaid-7B-f16_old.gguf filter=lfs diff=lfs merge=lfs -text
	ggml-Orca-2-13b-SFT-v6-IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	workspace/opt/llama.cpp/ggml-OpenCodeInterpreter-CL-13B-IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	ggml-multi_verse_model-f16.gguf filter=lfs diff=lfs merge=lfs -text
	UNA-ThePitbull-21.4-v1-bf16-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	UNA-ThePitbull-21.4-v1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3-8B-Instruct-F32.IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Hathor_Fractionate-L3-8B-v.05-F32.IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Users/jsarnecki/opt/workspace/meta-llama/Meta-Llama-3-8B-Instruct/Meta-Llama-3-8B-Instruct-BF16.IQ5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3-8B-Instruct-BF16.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3-8B-Instruct-BF16.IQ5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Phi-3-mini-4k-instruct-F32.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Phi-3-mini-4k-instruct-IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Phi-3-mini-4k-instruct-F32.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Phi-3-mini-4k-instruct-q8_0.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Smegmma-Deluxe-9B-v1-F32.IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	gemma-2-9b-it-F32.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	gemma-2-9b-it-F16.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	gemma-2-9b-it-Q8_0.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	gemma-2-9b-it.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Phi-3-mini-4k-instruct-F16.IQ3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Llama-3SOME-8B-v2-F32.IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Replete-Coder-Instruct-8b-Adapted-Merged-F32.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Replete-Coder-Instruct-8b-Adapted-Merged-F32.IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-F16.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-F32.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-F32.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-F16.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-F32.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Mistral-Nemo-12B-Instruct-2407-Q8_0.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Mistral-Nemo-12B-Instruct-2407-F16.IQ5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-Q8_0.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-Q8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Lumimaid-v0.2-8B-Q8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Replete-AI_Replete-LLM-Qwen2-7.6B-7b_Beta-Preview-Q8_0.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-Q8_0.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	DeepSeek-Coder-V2-Lite-64x1.5B-Instruct-Q8_0.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Mistral-Nemo-12B-Instruct-2407-Q8_0.IQ4_K.gguf filter=lfs diff=lfs merge=lfs -text
	Qwen2-1.5B-Q8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	arcee-lite-Qwen2-1.5B-Q8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	arcee-lite-1.5B-Q8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-Q8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-Q8_0.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Mistral-Nemo-12B-Instruct-2407-F32.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EF16.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EF32.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EF32.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EF32.IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EBF16.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Llama-3.1-SuperNova-Lite-8.0B-OQ8_0.EF32.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Llama-3.1-SuperNova-Lite-8.0B-F32.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EQ8_0.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-3-Llama-3.1-8B-OQ8_0.EQ8_0.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	DeepSeek-R1-Distill-Llama-8B-OF16.EF16.IQ6_K.gguf filter=lfs diff=lfs merge=lfs -text
	DeepHermes-3-Llama-3-8B-Preview.IQ4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
	DeepHermes-3-Llama-3-8B-Preview.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
	DeepHermes-3-Llama-3-8B-Preview_No_Output_Tensor_Weights_Included_from_Imatrix-OF32.EF32.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Llama-3.1-SuperNova-Lite-ATTN_Q_F32-ATTN_K_F32_ATTN_V_F32-OQ8_0.EF32.IQ8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Meta-Llama-3.1-8B-Instruct-OQ8_0.EF32.IQ8_0XL.gguf filter=lfs diff=lfs merge=lfs -text
	Andy-4-0516-IQ5_0.gguf filter=lfs diff=lfs merge=lfs -text
	Andy-4-0516-IQ4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Andy-4-0516-IQ5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	gpt-oss-20b-.MXFP4_MOE.gguf filter=lfs diff=lfs merge=lfs -text
	gpt-oss-20b.MXFP4_MOE.gguf filter=lfs diff=lfs merge=lfs -text
	gpt-oss-20b.MXFP4.gguf filter=lfs diff=lfs merge=lfs -text
	gpt-oss-20B-MXFP4.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-4-14B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-4-14B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-4-14B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
	Hermes-4-14B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text