Instructions to use OmnipotentFool/Aurvion with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use OmnipotentFool/Aurvion with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf OmnipotentFool/Aurvion:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf OmnipotentFool/Aurvion:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf OmnipotentFool/Aurvion:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf OmnipotentFool/Aurvion:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf OmnipotentFool/Aurvion:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf OmnipotentFool/Aurvion:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf OmnipotentFool/Aurvion:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf OmnipotentFool/Aurvion:Q4_K_M

Use Docker

docker model run hf.co/OmnipotentFool/Aurvion:Q4_K_M

LM Studio
Jan

vLLM

How to use OmnipotentFool/Aurvion with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OmnipotentFool/Aurvion"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OmnipotentFool/Aurvion",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OmnipotentFool/Aurvion:Q4_K_M

Ollama
How to use OmnipotentFool/Aurvion with Ollama:
```
ollama run hf.co/OmnipotentFool/Aurvion:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use OmnipotentFool/Aurvion with Docker Model Runner:
```
docker model run hf.co/OmnipotentFool/Aurvion:Q4_K_M
```

Lemonade

How to use OmnipotentFool/Aurvion with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull OmnipotentFool/Aurvion:Q4_K_M

Run and chat with the model

lemonade run user.Aurvion-Q4_K_M

List all available models

lemonade list

Atomic Chat

Aurvion / llama.cpp /examples /model-conversion /scripts /embedding /compare-embeddings-logits.sh

OmnipotentFool

Add files using upload-large-folder tool

db65755 verified 4 months ago

Raw History Blame Contribute Delete

2.3 kB

#!/usr/bin/env bash

set -e

# Parse command line arguments
MODEL_PATH=""
MODEL_NAME=""
PROMPTS_FILE=""

# First argument is always model path
if [ $# -gt 0 ] && [[ "$1" != --* ]]; then
    MODEL_PATH="$1"
    shift
fi

# Parse remaining arguments
while [[ $# -gt 0 ]]; do
    case $1 in
        --prompts-file|-pf)
            PROMPTS_FILE="$2"
            shift 2
            ;;
        *)
            # If MODEL_NAME not set and this isn't a flag, use as model name
            if [ -z "$MODEL_NAME" ] && [[ "$1" != --* ]]; then
                MODEL_NAME="$1"
            fi
            shift
            ;;
    esac
done

# Set defaults
MODEL_PATH="${MODEL_PATH:-"$EMBEDDING_MODEL_PATH"}"
MODEL_NAME="${MODEL_NAME:-$(basename "$MODEL_PATH")}"

CONVERTED_MODEL_PATH="${CONVERTED_EMBEDDING_PATH:-"$CONVERTED_EMBEDDING_MODEL"}"
CONVERTED_MODEL_NAME="${CONVERTED_MODEL_NAME:-$(basename "$CONVERTED_MODEL_PATH" .gguf)}"

if [ -t 0 ]; then
    CPP_EMBEDDINGS="data/llamacpp-${CONVERTED_MODEL_NAME}-embeddings.bin"
else
    # Process piped JSON data and convert to binary (matching logits.cpp format)
    TEMP_FILE=$(mktemp /tmp/tmp.XXXXXX.binn)
    python3 -c "
import json
import sys
import struct

data = json.load(sys.stdin)

# Flatten all embeddings completely
flattened = []
for item in data:
    embedding = item['embedding']
    for token_embedding in embedding:
        flattened.extend(token_embedding)

print(f'Total embedding values: {len(flattened)}', file=sys.stderr)

# Write as binary floats - matches logitc.cpp fwrite format
with open('$TEMP_FILE', 'wb') as f:
    for value in flattened:
        f.write(struct.pack('f', value))
"
    CPP_EMBEDDINGS="$TEMP_FILE"
    trap "rm -f $TEMP_FILE" EXIT
fi

# Build the semantic_check.py command
SEMANTIC_CMD="python scripts/utils/semantic_check.py --model-path $MODEL_PATH \
    --python-embeddings data/pytorch-${MODEL_NAME}-embeddings.bin \
    --cpp-embeddings $CPP_EMBEDDINGS"

# Add prompts file if specified, otherwise use default prompt
if [ -n "$PROMPTS_FILE" ]; then
    SEMANTIC_CMD="$SEMANTIC_CMD --prompts-file \"$PROMPTS_FILE\""
else
    SEMANTIC_CMD="$SEMANTIC_CMD --prompt \"Hello world today\""
fi

# Execute the command
eval $SEMANTIC_CMD

	#!/usr/bin/env bash

	set -e

	# Parse command line arguments
	MODEL_PATH=""
	MODEL_NAME=""
	PROMPTS_FILE=""

	# First argument is always model path
	if [ $# -gt 0 ] && [[ "$1" != --* ]]; then
	MODEL_PATH="$1"
	shift
	fi

	# Parse remaining arguments
	while [[ $# -gt 0 ]]; do
	case $1 in
	--prompts-file\|-pf)
	PROMPTS_FILE="$2"
	shift 2
	;;
	*)
	# If MODEL_NAME not set and this isn't a flag, use as model name
	if [ -z "$MODEL_NAME" ] && [[ "$1" != --* ]]; then
	MODEL_NAME="$1"
	fi
	shift
	;;
	esac
	done

	# Set defaults
	MODEL_PATH="${MODEL_PATH:-"$EMBEDDING_MODEL_PATH"}"
	MODEL_NAME="${MODEL_NAME:-$(basename "$MODEL_PATH")}"

	CONVERTED_MODEL_PATH="${CONVERTED_EMBEDDING_PATH:-"$CONVERTED_EMBEDDING_MODEL"}"
	CONVERTED_MODEL_NAME="${CONVERTED_MODEL_NAME:-$(basename "$CONVERTED_MODEL_PATH" .gguf)}"

	if [ -t 0 ]; then
	CPP_EMBEDDINGS="data/llamacpp-${CONVERTED_MODEL_NAME}-embeddings.bin"
	else
	# Process piped JSON data and convert to binary (matching logits.cpp format)
	TEMP_FILE=$(mktemp /tmp/tmp.XXXXXX.binn)
	python3 -c "
	import json
	import sys
	import struct

	data = json.load(sys.stdin)

	# Flatten all embeddings completely
	flattened = []
	for item in data:
	embedding = item['embedding']
	for token_embedding in embedding:
	flattened.extend(token_embedding)

	print(f'Total embedding values: {len(flattened)}', file=sys.stderr)

	# Write as binary floats - matches logitc.cpp fwrite format
	with open('$TEMP_FILE', 'wb') as f:
	for value in flattened:
	f.write(struct.pack('f', value))
	"
	CPP_EMBEDDINGS="$TEMP_FILE"
	trap "rm -f $TEMP_FILE" EXIT
	fi

	# Build the semantic_check.py command
	SEMANTIC_CMD="python scripts/utils/semantic_check.py --model-path $MODEL_PATH \
	--python-embeddings data/pytorch-${MODEL_NAME}-embeddings.bin \
	--cpp-embeddings $CPP_EMBEDDINGS"

	# Add prompts file if specified, otherwise use default prompt
	if [ -n "$PROMPTS_FILE" ]; then
	SEMANTIC_CMD="$SEMANTIC_CMD --prompts-file \"$PROMPTS_FILE\""
	else
	SEMANTIC_CMD="$SEMANTIC_CMD --prompt \"Hello world today\""
	fi

	# Execute the command
	eval $SEMANTIC_CMD