Instructions to use Hypereum/HivemindEval with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Hypereum/HivemindEval with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Hypereum/HivemindEval")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Hypereum/HivemindEval")
model = AutoModelForCausalLM.from_pretrained("Hypereum/HivemindEval")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

llama-cpp-python

How to use Hypereum/HivemindEval with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Hypereum/HivemindEval",
	filename="hivemind_evaluator_q3.Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use Hypereum/HivemindEval with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Hypereum/HivemindEval:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Hypereum/HivemindEval:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Hypereum/HivemindEval:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Hypereum/HivemindEval:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Hypereum/HivemindEval:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Hypereum/HivemindEval:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Hypereum/HivemindEval:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Hypereum/HivemindEval:Q4_K_M

Use Docker

docker model run hf.co/Hypereum/HivemindEval:Q4_K_M

LM Studio
Jan

vLLM

How to use Hypereum/HivemindEval with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Hypereum/HivemindEval"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hypereum/HivemindEval",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Hypereum/HivemindEval:Q4_K_M

SGLang

How to use Hypereum/HivemindEval with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Hypereum/HivemindEval" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hypereum/HivemindEval",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Hypereum/HivemindEval" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hypereum/HivemindEval",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use Hypereum/HivemindEval with Ollama:
```
ollama run hf.co/Hypereum/HivemindEval:Q4_K_M
```

Unsloth Studio new

How to use Hypereum/HivemindEval with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Hypereum/HivemindEval to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Hypereum/HivemindEval to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Hypereum/HivemindEval to start chatting

Pi new

How to use Hypereum/HivemindEval with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Hypereum/HivemindEval:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Hypereum/HivemindEval:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Hypereum/HivemindEval with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Hypereum/HivemindEval:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Hypereum/HivemindEval:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use Hypereum/HivemindEval with Docker Model Runner:
```
docker model run hf.co/Hypereum/HivemindEval:Q4_K_M
```

Lemonade

How to use Hypereum/HivemindEval with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Hypereum/HivemindEval:Q4_K_M

Run and chat with the model

lemonade run user.HivemindEval-Q4_K_M

List all available models

lemonade list

HivemindEval v2.0

HivemindEval v2.0 is an open-source AI output quality evaluator fine-tuned on Qwen3-8B for assessing multi-agent AI agent outputs across 6 quality dimensions. Released by Hypereum Ltd as the open-source component of the Hivemind AX multi-agent compliance orchestration platform.

The remaining six models in the Hivemind AX stack (planner, generic executor, three sector-specialised executors, synthesizer) are proprietary and not released.

What's New in v2.0

Base model: upgrade from Qwen2.5-7B-Instruct (v1.0) to Qwen3-8B (Apache 2.0, April 2025)
Improved structured output generation and instruction following
Longer context: 40,960 tokens (vs 32,768 in v1.0)
Trained on Cambridge Dawn HPC (Intel Data Centre GPU Max 1550) via UKRI AIRR programme

Model Description

Base model: Qwen/Qwen3-8B (Apache 2.0)
Fine-tuning method: LoRA (Cfg E winner: rank 64, alpha 128, learning rate 2e-5)
Training compute: UKRI AIRR — Cambridge Dawn HPC, Intel PVC Max 1550
Training data: ~20,000 synthetic compliance evaluation pairs (Wave 7 Cfg E)
Training duration: 20 hours wall-clock on 1× Intel PVC tile, 8 epochs
Validation loss: 0.4455 on held-out compliance scenarios

Evaluation Dimensions

HivemindEval scores AI agent outputs on 6 dimensions (0-100 each), output as structured JSON:

Accuracy — Factual correctness, regulatory citation validity
Completeness — Task coverage, gap detection
Regulatory Alignment — Citation accuracy, scope relevance
Actionability — Remediation specificity, implementability
Coherence — Logical structure, argumentation quality
Evidence Quality — Data points, observations, test results

Validated Performance

Internal benchmark of 50 realistic UK compliance scenarios across PSD2, GDPR, NHS DSPT, MOD JSP 440, Cyber Essentials Plus, and EU AI Act:

Inference strategy	Valid JSON rate	Median latency (Intel PVC)
Greedy, single-shot, 1200 max_tokens	50%	15.1s
Best-of-N (N=5), 2048 max_tokens, robust parser	100%	27.1s

Production deployments should use the best-of-N pattern (see below). The greedy single-shot result is reported for full transparency.

Recommended Inference Pattern

For production use, achieve 100% structured output reliability via best-of-N sampling with progressive temperature relaxation and robust JSON parsing:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, json, re

model = AutoModelForCausalLM.from_pretrained("hypereum/HivemindEval", torch_dtype=torch.float16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("hypereum/HivemindEval")

def robust_json_parse(text):
    try: return json.loads(text)
    except: pass
    try:
        s = text[text.index("{"):text.rindex("}")+1]
        s = re.sub(r",(\s*[}\]])", r"\1", s)
        return json.loads(s)
    except: pass
    return None

def evaluate(agent_output):
    prompt = f"Evaluate this AI compliance finding across 6 dimensions (Accuracy, Completeness, Regulatory Alignment, Actionability, Coherence, Evidence Quality). Provide scores 0-100 and reasoning. Respond in JSON.\n\n{agent_output}"
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    strategies = [
        {"do_sample": False},
        {"do_sample": True, "temperature": 0.05, "top_p": 0.9,  "top_k": 20, "repetition_penalty": 1.1},
        {"do_sample": True, "temperature": 0.1,  "top_p": 0.9,  "top_k": 30, "repetition_penalty": 1.1},
        {"do_sample": True, "temperature": 0.2,  "top_p": 0.95, "top_k": 40, "repetition_penalty": 1.05},
        {"do_sample": True, "temperature": 0.3,  "top_p": 0.95, "top_k": 50, "repetition_penalty": 1.05},
    ]
    for s in strategies:
        out = model.generate(**inputs, max_new_tokens=2048, pad_token_id=tokenizer.eos_token_id, **s)
        resp = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
        parsed = robust_json_parse(resp)
        if parsed is not None: return parsed
    return None  # all 5 attempts failed (observed 0% in internal benchmark)

GGUF (Ollama / llama.cpp)

hivemind_evaluator_q3.Q4_K_M.gguf — production deployment (4.7 GB)
hivemind_evaluator_q3.Q8_0.gguf — higher quality, 8-bit (8.2 GB)
hivemind_evaluator_q3.f16.gguf — full precision reference (16 GB)

Ollama Modelfile included. Best-of-N pattern via API loop, not via Modelfile parameters.

Known Limitations

Greedy single-shot is unreliable: 50% valid JSON rate. Production requires best-of-N pattern as documented above.
No jurisdiction-aware scoring: Current version does not flag regulatory framework jurisdiction mismatches (e.g., HIPAA cited for UK organisations). Roadmap v2.1 addresses this.
Validation loss higher than peer models: 0.4455 vs 0.23-0.27 for other Hivemind AX role models. Reflects the longer structured output schema (6 dimensions × {score + reasoning}) compared to simpler agent roles.
English only: Training data is English. Multilingual variants not supported.
Synthetic training data: Compliance scenarios generated via Qwen3-14B teacher distillation. Real-world deployment validation is ongoing.
Compliance focus is UK / EU regulated industries: Strongest performance on PSD2, GDPR, NHS DSPT, MOD JSP 440, Cyber Essentials Plus, EU AI Act. Other regulatory regimes are out of distribution.

Roadmap

v1.0 (May 2026): Qwen2.5-7B base — stable LTS, maintained
v2.0 (this release, May 2026): Qwen3-8B base — recommended
v2.1 (Q3 2026): Jurisdiction-aware scoring, expanded UK dataset
v3.0 (Q4 2026): Qwen3.6-27B base — commercial compute required

License

Apache 2.0 (inherited from Qwen3-8B base).

Citation

@misc{hivemindeval2026v2,
  author = {De Lillo, Giovanni and Hypereum Ltd},
  title = {HivemindEval v2.0: Open-Source AI Output Quality Evaluator (Qwen3-8B base)},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/hypereum/HivemindEval}
}

Acknowledgements

Model training supported by the UKRI AI Research Resource (AIRR) programme on the Dawn supercomputer at the University of Cambridge. Open Zettascale Lab, Research Computing Services. Intel Data Centre GPU Max 1550 compute platform. Qwen team for the Qwen3-8B base model.

Contact

Giovanni De Lillo — Founder and CEO, Hypereum Ltd
hypereum.tech — hello@hypereum.tech

Released as the open-source component of Hivemind AX. The full multi-agent compliance stack (Planner, Executor, sector-specialised Executors for Healthcare, Defence, and Fintech, and Synthesizer) is proprietary.

Downloads last month: -

Safetensors

Model size

8B params

Tensor type

F16

Model tree for Hypereum/HivemindEval

Base model

Qwen/Qwen3-8B-Base

Finetuned

Qwen/Qwen3-8B

Quantized

(279)

this model