SolarHive

SolarHive E4B GGUF β€” Edge Solar Energy Intelligence

Fine-tuned Gemma 4 E4B (8B) in GGUF format for Ollama and llama.cpp edge deployment. A standard Q4_K_M text quantization (shipped under two interchangeable filenames, each with its own Modelfile) plus one multimodal projector companion file. In one local run on a CPU-only Microsoft Surface Pro 8 (Intel i5-1135G7, 16 GB RAM), with the GGUF and Ollama blob cache stored on an external USB drive, it scored 5 of 5 on tool routing and 2 of 3 on the When2Call probes. The Q&A half was recorded, not graded. The results file from that run is not committed.

Built for the Gemma 4 Good Hackathon (Google DeepMind Γ— Kaggle).

Base Model google/gemma-4-e4b-it
Architecture Dense + PLE β€” 8B total, 4.5B effective
Quantization Q4_K_M β€” standard recipe (Q6_K on the PLE tensor)
Files 5.34 GB Q4_K_M text (Γ—2 filenames, same recipe) Β· 992 MB mmproj
Total footprint 6.3 GB (text + mmproj)
Modalities Text, Image, Audio (via mmproj companion)
Function Calling Native Gemma 4 protocol
Benchmark 10/10 in one local run: 5/5 Q&A recorded (not graded) + 5/5 tool routing, plus 2/3 When2Call. Project prompts, not held out. Results file not committed
Local deployment CPU-only Surface Pro 8 (Intel i5-1135G7 @ 2.4 GHz, 16 GB RAM, Intel Iris Xe unused), Ollama 0.21.0 with llama.cpp ggml-cpu-icelake.dll (AVX2 + AVX512 + VNNI). External USB drive holds the 5.34 GB GGUF + Ollama blob cache.
Fine-Tuning LoRA via Unsloth (BF16)
Training Data 1,727 examples (solarhive-community-solar-multimodal) β€” text-only fine-tune; VQA at inference uses the base Gemma 4 vision encoder (~150M params for E4B), unmodified by our LoRA per the Vertex AI SFT recipe. The 992 MB mmproj-solarhive-e4b-BF16.gguf companion file packages this base vision encoder + audio encoder for llama-server --mmproj.
Converged Loss 0.9218
Training Time 420 seconds on RTX PRO 6000 Blackwell
License MIT (SolarHive files in this repository) / Apache 2.0 (base model, per Google's Gemma 4 license page and Hugging Face model page)

Model Overview

SolarHive is an AI energy advisor for community solar microgrids. It helps suburban neighborhoods collectively optimize distributed solar generation and shared battery storage through natural language conversation, visual inspection, and live data integration.

This is the edge deployment artifact. The 5.34 GB Q4_K_M text GGUF pairs with the 992 MB multimodal projector and runs on a 16 GB laptop CPU with no GPU. Prompts stay on the device, but the tools still fetch live data from cloud APIs. Companion to SolarHive 26B A4B LoRA (cloud inference with full multimodal VQA) and SolarHive E4B Ollama (merged safetensors for transformers research).

Local inference. Running the GGUF locally keeps prompts on the device. Tool calls for weather, solar, grid and PVWatts data still go to cloud APIs.


Files in This Repository

File Size Use
solarhive-e4b-q4_k_m.gguf 5.34 GB Text + tool calling β€” standard Q4_K_M (Q6_K-PLE)
solarhive-e4b-q4_k_m-standard.gguf 5.34 GB Text + tool calling β€” same standard Q4_K_M (Q6_K-PLE), explicit -standard name
mmproj-solarhive-e4b-BF16.gguf 992 MB Vision (SigLIP, 658 tensors) + audio (Conformer, 751 tensors) projector β€” pairs with the text GGUF
Modelfile β€” Ollama recipe pointing at solarhive-e4b-q4_k_m.gguf
Modelfile.standard β€” Ollama recipe pointing at solarhive-e4b-q4_k_m-standard.gguf

Quantization

The repo ships the SolarHive E4B fine-tune as a standard llama.cpp Q4_K_M GGUF (5.34 GB). The Q4_K_M mixed-precision recipe assigns Q4_K to most tensors and Q6_K to the largest / most sensitive ones β€” including the PLE tensor per_layer_token_embd.weight [10752, 262144] (2.82 B params, the largest single tensor in the model). The quant was produced on a high-RAM cloud notebook from the merged safetensors.

Two interchangeable copies ship so you can ollama create against a ready-made Modelfile without editing paths:

File Recipe Modelfile
solarhive-e4b-q4_k_m.gguf standard Q4_K_M (Q6_K-PLE) Modelfile
solarhive-e4b-q4_k_m-standard.gguf standard Q4_K_M (Q6_K-PLE) Modelfile.standard

Both files are the same Q4_K_M recipe and behave identically at inference time (β‰₯7 GB RAM at 4K context).

Quantizing on 16 GB RAM (reproducibility recipe)

The standard Q4_K_M conversion of the Q6_K PLE tensor needs a ~10.7 GB float32 intermediate buffer β€” which OOMs llama-quantize on 16 GB hardware (bad allocation at tensor 4/720). To reproduce the quant on laptop-class hardware, override that one tensor to Q4_0:

llama-quantize --tensor-type per_layer_token_embd.weight=q4_0 \
  solarhive-e4b-bf16.gguf solarhive-e4b-q4_k_m.gguf Q4_K_M

This bypasses the buffer and yields a smaller (~4.6 GB) GGUF. This variant was tried in development. No quality results for it are committed. The artifacts shipped in this repo use the standard recipe.

One mmproj, one text GGUF

The mmproj-solarhive-e4b-BF16.gguf companion is independent of the text model's quantization. llama-quantize operates exclusively on the text tower β€” vision and audio tensors were split out into the mmproj file during convert_hf_to_gguf.py --mmproj. At inference time, llama-server --model X --mmproj Y pairs them on demand. One mmproj serves any current or future text variant (Q4/Q5/Q6/Q8) with no per-variant duplication.


Inference Path Selection

This repo ships alongside solarhive_inference_e4b_gguf_ollama.py (in the GitHub repo), which implements two inference approaches against Ollama. The recommended demo path is the /api/generate raw mode + manual prompt builder approach. Both are documented for transparency and reproducibility.

Path A β€” OpenAI-compatible /api/chat + Ollama's built-in gemma4.go parser

Definition. Send messages and OpenAI-style tool schemas to Ollama's standard /api/chat endpoint. Ollama internally renders the chat template, parses model output, and extracts tool calls via its dedicated Gemma 4 parser (gemma4.go). The simplest path β€” closest to "drop-in replacement" for any OpenAI-compatible client.

import requests, json
resp = requests.post("http://127.0.0.1:11434/api/chat", json={
    "model": "solarhive",
    "messages": [{"role": "user", "content": "What's the current battery state?"}],
    "tools": [{"type": "function", "function": {...}}],
    "stream": False,
    "options": {"temperature": 1.0, "top_p": 0.95, "top_k": 64, "num_ctx": 4096},
})

Rationale. This is the path most developers will reach for first. OpenAI-compatible JSON, no manual prompt construction, leverages Ollama's first-party Gemma 4 support.

Why Path A was not used for the reported score. Ollama 0.21.0's gemma4.go:306 parser detects the model's native call:fn{} output but rejects it because the arguments use bare unquoted keys and <|"|> string delimiters (the Gemma 4 native format the model was fine-tuned on) instead of strict JSON. Server log evidence:

level=WARN source=gemma4.go:306 msg="gemma4 tool call parsing failed"
error="invalid character '\'' looking for beginning of object key string
       repair failed to produce valid JSON arguments"
content="call:get_status{battery:{level:65,pct:77,kwh:77,...,dispatched:<|\"|>OK<|\"|>},...}"

The repair logic also fails to recover valid JSON. Result: tool calls silently dropped, content empty. This path gave inconsistent results on the 10 prompt check, and no scores from it are committed. It is not a viable path for the demo without an upstream Ollama patch.

Path B β€” /api/generate raw mode + manual Gemma 4 prompt builder + native parser (recommended)

Definition. Bypass gemma4.go entirely. Build the Gemma 4 native prompt manually from chat_template.jinja, send via Ollama's /api/generate endpoint with raw: True (no template rendering by Ollama), parse the raw output ourselves using regex matching tokenizer_config.json's response_schema.

prompt = build_gemma4_prompt(messages, tools)   # byte-identical to apply_chat_template
resp = requests.post("http://127.0.0.1:11434/api/generate", json={
    "model": "solarhive",
    "prompt": prompt,
    "raw": True,
    "stream": False,
    "options": {"temperature": 1.0, "top_p": 0.95, "top_k": 64, "num_ctx": 4096},
})
content, tool_calls = parse_gemma4_output(resp.json()["response"])

Rationale. We control both endpoints of the prompt cycle:

  1. Prompt structure matches what the model was fine-tuned on, exactly. build_gemma4_prompt produces byte-identical output to apply_chat_template(messages, tools, enable_thinking=False, add_generation_prompt=True) β€” the same call used by the cloud 26B A4B path.
  2. Output parser matches the exact regex from tokenizer_config.json response_schema: r"<\|tool_call>(.*?)<tool_call\|>" for tool blocks, r"call:(\w+)\{(.*)\}$" for argument extraction.
  3. Tool responses fed back as <|turn>tool\n{json}<turn|> β€” matches what training pipeline rendered from the {role:"tool", content:"{...}"} OpenAI-style messages in the training data.

Result. In one local run: 5/5 tool routing, 5/5 Q&A recorded (not graded) and 2/3 on the When2Call probes. The results file is not committed.

Trade-off. ~30 lines of additional Python in solarhive_inference_e4b_gguf_ollama.py to build the prompt and parse the output. Worth it to bypass the upstream gemma4.go parser issue.

Recommended path

For the SolarHive Ollama + llama.cpp demo we use Path B (/api/generate raw mode + manual prompt builder). The solarhive_inference_e4b_gguf_ollama.py script in the GitHub repo provides a complete reference implementation of build_gemma4_prompt and parse_gemma4_output. Set OLLAMA_MODEL=solarhive or OLLAMA_MODEL=solarhive-standard (same Q4_K_M quant, two Modelfiles) and run the script. Both Modelfiles point at the same Q4_K_M quant.

For multimodal use cases (vision, audio), use llama-server --mmproj from llama.cpp directly β€” see "How to Use" below. Ollama 0.21.0's Modelfile syntax for Gemma 4 multimodal projector declaration is still evolving; first-class Ollama vision support for Gemma 4 will arrive in a future release.

Path C β€” Interactive chat via Unsloth Studio

For community users who want to chat with the SolarHive E4B GGUF without writing a Modelfile or running the HTTP harness by hand, Unsloth Studio provides an open-source no-code local web UI:

# macOS / Linux / WSL
curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio -H 0.0.0.0 -p 8888

# Windows PowerShell
irm https://unsloth.ai/install.ps1 | iex
unsloth studio -H 0.0.0.0 -p 8888

Studio supports running GGUF + safetensor models locally with self-healing tool calling and code execution, model arena (side-by-side comparison), and saves to GGUF / 16-bit safetensor formats. Use Studio for interactive exploration; use Path B (/api/generate raw mode + solarhive_inference_e4b_gguf_ollama.py) for reproducible evaluation. Studio does not expose the per-question evaluation harness needed to reproduce the SolarHive score.


Benchmark Results (one local run, results file not committed)

How to read these scores. The Q&A half of each 8/8, 9/10 and 10/10 score is not graded. The benchmark script records 5 of 5 Q&A once it generates five answers. Only tool routing and When2Call are scored by code. The 8/8 score comes from the earlier check at the end of the April 2026 fine tune notebook (notebook 2: 5 Q&A recorded, not graded, plus 3 tool prompts). The 9/10 and 10/10 scores come from the final May 2026 inference run (notebook 3: 5 Q&A recorded, not graded, plus 5 tool prompts and 3 When2Call probes). The two checks use different tool prompts. The prompts are project prompts, not a public benchmark, and they are not held out. Close paraphrases of the three When2Call probes appear in the training data, and the When2Call matcher is a lenient keyword check. Each score comes from a single run at temperature 1.0. The archived notebooks are in archive/final_run/. The GGUF result below comes from a separate local run, not from notebook 2 or notebook 3, and its results file is not committed.

This GGUF ran the 10 prompt check (5 domain Q&A with no tool expected, plus 5 tool prompts) and the 3 When2Call probes once, through the /api/generate raw mode + manual Gemma 4 prompt builder path (Path B above), on Ollama 0.21.0, CPU-only Microsoft Surface Pro 8 (Intel i5-1135G7 @ 2.4 GHz, 4 cores, 16 GB RAM, Intel Iris Xe unused), with the 5.34 GB GGUF + Ollama blob cache stored on an external USB drive. The harness writes a results file, but the file from this run is not committed to the GitHub repository.

Domain Q&A (5/5 recorded, not graded)

Question Expected behavior Result
Solar production when humidity exceeds 80%? Direct answer, no tool call Recorded
At what battery SOC should we stop exporting to the grid? Direct answer, no tool call Recorded
Home #3 underperforming 22%: diagnostic checklist? Direct answer, no tool call Recorded
Winter snow on panels: prioritize actions? Direct answer, no tool call Recorded
Grid frequency 59.8 Hz: microgrid implications? Direct answer, no tool call Recorded

Tool Calling (5/5)

Question Expected tool Result
What's the current battery state? get_battery_state βœ… fired + synthesized
Current weather and how does it affect solar production? get_weather βœ… fired + synthesized
What are general maintenance tips for panels? None (no tool needed) βœ… correctly no tool call
What is the weather expected to be like this week? get_weather βœ… fired + synthesized
How should we plan energy consumption and storage given the weather forecast? get_weather (+ get_grid_status) βœ… fired + synthesized

Prompt mismatch to resolve. solarhive_inference_e4b_gguf_ollama.py reads TOOL_BENCHMARK_QS from solarhive_inference.py. In the cloud run (notebook 3) those five prompts end with "What's the grid pricing right now and what's the renewable mix?" and the multi city prompt "Compare today's irradiance forecast across Ann Arbor, Phoenix, and Seattle." The last two rows above list different prompts. Without the results file, this card cannot confirm which prompts ran.

When2Call probes (2/3 in one local run)

The probe categories come from Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025). When2Call: When (not) to Call Tools. arXiv:2504.18851. The paper links missing follow up and unable to answer training data to higher tool hallucination. The SolarHive probes are not held out: close paraphrases of all three appear in the training data (solarhive_datagen.py lines 2717, 5079 and 5107), and the matcher is a lenient keyword check (solarhive_inference.py lines 1232 to 1252).

Category Question Expected behavior Result on this GGUF
(b) Well specified, in scope "What's the current grid rate?" Call get_grid_status PASS: called get_grid_status
(c) Under specified "How much will a 10 kW array produce today?" Follow up question that asks for the location PASS: asked for current weather conditions
(d) Out of scope "What's the current air quality index in Ann Arbor?" Refusal + redirect, with no tool call FAIL: called get_weather. The cloud E4B variants passed this probe and failed (c) instead

Headline: 2/3 nominal, the same total as the cloud E4B LoRA and E4B merged variants, but with a different failed probe. The cloud E4B variants failed probe (c), and this GGUF failed probe (d). The GGUF harness comment also records a strict score of 1 of 3 for the cloud E4B merged variant (solarhive_inference_e4b_gguf_ollama.py line 1561). Three probes and one run cannot show that GGUF quantization is lossless, and they cannot show that the A4B and E4B gap comes from model size.

End-to-end agentic loop probe

A multi-tool community-energy-audit query runs through the full agentic loop (extract, execute, feed back, max 3 rounds), mirroring the cloud notebook's Β§13g cell. Headline trace from the original card:

"Full community energy audit β€” check current weather, solar production, battery state, and grid pricing. Give a 3-sentence status report."

  • Rounds completed: 2
  • Tools executed: get_weather, get_solar_production, get_battery_state, get_grid_status (4 of 5 tools called in parallel in round 1)
  • Final answer (excerpt): "Community Energy Status Report (Midday): Partly cloudy with 30% cloud cover and 72Β°F. Array is producing 56% of capacity at 40.4 kW. Battery is at 72% SOC and actively charging from surplus. Grid pricing is Peak rates ($0.28/kWh). Status: Excellent. Maximize self-consumption by running heavy loads now…"

This is one trace. It shows that the loop can combine four tool results in a reply. It does not measure answer quality.

Multi-Variant Deployment Validation

The five cloud variants were measured once on Colab Pro G4 (NVIDIA RTX PRO 6000 Blackwell) in notebook 3. This GGUF variant was measured once on the CPU-only Surface Pro 8 described above.

Variant Q&A (recorded, not graded) Tool W2C Total Backend Hardware Source
a4b_lora (baseline) 5/5 4/5 3/3 9/10 transformers + Unsloth Colab Pro G4 GPU Notebook 3, cells 31 and 33
e4b_lora 5/5 5/5 2/3 10/10 transformers + Unsloth Colab Pro G4 GPU Notebook 3, cell 39
e4b_merged 5/5 4/5 2/3 9/10 transformers BF16 Colab Pro G4 GPU Notebook 3, cell 41
a4b_merged 5/5 4/5 3/3 9/10 transformers BF16 Colab Pro G4 GPU Notebook 3, cell 45
a4b_nf4 5/5 4/5 3/3 9/10 transformers NF4 (BnB) Colab Pro G4 GPU Notebook 3, cell 47
e4b_gguf (this artifact) 5/5 5/5 2/3 10/10 Ollama HTTP raw + manual prompt builder CPU-only Surface Pro 8 (i5-1135G7, 16 GB RAM) Local harness run; results file not committed

Observations from one run:

  1. This GGUF on a CPU only laptop matched the cloud E4B LoRA on the tool routing column (5 of 5). The cloud GPU was an RTX PRO 6000, not an A100.
  2. The GGUF and the cloud E4B variants scored 2 of 3 on When2Call, but they failed different probes: the cloud E4B variants failed (c), and the GGUF failed (d).
  3. Three probes and one run are too few to call any step lossless or to attribute the A4B and E4B gap to model size.

Reproduce locally:

$env:OLLAMA_HOST  = 'http://localhost:11434'
$env:OLLAMA_MODEL = 'solarhive'
python -m pytest solarhive_inference_e4b_gguf_ollama.py -v --tb=short

The evaluation run auto-writes a markdown summary to archive/ollama_local_e4b_gguf_results_YYYYMMDD_HHMMSS.md with the full per-question / per-probe trace. The summary from the run reported on this card is not committed.


How to Use

Path 1: Text + tool calling via Ollama (demo path)

# Download the text GGUF + Modelfile + LICENSE
hf download Truthseeker87/solarhive-e4b-gguf \
  solarhive-e4b-q4_k_m.gguf Modelfile LICENSE \
  --local-dir ./solarhive-gguf

cd ./solarhive-gguf

# Register with Ollama (Modelfile uses ./solarhive-e4b-q4_k_m.gguf relative path)
ollama create solarhive -f Modelfile

# Quick sanity check
ollama run solarhive "What is the current solar production for our community?"

# Full 10 prompt + When2Call check, using /api/generate raw mode + manual prompt builder
git clone https://github.com/youshen-lim/the-gemma4-good-hackathon-solarhive
cd the-gemma4-good-hackathon-solarhive
$env:OLLAMA_MODEL = 'solarhive'
python -m pytest solarhive_inference_e4b_gguf_ollama.py -v --tb=short

To use the -standard-named copy instead (same Q4_K_M quant), swap solarhive-e4b-q4_k_m.gguf and Modelfile for solarhive-e4b-q4_k_m-standard.gguf and Modelfile.standard, then ollama create solarhive-standard -f Modelfile.standard and OLLAMA_MODEL=solarhive-standard.

Path 2: Text + image + audio via llama.cpp llama-server

# Download a text variant + the mmproj companion
hf download Truthseeker87/solarhive-e4b-gguf \
  solarhive-e4b-q4_k_m.gguf mmproj-solarhive-e4b-BF16.gguf \
  --local-dir ./solarhive-gguf

# Start llama-server with the mmproj
llama-server \
  --model   ./solarhive-gguf/solarhive-e4b-q4_k_m.gguf \
  --mmproj  ./solarhive-gguf/mmproj-solarhive-e4b-BF16.gguf \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --port 8080

# POST chat-completion requests with image_url to http://localhost:8080/v1/chat/completions

The mmproj bundles both vision (SigLIP, sky photos and panel inspection) and audio (Conformer, voice queries up to 30 s) β€” same recipe serves both modalities.

Path 3 (planned, not tested): Jetson Orin Nano Super (llama.cpp + CUDA)

The Jetson Orin Nano Super Developer Kit (7 to 25 W power envelope per NVIDIA; see the vendor page for current specifications and pricing) is a planned target for this GGUF. This project has not run the GGUF on a Jetson. The commands below follow NVIDIA's Gemma 4 Jetson recipe and are untested with this GGUF:

# On the Jetson (Ubuntu via JetPack SDK)
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="87" \
  -DGGML_NATIVE=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j4

# Download the same GGUFs as Path 1 / Path 2
hf download Truthseeker87/solarhive-e4b-gguf \
  solarhive-e4b-q4_k_m.gguf mmproj-solarhive-e4b-BF16.gguf \
  --local-dir ~/models

# Run with full GPU offload (-ngl 99) for maximum throughput
./build/bin/llama-server \
  --model   ~/models/solarhive-e4b-q4_k_m.gguf \
  --mmproj  ~/models/mmproj-solarhive-e4b-BF16.gguf \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --port 8080 --host 0.0.0.0 \
  -ngl 99 --flash-attn on \
  --no-mmproj-offload --jinja -np 1

Planned design, not tested: a low power hub such as a Jetson could serve the model to phones on the local network at http://<hub-ip>:8080. This project has not measured power draw, speed or quality on a Jetson.

Modelfile reference

Both Modelfiles use a relative path so the same file works regardless of where you cloned the repo:

FROM ./solarhive-e4b-q4_k_m.gguf

SYSTEM """You are SolarHive, an AI energy advisor for a community of 12 homes with rooftop solar and shared battery storage in Ann Arbor, Michigan. Use the available tools to get real-time data before answering. Be specific, reference actual data, and keep responses concise (3-5 sentences)."""

PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 64
PARAMETER num_ctx 4096

Core Capabilities

1. Multimodal Visual Question Answering (3 Modes)

Available via llama-server --mmproj (Path 2 above). Tensor inventory of mmproj-solarhive-e4b-BF16.gguf: 1411 tensors total β€” 658 vision (SigLIP), 751 audio (Conformer), 2 multimodal projectors (mm.input_projection.weight, mm.a.input_projection.weight).

Mode Input Output
Sky Analysis Sky photograph Cloud coverage %, production forecast, storage recommendation
Panel Inspection Panel photograph Dirt/damage/shading detection, efficiency impact estimate
Neighborhood Assessment Aerial/satellite image Panel inventory, expansion priorities, shading analysis

2. Native Function Calling (5 Tools β€” all 3 keyed APIs wired)

Available via either Path 1 (Ollama) or Path 2 (llama-server). Tool schemas and reference implementations are in solarhive_inference_e4b_gguf_ollama.py (project root). The Β§13g cell of solarhive_inference.py runs an end-to-end agentic-loop probe via Ollama HTTP raw mode using these same 5 tools.

Tool API Returns
get_weather(location) OpenWeatherMap (OWM_API_KEY) Temperature, clouds %, wind, humidity, sunrise/sunset
get_solar_production(clouds_pct, temp_f) Open-Meteo GHI (keyless) Production kW, efficiency %, GHI W/mΒ², temp derating
get_battery_state() Community BMS (simulated) State of charge, capacity, charging status
get_grid_status() EIA Open Data (EIA_API_KEY) Pricing period, rate/kWh, renewable %, CO2 intensity
get_nrel_pvwatts_baseline() NREL PVWatts v8 (NREL_API_KEY) Annual + current-month typical kWh + avg kW for the 72 kW array

Tool results feed back as a 2-message sequence matching the training distribution: {"role": "assistant", "tool_calls": [...]} then {"role": "tool", "name": "<fn>", "content": json.dumps(result)} per call. The _build_gemma4_prompt() helper in solarhive_inference_e4b_gguf_ollama.py renders this format byte-identically β€” same as solarhive_datagen.py (training-data generation) and solarhive_finetune.py (SFT preprocessing). Inference matches training distribution exactly.

3. Selective Tool Reasoning

The model decides when to call tools β€” not blindly invoking all of them. Validated by the 5/5 tool-calling sub-check above:

"What time does peak pricing start?"
β†’ Calls: get_grid_status() only

"Is today's production above typical for January?"
β†’ Calls: get_solar_production() + get_nrel_pvwatts_baseline()

"Should I run my pool heater now?"
β†’ Calls: get_weather() + get_solar_production() + get_battery_state() + get_grid_status()

"What are general maintenance tips for panels?"
β†’ Calls: none (answers from training knowledge)

4. Inference-time When2Call Validation (solarhive_inference.py Β§11b)

Three project probes cover 3 of the 4 categories in the When2Call taxonomy. The probe categories come from Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025). When2Call: When (not) to Call Tools. arXiv:2504.18851. The paper links missing follow up and unable to answer training data to higher tool hallucination. The SolarHive probes are not held out: close paraphrases of all three appear in the training data (solarhive_datagen.py lines 2717, 5079 and 5107), and the matcher is a lenient keyword check (solarhive_inference.py lines 1232 to 1252).

  • (b) "What's the current grid rate?" β†’ expect a get_grid_status call (well specified, in scope)
  • (c) "How much will a 10 kW array produce today?" β†’ expect a follow up question that asks for the location
  • (d) "What's the current air quality index in Ann Arbor?" β†’ expect a refusal and a redirect, with no tool call

The training corpus includes the _UNABLE_TO_ANSWER and _FOLLOW_UP_QUESTIONS categories from solarhive_datagen.py. In the May 2026 cloud run, the A4B variants scored 3/3. The cloud E4B variants scored 2/3 and failed (c). This GGUF scored 2/3 in a separate local run and failed (d) (see Benchmark Results above).


Training Details

Parameter Value
Method LoRA via Unsloth FastVisionModel (BF16, RTX PRO 6000 96 GB)
LoRA rank 16
LoRA alpha 16
LoRA dropout 0
Target modules All linear layers
Learning rate 2e-4
Optimizer AdamW 8-bit
Warmup steps 5
Epochs 3
Max sequence length 2048
Precision BF16
Seed 3407
Trainable parameters 41.2 M / 8.0 B (0.51%)

Training Data β€” 1,727 Examples

Same canonical training corpus as the 26B A4B model β€” solarhive-community-solar-multimodal, 1,727 rows:

  • 413 hand-crafted examples spanning 15+ US cities and 9 energy domains (sky conditions, battery management, panel health, consumption optimization, community/grid coordination, emergency resilience, seasonal planning, multi-step reasoning, alternative storage)
  • ~1,117 API-grounded examples from live Open-Meteo (GHI/DNI/DHI, low/mid/high cloud cover), PVWatts, OpenWeatherMap, and EIA APIs β€” every numeric claim traces to a real API response, joined on (location, hourly timestamp) for cross-source coherence
  • 183 tool-calling examples following the When2Call taxonomy β€” 106 should-call, 53 should-not-call, 10 unable-to-answer, 6 follow-up clarification, 8 failure-recovery
  • 14 image-grounded Q&A turns from 7 manually-labeled Ann Arbor sky photographs, paired with the same temperature-derated GHI formula used in text rows (the original formula, since fixed in code; see Limitations)

See the SolarHive Dataset for full documentation.

Fine-tuning is text-only on the multimodal-capable corpus (image rows skipped at the data-prep layer). VQA at inference uses the base Gemma 4 E4B model's pretrained vision encoder (~150M params per the official model card). Our LoRA targets only the language-model linear layers (target=all-linear); the vision tower is unmodified, matching the Vertex AI Gemma 4 SFT recipe documented in the Hugging Face blog, which explicitly freezes both vision and audio towers during text-focused fine-tuning. The 992 MB mmproj-solarhive-e4b-BF16.gguf companion file packages this base vision encoder (658 SigLIP tensors) plus the base audio encoder (751 Conformer tensors) for llama-server --mmproj, giving the deployed GGUF full multimodal capability.

Training Loss

Metric Value
Converged loss (last 20 steps) 0.9218
Final step loss 0.0635
Minimum loss 0.0635
Total steps 324
Training time 420 seconds

Canonical metric: the bolded Converged loss (last 20 steps) is the only smoothed convergence indicator. Final step and Minimum are single-batch point statistics β€” mini-batch loss is noisy step-to-step, so one easy batch can drop a point estimate well below the rolling-average trend.

Hardware

  • GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB GDDR7)
  • Platform: Google Colab Pro (G4 VM)
  • Precision: BF16 (no quantization during training)

GGUF Production Pipeline

The shipped Q4_K_M GGUF was produced on a high-RAM cloud notebook:

Step Command Notes
Source merged safetensors from solarhive-e4b-ollama β€”
Convert to BF16 GGUF convert_hf_to_gguf.py --outtype bf16 ~30 min, 14 GB intermediate
Quantize text tower llama-quantize ... Q4_K_M ~23 sec, 5.34 GB (needs β‰₯30 GB RAM)
Reproducibility notebook solarhive_colab_quantize_e4b.ipynb (not in the public GitHub repository)

To reproduce on a 16 GB laptop instead of a high-RAM notebook, add --tensor-type per_layer_token_embd.weight=q4_0 (see "Quantizing on 16 GB RAM" above) β€” this bypasses the Q6_K-PLE intermediate buffer and yields a ~4.6 GB GGUF.

The mmproj companion is produced once via convert_hf_to_gguf.py --mmproj on the merged safetensors β€” independent of the text quantization step.


Community Model

Parameter Value
Location Ann Arbor, Michigan (42.2808Β°N, 83.7430Β°W)
Community size 12 homes
Total panel capacity 72 kW
Shared battery storage 100 kWh
Grid region MISO (Midcontinent Independent System Operator)

Runtime Performance

CPU-only inference on Intel i5-1135G7 @ 2.4 GHz, 4 cores, 16 GB RAM, Ollama 0.21.0. These timings come from the original card. No timing log is committed to the GitHub repository:

Phase Time Speed Notes
First query (cold) ~65 s ~2.2 tok/s Includes ~55–60 s model load
Warm advisory query ~10 s ~9–10 tok/s Single forward pass, no tools
Warm tool-calling loop 25–60 s ~9–10 tok/s 2–3 rounds with live API latency

GGUF blob ingestion (one-time, after ollama create): ~3–5 min for the 5.34 GB variant on a typical laptop SSD.


Technical Notes

  • PLE tensor override. The Q4_K_M mixed-precision strategy assigns Q6_K to per_layer_token_embd.weight [10752, 262144] (2.82 B params, the largest single tensor in the model). The float32 intermediate buffer for Q6_K conversion (~10.7 GB) OOMs llama-quantize on 16 GB RAM. The --tensor-type per_layer_token_embd.weight=q4_0 override eliminates the buffer. This variant was tried in development. No quality results for it are committed.
  • One mmproj for any text quant. The mmproj companion file is independent of text-model quantization β€” llama-quantize only sees the text tower. The same 992 MB mmproj pairs with the shipped Q4_K_M or any future Q5/Q6/Q8 quant.
  • Ollama /api/chat content-drop issue. Ollama 0.21.0's gemma4.go:306 parser detects but rejects fine-tuned Gemma 4's native tool-call format (bare keys + <|"|> delimiters). Use the /api/generate raw mode + manual prompt builder path (Path B in "Inference Path Selection" above) for tool-calling workloads.
  • Ollama multimodal support is evolving. Ollama 0.21.0's Modelfile syntax for Gemma 4 mmproj projector declaration is not yet first-class. For text + image / text + audio today, use llama-server --mmproj from llama.cpp b8863+ directly.
  • Sampling defaults. temperature=1.0, top_p=0.95, top_k=64 (Kaggle-recommended Gemma 4 defaults). Set in both Modelfile and Modelfile.standard.
  • Context window. Modelfiles set num_ctx=4096. The base architecture supports up to 128 K; raise num_ctx for longer multi-round agentic loops at the cost of more RAM at inference.
  • No Unsloth dependency at inference. Once quantized, the GGUF files run via stock Ollama or llama.cpp. Unsloth was used only during fine-tuning.

Limitations

  • Prototype tested on a single community model (12 homes, Ann Arbor, MI). Real-world deployment requires validation across diverse geographies and community sizes.
  • The OpenAI-compatible /api/chat path is not the demo path β€” see "Inference Path Selection" above for the gemma4.go content-drop reasoning. Use the /api/generate raw mode + manual prompt builder path for production inference.
  • Image and audio modalities require llama-server --mmproj; Ollama-native multimodal recipe is pending upstream Ollama support.
  • The model occasionally uses "60 kW" instead of the correct 72 kW community capacity in direct (no-tool) responses β€” base-model tendency, addressed by tool-calling path which queries actual capacity.
  • Tool responses depend on external API availability. Open-Meteo and EIA have rate limits; OpenWeatherMap free tier allows 1,000 calls/day.
  • The battery state is currently a deterministic simulator (get_battery_state() in solarhive_inference.py) β€” real deployment requires integration with actual battery management systems.
  • The shipped GGUF uses the standard Q4_K_M (Q6_K-PLE) recipe. It scored 5/5 on tool routing in one local run (Q&A not graded; results file not committed). The per_layer_token_embd.weight=q4_0 override for 16 GB hardware is a reproducibility recipe, not the shipped artifact. No quality results for it are committed.
  • Small, single run evaluation. Each benchmark ran once at temperature 1.0 on 5 tool prompts and 3 When2Call probes. The Q&A half of each score is recorded, not graded. The prompts are not held out.
  • VQA and audio were not scored.
  • Jetson is untested. Path 3 is a planned design.
  • Training data predates the November 20, 2026 code fixes. The dataset that trained this model was generated with the original temperature derating, which applied 0.4% loss per Β°F above 77Β°F to air temperature. Panel temperature coefficients are stated per Β°C and refer to cell temperature. The data generation code also called developer.nrel.gov, a host that stopped serving requests on May 29, 2026. As of November 20, 2026, the code on main in the GitHub repository (commit a89b751) applies 0.4% loss per Β°C above 25Β°C, with the original 0.75 floor, to a cell temperature estimated with the NOCT model from air temperature, GHI and wind, and it calls developer.nlr.gov. The published dataset, this model and every score on this card still come from the old formula and the old host. Dataset regeneration, a new fine tune and a rerun are pending. See Known Limitations in the GitHub README.

Future Iteration β€” Multi-Token Prediction (MTP) Drafters on Edge GGUF Runtimes

Not in the measured numbers above. Google announced Gemma 4 MTP drafters on May 5, 2026 (blog, overview, HF collection, Kaggle, @GoogleGemma), after this artifact's benchmark was captured. The numbers above reflect standard autoregressive decoding only. MTP integration is documented here as future iteration; no measured speedup is claimed in this release.

Theoretical foundation. Speculative decoding (Leviathan, Kalman & Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023, arXiv:2211.17192) accelerates generation without changing the output distribution under argmax decoding: a smaller drafter proposes Ξ³ candidate tokens, the target verifies all Ξ³ in a single parallel forward pass, accepted tokens are kept, and any rejection is resampled from a corrected distribution. The output distribution is preserved exactly regardless of drafter quality; only acceptance rate Ξ±, and therefore walltime speedup, varies.

Released drafter for E4B. google/gemma-4-E4B-it-assistant (~78.8 M params) is the canonical pair for google/gemma-4-E4B-it. Per the MTP overview, the drafter shares the input embedding table with the target and consumes the target's last-layer activations. Google reports up to 3Γ— decode speedup on the 26B-A4B configuration; per-variant E4B numbers were not enumerated in the announcement.

Runtime support is partial for GGUF deployments.

Runtime Listed by Google? Source
Ollama βœ… Tested-runtime list Google blog
llama.cpp ⚠️ Appears in docs runtime nav but not in the blog's tested-runtime list Docs nav
LiteRT-LM, MLX, Hugging Face Transformers, vLLM, SGLang βœ… Tested-runtime list Google blog

Implementation paths on this edge GGUF tier (post-hackathon):

  1. Drafter GGUF conversion. Google ships the drafter as HF safetensors. To use against this Q4_K_M target via Ollama or llama.cpp, the drafter weights would need conversion through convert_hf_to_gguf.py β€” feasible reuse of the same toolchain that produced this target's GGUF, but the conversion is not in the canonical SolarHive registry today.
  2. llama.cpp speculative decoding. llama-speculative and llama-server --draft-model support vanilla speculative decoding per the 2023 paper. Whether Gemma 4 MTP drafters' embedding-sharing + last-layer-activation conditioning architecture maps cleanly to llama.cpp's existing --draft-model plumbing is unverified β€” Google's docs list the runtime but the blog omits it from the tested set.
  3. Ollama paired drafter. Ollama is in Google's tested-runtime list; the exact CLI/API surface for drafter pairing is not yet documented in Ollama's public docs as of writing.

Planned measurement (post-hackathon). (a) Convert google/gemma-4-E4B-it-assistant β†’ Q4_K_M GGUF via convert_hf_to_gguf.py + llama-quantize. (b) Re-run the parity check with the drafter paired via llama.cpp's --draft-model flag. (c) Capture acceptance rate Ξ± + decode-tps + walltime. (d) Cross-check against the Ollama paired-drafter API once documented. Correctness is invariant per the 2023 speculative-sampling guarantee β€” only Ξ± varies under target Γ— drafter distribution mismatch.


Companion Repositories

Model Repository Purpose
SolarHive 26B A4B LoRA solarhive-26b-a4b-lora Cloud inference with full multimodal + function calling (LoRA adapters + Unsloth)
SolarHive 26B A4B NF4 solarhive-26b-a4b-nf4 Pre-quantized partial NF4 cloud model. 48.34 GB on disk; used 49.2 GB of GPU memory on one GPU in the May 2026 run
SolarHive E4B LoRA solarhive-e4b-lora E4B adapter weights (~200 MB) β€” apply over base via Unsloth
SolarHive E4B Safetensors solarhive-e4b-ollama Merged safetensors for transformers-native multimodal research use
SolarHive E4B GGUF This repo Edge deployment β€” 2 text quants + 1 mmproj for Ollama / llama.cpp
SolarHive Dataset solarhive-community-solar-multimodal 1,727 training examples (1,713 text + 14 image-grounded)
LiteRT-LM Python runtime demo (base model) solarhive_e4b_litert_v3.1.ipynb LiteRT Special Tech Track entry. It runs the upstream base litert-community/gemma-4-E4B-it-litert-lm .litertlm bundle (3.66 GB), not the SolarHive fine tune, on a Colab x86 CPU (notebook 5: 8.29 tok/s decode, 41.38 s median time to first token). The fine tune did not produce a LiteRT artifact. Current LiteRT Torch releases list Gemma 4 support, so a fine tuned bundle needs a new conversion attempt.
GitHub (source) the-gemma4-good-hackathon-solarhive Full source code, training notebooks, solarhive_inference.py (cloud), solarhive_inference_e4b_gguf_ollama.py (local laptop)

Corrections (November 21, 2026)

I corrected the following items on November 21, 2026 after I rechecked this card against the merged GitHub README (Corrections, November 20, 2026), the archived notebooks in archive/final_run/ and the Hugging Face file listings. The files in this repository other than this card are unchanged.

Item Previously stated Corrected to Cause
Domain Q&A metadata and scores Domain Q&A accuracy 1.0 in metadata; 5/5 Q&A presented as correct answers Removed from metadata. Q&A is recorded as 5/5 by the script, not graded. The script sets Q&A to the number of questions (solarhive_inference.py lines 1436 to 1897)
Score metadata 10 prompt total accuracy 1.0 and Q&A accuracy 1.0 Removed. Kept tool routing 1.0 (5 of 5), marked unverified. Q&A is not graded; results file not committed
Headline claims Joint best in class across all six variants; validated end to end 5 of 5 tool routing in one local run; results file not committed No committed results file; single run
Path A heading Why it does not score 10/10 Why Path A was not used for the reported score The reported 10/10 used Path B
E4B failed probe profile Same (b)+(c) PASS, (d) FAIL profile as the BF16 LoRA and cloud E4B merged Cloud E4B variants failed (c). This GGUF failed (d). Notebook 3, cells 39 and 41
Quantization claims Q4_K_M is lossless; preserves agentic reasoning quality; the gap is a model size signature Removed. One run on three probes and one trace. Single run
Cloud GPU A100 class cloud accelerator; 4 year old laptop RTX PRO 6000 Blackwell Notebook 3, cell 3
Tool prompts Same 10 prompts as the cloud checks Added a note: two listed tool prompts differ from TOOL_BENCHMARK_QS in the cloud run Notebook 3, cell 31; harness reads TOOL_BENCHMARK_QS
PLE override Validated quality safe; project scores prove it Tried in development. No quality results committed. No results file in the GitHub repository
Jetson path The GGUF runs CUDA accelerated on a $249 Jetson hub Planned target, not tested. Price removed. No Jetson run exists
Local inference No cloud dependency, no internet requirement Prompts stay on the device. Live data still comes from cloud APIs. The tools call cloud APIs
Held out wording "Held-out" questions and probes Not held out. Close paraphrases of the When2Call probes appear in the training data. solarhive_datagen.py lines 2717, 5079 and 5107
When2Call paper 9 to 67% tool hallucination rates on (c) and (d) in untrained models Removed. The paper links missing follow up and unable to answer data to higher hallucination. The range is not stated in the paper and is not per category
NF4 size and GPU fit For HF Spaces and 24 GB+ GPUs 48.34 GB on disk. 49.2 GB GPU memory on one GPU in the May 2026 run. Hugging Face file listing; notebook 3, cell 47
LiteRT companion row On device agentic loop; fine tuned bundle waits on an ai_edge_torch example module Base model run on a Colab x86 CPU. The fine tune did not produce a LiteRT artifact. Current LiteRT Torch releases list Gemma 4 support. Notebook 5 uses the base model; LiteRT Torch releases
Quantize notebook solarhive_colab_quantize_e4b.ipynb in the GitHub repository Marked as not in the public GitHub repository GitHub file tree, November 21, 2026
Training data formula and host Not mentioned Added a limitation: the data used 0.4% per Β°F on air temperature and the old NREL host. The code is fixed. Regeneration and retraining are pending. GitHub README Known Limitations, November 20, 2026
Base model license wording Gemma Terms (base model) Apache 2.0 (base model). The SolarHive files stay MIT. Live Google Gemma 4 Hugging Face model pages and license page, November 21, 2026

I rechecked the scores, sizes, probe results and file names against notebook 2 (2_solarhive_finetune_finalrun_Apr2026.ipynb), notebook 3 (3_solarhive_inference_finalrun_May2026.ipynb), solarhive_inference_e4b_gguf_ollama.py, the GitHub file tree and the Hugging Face API on November 21, 2026.


Citation

@misc{solarhive2026,
  title={SolarHive: AI-Powered Community Solar Energy Intelligence},
  author={Youshen Lim},
  year={2026},
  url={https://github.com/youshen-lim/the-gemma4-good-hackathon-solarhive},
  note={Gemma 4 Good Hackathon submission β€” Google DeepMind x Kaggle}
}

Links

Built with Gemma 4 in Ann Arbor, Michigan. April 2026.

Gemma is a trademark of Google LLC.

Downloads last month
40
GGUF
Model size
8B params
Architecture
gemma4
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train Truthseeker87/solarhive-e4b-gguf

Space using Truthseeker87/solarhive-e4b-gguf 1

Papers for Truthseeker87/solarhive-e4b-gguf