affectively-ai/llama-3.1-70b-instruct-gguf

Llama 3.1 70B Instruct (GGUF, Q4_K_M)

Production-ready GGUF quantization of meta-llama/Llama-3.1-70B-Instruct for distributed text generation and conversation — powered by the Aether edge inference runtime.

Highlights

• 70B parameters — Llama 3.1 70B instruction-tuned. Frontier-class reasoning distributed across multiple Aether nodes.
• ~42 GB Q4_K_M quantized — optimized for distributed edge inference
• LLaMA architecture — proven, stable, well-tested
• Aether runtime compatible — layer-sharded across distributed nodes via Edgework.ai

Model Details

Property	Value
Base model	meta-llama/Llama-3.1-70B-Instruct
Parameters	70B
Architecture	LLaMA
Quantization	Q4_K_M
Format	GGUF
Size	~42 GB
License	llama3.1

Usage

With llama.cpp

./llama-cli -m llama-3.1-70b-instruct-q4_k_m.gguf -p "Your prompt here" -n 256

With Aether (Distributed Inference)

This model is deployed across the Aether distributed inference network. Weights are layer-sharded and distributed across multiple edge nodes for parallel inference.

Deployment Architecture

This model runs on the Aether distributed inference runtime — our custom engine that shards model layers across multiple nodes for parallel execution:

1. Coordinator receives requests and manages token generation
2. Layer nodes each hold a subset of model layers
3. Hidden states flow between nodes via gRPC
4. Zero cold start via warm pool scheduling

Deployed via Edgework.ai — bringing fast, cheap, and private inference as close to the user as possible.

About

Published by AFFECTIVELY · Managed by @buley

We quantize and publish production-ready models for distributed edge inference via the Aether runtime. Every release is tested for correctness and stability before publication.

• All models · GitHub · Edgework.ai