File size: 1,369 Bytes
c0822b6 9991e0c c0822b6 9991e0c c0822b6 9991e0c 097dfdf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | ---
title: README
emoji: 🐨
colorFrom: red
colorTo: yellow
sdk: static
pinned: false
thumbnail: >-
https://cdn-uploads.huggingface.co/production/uploads/60fa66a3c4c6bd8c56ee541f/FEldjz5JpuoMy8dauKA2M.png
short_description: pool compute for huge model inference
---
mesh-llm turns spare compute into a peer-to-peer inference cloud for open models.
mesh-llm pools GPUs across macOS and Linux machines so teams, researchers, and agents can run local or open-weight models through one OpenAI-compatible endpoint. It can serve a model on one node, distribute large models across nearby peers, route requests to specialized models, and let agents coordinate through mesh gossip.
What it is for
* Share spare GPU capacity across trusted machines.
* Run open models locally without a centralized inference provider.
* Serve an OpenAI-compatible API at http://localhost:9337/v1.
* Route requests across multiple nodes, models, and capabilities.
* Experiment with distributed inference, MoE expert sharding, and agent collaboration.
see: https://docs.anarchai.org/
and: https://github.com/mesh-LLM/
Mesh uses a pipelined/network aware distributed inference approach built on llama.cpp called "skippy" - https://github.com/Mesh-LLM/hf-mesh-skippy-splitter contains current code which prepares models so layers can be efficiently JIT downloaded for participating nodes. |