distributed-llm

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。

Distributed LLM Router: an OpenAI-compatible FastAPI gateway that intelligently routes requests across multiple vLLM nodes with load balancing, failover retries, backpressure, and real-time Prometheus/Grafana observability.

Run a model too big for one GPU in LM Studio across two PCs over LAN (distributed inference via llama.cpp RPC): two DLLs + one env var GGML_RPC_SERVERS, no LM Studio changes. Windows/Vulkan/AMD tested. Home-lab hack by Robert + Kod (AI).

LAnguage-Driven composition of APIs on Distributed Edge Devices

POLLEN: Prompt-based Orchestration of LLM-decomposed services over Edge Nodes

Run Qwen3.6 GGUF models locally on Pi using llama.cpp with automated runtime management and hardware-optimized inference.

Heterogeneous distributed AI execution with multi-role peers, exact MoE inference, locality-aware scheduling, replication and failover.