Local LLM & inference tools rankings
Local LLM & inference tools, ordered by GitHub stars · 19 projects
Sort
1
llama.cppggml-orgA C/C++ implementation for LLM and VLM inference with minimal setup and state-of-the-art performance, supporting multiple model formats with CLI, server, and built-in Web UI.C++ 131.7k 2026.10.192
OmniRoutediegosouzapwA free MIT AI gateway providing a unified endpoint to 290+ providers and 500+ models with quota-aware auto-fallback and token compression.TypeScript 72.8k 2026.09.303
llmfitAlexsJonesA command-line tool that recommends the best local LLMs for your hardware configuration, supporting multi-GPU, MoE architectures, and multiple quantization formats.Rust 37.7k 2026.10.084
freellmapitashfeenahmedA unified API gateway aggregating 34 free LLM providers and 635 free model endpoints, offering approximately 7.4 billion free tokens per month with smart routing, automatic failover, and encrypted key storage.TypeScript 37.4k 2026.10.195
omlxjundotAn LLM inference server optimized for Apple Silicon with continuous batching and SSD caching, managed directly from the macOS menu bar.Python 22.9k 2026.10.156
magpieyetoneMagpie is a locally running AI gateway that lets coding agents such as Claude Code, Codex, Gemini CLI and OpenCode use models like Kimi, DeepSeek, GLM and ChatGPT, with one-click menu bar switching and automatic failover to the next account when quota runs out.Go 16.8k 2026.10.197
higresshigress-groupA cloud-native AI gateway based on Istio and Envoy, extensible with Wasm plugins, providing unified LLM API and MCP API management with support for mainstream model providers.Go 9.5k 2026.10.048
semantic-routervllm-projectA programmable Mixture-of-Models router for heterogeneous LLM inference that evaluates request signals, user preferences, and application policies to select or compose the optimal model path for each request, improving quality, cost, latency, privacy, and safety.Go 6.0k 2026.09.279
lemonadelemonade-sdkA local AI server that gives you the same capabilities as cloud APIs, 100% free and private, running the latest models for chat, coding, speech, and image generation on your own NPU and GPU.C++ 5.9k 2026.10.1610
routerweave-osA model router for agentic systems that routes every prompt to the right model in <50ms, cutting costs 40-70% with just an endpoint change. Supports Anthropic, OpenAI, Gemini, and open-source models.Go 5.6k 2026.07.1311
lean-ctxyvgudeThe context intelligence layer for AI agents — a local Rust binary with 76 MCP tools and 30+ agents support that selects, shapes, reuses, and recovers project context, reducing token usage by 60–90%.Rust 3.9k 2026.10.1112
MTPLXyoussofalA native Mac app and CLI for Apple Silicon that doubles local LLM inference speed using Multi-Token Prediction (MTP), supporting models like Qwen 3.8 27B with no external drafter model.Python 2.5k 2026.10.0313
spark-vllm-dockereugrOptimized vLLM Docker configuration and startup scripts for NVIDIA DGX Spark, supporting single-node and multi-node clusters using Ray or native PyTorch distributed mode, with InfiniBand/RDMA (NCCL), high-performance model loading, and custom environment configuration.Python 2.3k 2026.10.0414
FastFlowLMROCmAn LLM runtime optimized for AMD Ryzen AI NPUs, supporting vision, audio, embedding, and MoE models, with no GPU required, ultra-lightweight (17 MB), and 10x more power-efficient.C++ 2.0k 2026.10.1515
mlx-serveddalcuA native LLM inference server optimized for Apple Silicon, supporting OpenAI and Anthropic-compatible APIs with no Python required, supporting both MLX and GGUF models, and including a macOS menu-bar app.Zig 1.9k 2026.10.1216
vigoliumvigoliumA high-fidelity vulnerability scanner fusing agentic AI with native speed, modularity, and precision, offering deterministic multi-phase scanning and AI-driven code audit modes.Go 1.1k 2026.10.1817
llama-cpp-desktopQiao-920Llama.cpp Desktop is a Windows desktop control panel for local llama.cpp servers, combining service startup, model configuration, log viewing, direct chat, and OpenAI-compatible client access in a single window, letting users manage local llama.cpp inference without the command line.JavaScript 385 2026.07.1318
llama.cpp-launcheryihuishouA lightweight llama.cpp GUI launcher built with Rust, providing visual configuration, RPC mode, preset management, real-time logs, and a multi-language interface to help users easily manage and run llama.cpp servers.Rust 133 2026.10.1919
dliazorak1103An AI-powered Docker log monitoring agent that uses LLMs to intelligently analyze container logs, detect anomalies, and provide contextual insights over time.Go 7 2026.10.02
llama.cppggml-orgA C/C++ implementation for LLM and VLM inference with minimal setup and state-of-the-art performance, supporting multiple model formats with CLI, server, and built-in Web UI.C++ 131.7k 2026.10.192OmniRoutediegosouzapwA free MIT AI gateway providing a unified endpoint to 290+ providers and 500+ models with quota-aware auto-fallback and token compression.TypeScript 72.8k 2026.09.303llmfitAlexsJonesA command-line tool that recommends the best local LLMs for your hardware configuration, supporting multi-GPU, MoE architectures, and multiple quantization formats.Rust 37.7k 2026.10.084freellmapitashfeenahmedA unified API gateway aggregating 34 free LLM providers and 635 free model endpoints, offering approximately 7.4 billion free tokens per month with smart routing, automatic failover, and encrypted key storage.TypeScript 37.4k 2026.10.195omlxjundotAn LLM inference server optimized for Apple Silicon with continuous batching and SSD caching, managed directly from the macOS menu bar.Python 22.9k 2026.10.156magpieyetoneMagpie is a locally running AI gateway that lets coding agents such as Claude Code, Codex, Gemini CLI and OpenCode use models like Kimi, DeepSeek, GLM and ChatGPT, with one-click menu bar switching and automatic failover to the next account when quota runs out.Go 16.8k 2026.10.197higresshigress-groupA cloud-native AI gateway based on Istio and Envoy, extensible with Wasm plugins, providing unified LLM API and MCP API management with support for mainstream model providers.Go 9.5k 2026.10.048semantic-routervllm-projectA programmable Mixture-of-Models router for heterogeneous LLM inference that evaluates request signals, user preferences, and application policies to select or compose the optimal model path for each request, improving quality, cost, latency, privacy, and safety.Go 6.0k 2026.09.279lemonadelemonade-sdkA local AI server that gives you the same capabilities as cloud APIs, 100% free and private, running the latest models for chat, coding, speech, and image generation on your own NPU and GPU.C++ 5.9k 2026.10.1610routerweave-osA model router for agentic systems that routes every prompt to the right model in <50ms, cutting costs 40-70% with just an endpoint change. Supports Anthropic, OpenAI, Gemini, and open-source models.Go 5.6k 2026.07.1311lean-ctxyvgudeThe context intelligence layer for AI agents — a local Rust binary with 76 MCP tools and 30+ agents support that selects, shapes, reuses, and recovers project context, reducing token usage by 60–90%.Rust 3.9k 2026.10.1112MTPLXyoussofalA native Mac app and CLI for Apple Silicon that doubles local LLM inference speed using Multi-Token Prediction (MTP), supporting models like Qwen 3.8 27B with no external drafter model.Python 2.5k 2026.10.0313spark-vllm-dockereugrOptimized vLLM Docker configuration and startup scripts for NVIDIA DGX Spark, supporting single-node and multi-node clusters using Ray or native PyTorch distributed mode, with InfiniBand/RDMA (NCCL), high-performance model loading, and custom environment configuration.Python 2.3k 2026.10.0414FastFlowLMROCmAn LLM runtime optimized for AMD Ryzen AI NPUs, supporting vision, audio, embedding, and MoE models, with no GPU required, ultra-lightweight (17 MB), and 10x more power-efficient.C++ 2.0k 2026.10.1515mlx-serveddalcuA native LLM inference server optimized for Apple Silicon, supporting OpenAI and Anthropic-compatible APIs with no Python required, supporting both MLX and GGUF models, and including a macOS menu-bar app.Zig 1.9k 2026.10.1216vigoliumvigoliumA high-fidelity vulnerability scanner fusing agentic AI with native speed, modularity, and precision, offering deterministic multi-phase scanning and AI-driven code audit modes.Go 1.1k 2026.10.1817llama-cpp-desktopQiao-920Llama.cpp Desktop is a Windows desktop control panel for local llama.cpp servers, combining service startup, model configuration, log viewing, direct chat, and OpenAI-compatible client access in a single window, letting users manage local llama.cpp inference without the command line.JavaScript 385 2026.07.1318llama.cpp-launcheryihuishouA lightweight llama.cpp GUI launcher built with Rust, providing visual configuration, RPC mode, preset management, real-time logs, and a multi-language interface to help users easily manage and run llama.cpp servers.Rust 133 2026.10.1919dliazorak1103An AI-powered Docker log monitoring agent that uses LLMs to intelligently analyze container logs, detect anomalies, and provide contextual insights over time.Go 7 2026.10.02