lastmile-ai/mcp-agentjust droppedNEW
dottxt-ai/outlinesjust droppedNEW
getomni-ai/zeroxjust droppedPIPETrendingRepoLIVE · 30 mention sourcesNEWlastmile-ai/mcp-agentjust droppedNEWdottxt-ai/outlinesjust droppedNEWgetomni-ai/zeroxjust droppedPIPETrendingRepoLIVE · 30 mention sourcesInference is the process of running a trained model to generate output — for an LLM, turning a prompt into tokens — and the stage where latency, throughput, and cost are decided.
Inference is the serving side of a model, distinct from training. For LLMs it is dominated by how fast tokens are generated and how many requests can be batched together; techniques like KV caching, continuous batching, and streaming responses are what make it usable at scale.
Open-source inference engines optimise throughput and memory so capable models run on the hardware you have. Quantization is often paired with inference to fit a model onto a single GPU or a laptop.
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replacement. Works with Claude Code, Cursor, Aider.
High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.
A list of free LLM inference resources accessible via API.
⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.