AI Bench Index

Inference engines

Local AI runtimes

Capabilities are factual support claims, not scores. First-party tok/s, when published, come from each runtime’s official benchmark tool — llama-bench, mlx_lm.benchmark, vllm bench serve, or sglang.bench_serving — not from a homemade microbenchmark. Open WebUI is a frontend, not an engine, so it is omitted.

Catalogue review recorded · individual claims have separate datesReferenceSourceMethodology

Documented means a project support claim, not our own test. Community and experimental claims remain separate. Apple support for vLLM is through the separately maintained vLLM-Metal plugin; consult the linked installation guide for its model and feature limits.

Local inference runtimes compared by OS, hardware, formats, and serving features
RuntimeDocumented focusWindowsLinuxmacOSNVIDIAAMDAppleCPUGGUFHF weightsMulti-GPUAPI
llama.cppPortable GGUF inference on CPU, GPU, and Apple SiliconDocumentedVerified claimVerified claimDocumentedDocumentedDocumentedVerified claimVerified claimunsupportedDocumentedDocumented
OllamaOne-command local model runner with a library and HTTP APIDocumentedVerified claimVerified claimDocumentedDocumentedDocumentedVerified claimVerified claimDocumentedDocumentedDocumented
LM StudioDesktop GUI for local GGUF/MLX models with a built-in serverDocumentedDocumentedVerified claimDocumentedcommunityVerified claimDocumentedVerified claimcommunitycommunityDocumented
MLX-LMApple Silicon unified-memory inference with MLXunsupportedunsupportedVerified claimunsupportedunsupportedVerified claimunsupportedunsupportedDocumentedunsupportedcommunity
vLLMHigh-throughput OpenAI-compatible serving with PagedAttentionexperimentalVerified claimDocumentedVerified claimDocumentedDocumentedexperimentalcommunityVerified claimVerified claimVerified claim
SGLangStructured, high-throughput serving with RadixAttentionunsupportedVerified claimDocumentedVerified claimDocumentedDocumentedunsupportedunsupportedVerified claimVerified claimVerified claim
More columns this way →
llama.cpp — when to use it

Documented focus: Portable GGUF inference on CPU, GPU, and Apple Silicon

When not to use: Do not use llama.cpp as a high-QPS multi-tenant OpenAI-compatible farm; vLLM or SGLang are the serving runtimes.

Noted version: current GitHub main, verified 2026-09-04

Install

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -S . -B build && cmake --build build --config Release

Official docs: https://github.com/ggml-org/llama.cpp

Verified 2026-09-04 · overhead 0.4 GB (labelled estimate)

Ollama — when to use it

Documented focus: One-command local model runner with a library and HTTP API

When not to use: Skip Ollama when you need custom GGUF flags, tensor parallel serving, or a non-library model file Ollama does not ship.

Noted version: current GitHub main, verified 2026-09-04

Install

curl -fsSL https://ollama.com/install.sh | sh

Official docs: https://github.com/ollama/ollama

Verified 2026-09-04 · overhead 0.7 GB (labelled estimate)

LM Studio — when to use it

Documented focus: Desktop GUI for local GGUF/MLX models with a built-in server

When not to use: Do not use LM Studio as a headless production inference cluster.

Noted version: docs.lmstudio.ai, verified 2026-09-04

Install

Download the installer from https://lmstudio.ai

Official docs: https://lmstudio.ai/docs

Verified 2026-09-04 · overhead 1 GB (labelled estimate)

MLX-LM — when to use it

Documented focus: Apple Silicon unified-memory inference with MLX

When not to use: MLX-LM does not run on NVIDIA or AMD discrete GPUs.

Noted version: ml-explore/mlx-lm, verified 2026-09-04

Install

pip install mlx-lm

Official docs: https://github.com/ml-explore/mlx-lm

Verified 2026-09-04 · overhead 0.5 GB (labelled estimate)

vLLM — when to use it

Documented focus: High-throughput OpenAI-compatible serving with PagedAttention

When not to use: vLLM is a PagedAttention OpenAI-compatible server. Do not treat stock `pip install vllm` as a Mac chat app — Apple Silicon is the separate vllm-metal plugin (macOS 15+, arm64 Python 3.12). For GGUF desktops use llama.cpp or Ollama.

Noted version: docs.vllm.ai and docs.vllm.ai/projects/vllm-metal, verified 2026-09-04

Install

pip install vllm

Official docs: https://docs.vllm.ai/en/latest/

Verified 2026-09-04 · overhead 1.8 GB (labelled estimate)

SGLang — when to use it

Documented focus: Structured, high-throughput serving with RadixAttention

When not to use: SGLang is a RadixAttention serving runtime for structured generation, not a one-click GGUF GUI. The documented Apple path is `SGLANG_USE_MLX=1` on macOS 14+ from the Apple Metal hardware page; MLX-LM remains the simpler laptop chat default.

Noted version: docs.sglang.ai hardware-platforms/apple_metal, verified 2026-09-04

Install

pip install sglang

Official docs: https://docs.sglang.ai/

Verified 2026-09-04 · overhead 1.8 GB (labelled estimate)