Browse & filter

Local LLM runner and model library with simple CLI and API for workstation inference.

Plain C/C++ inference for LLaMA-class models with broad community backends.

High-throughput LLM serving with PagedAttention, continuous batching, and OpenAI-compatible APIs for GPU clusters.

Structured generation language for fast serving: RadixAttention, constrained decoding, and multi-turn batching for frontier-class workloads.

Apple MLX-based LLM inference and training on Apple silicon: efficient Metal-backed transformers and examples for local chat models.

Single-file distributable LLM weights + llama.cpp runtime: run large models from one executable with broad OS CPU/GPU support.

Memory-efficient CUDA inference kernels for quantized Llama-class models—popular in consumer GPU chat UIs.

NVIDIA TensorRT–based library for optimized LLM inference on GPUs with multi-GPU and speculative decoding features.

Cross-platform inference accelerator for ONNX models: CPU, GPU, and mobile execution providers with graph optimizations.

Intel toolkit to optimize and deploy deep learning on Intel CPUs, GPUs, and NPUs with model conversion and runtime APIs.

Multi-framework inference server for TensorRT, ONNX, PyTorch, Python backends—dynamic batching, ensembles, and GPU sharing.

CTranslate2 reimplementation of Whisper for faster CPU/GPU inference with lower memory use than reference PyTorch.

Alibaba’s lightweight inference engine for mobile and edge—used for on-device LLMs and classic CV models with aggressive optimization.

Alibaba’s high-performance LLM inference engine (CUDA-focused) for production serving of diverse decoder architectures.

NVIDIA research-oriented toolkit for LLM KV-cache compression to stretch context within fixed VRAM budgets.

Flexible, high-performance serving system for TensorFlow (and related) models with versioning, batching, and gRPC/REST.

TypeScript/JavaScript libraries to call Inference API, manage Hub assets, and build browser or Node AI features.

Rust-based high-throughput server for sentence-transformers–class embedding models with GPU/CPU backends.