TensorFlow Serving
Flexible, high-performance serving system for TensorFlow (and related) models with versioning, batching, and gRPC/REST.
Why it is included
Long-running Apache-2.0 project featured in TAAFT’s #machine-learning repository list alongside core TensorFlow.
Best for
Teams standardized on TensorFlow exports who need battle-tested model servers.
Strengths
- Model versioning
- Batching
- Mature ops patterns
Limitations
- TensorFlow-centric vs PyTorch-first shops
Good alternatives
TorchServe · Triton Inference Server · BentoML
Related tools
AI & Machine Learning
TensorFlow
End-to-end platform for machine learning and deployment.
AI & Machine Learning
NVIDIA Triton Inference Server
Multi-framework inference server for TensorRT, ONNX, PyTorch, Python backends—dynamic batching, ensembles, and GPU sharing.
AI & Machine Learning
rtp-llm
Alibaba’s high-performance LLM inference engine (CUDA-focused) for production serving of diverse decoder architectures.
AI & Machine Learning
vLLM
High-throughput LLM serving with PagedAttention, continuous batching, and OpenAI-compatible APIs for GPU clusters.
AI & Machine Learning
SGLang
Structured generation language for fast serving: RadixAttention, constrained decoding, and multi-turn batching for frontier-class workloads.
AI & Machine Learning
MNN
Alibaba’s lightweight inference engine for mobile and edge—used for on-device LLMs and classic CV models with aggressive optimization.