TensorRT-LLM 1

Automated open-source AI model optimization and deployment API platform. RunInfra is an AI inference optimization and deployment platform that builds production-ready APIs from plain language descriptions of open-source models or full apps. Backed by Y Combinator, it benchmarks various serving engines (like vLLM, SGLang, and TensorRT-LLM) across different NVIDIA GPUs to find the most cost-effective and low-latency configuration. It leverages its automated Forge agent to optimize kernels, quantize models (AWQ, GPTQ, FP8), and handle continuous batching and KV cache tuning. Users can deploy on RunInfra's managed infrastructure with scale-to-zero capabilities or export the fully runnable deployment kit (Dockerfile, configs) to host on their own infrastructure like RunPod, Modal, or local hardware.