Skip to content
Inference Stack

Inference Stack

Everything you need to own your inference.
Deploy with one command.

Hugging FacevLLMLiteLLMllm-dGrafana
LiteLLM routes requests to models. llm-d selects a vLLM replica. Hugging Face supplies weights; Grafana shows metrics from Prometheus.

Quickstart

v0.1.2 preview

For one machine. Requires Docker with Compose, curl, and OpenSSL. Setup details

curl -fsSL https://github.com/BerriAI/inference-stack/releases/download/v0.1.2/install.sh | bash -s -- docker

Open localhost:8080 and sign in as admin with the generated password. Choose a model in vLLM, deploy it, then open LiteLLM → Playground to send a prompt.

Real models need NVIDIA GPUs. No GPU? Enable simulation in the model’s Settings for canned responses.

The stack

vLLM

Search Hugging Face, check GPU requirements, and deploy from the launcher. vLLM runs the model on your GPUs.

vLLM docs
The launcher searches Hugging Face and shows estimated GPU requirements for each model.

LiteLLM

Send requests through one API. Create keys, manage teams and budgets, or try a model in the Playground.

LiteLLM docs
LiteLLM Playground sends a request to a deployed simulated model.

llm-d

Selects a vLLM replica for each request using cache and queue data. Routing works by default; adjust the weights when you need to.

llm-d docs
The routing page shows two simulated replicas and optional cache and queue settings.

Grafana

See model health, request traffic, and response times in the included dashboards. Metrics come from Prometheus.

Grafana docs
Grafana shows model health and request traffic from a simulated deployment.