Skip to main content
Tandemn users submit jobs with a model identifier, a JSONL workload, and a deadline. The control plane uses the deployment’s available resources and configuration to decide how that workload should run.

​
Model identifiers

The quickstart uses a Hugging Face style model identifier:
Use model identifiers that your Tandemn deployment supports. Tandemn System can run HuggingFace models compatible with vLLM, and you can override placement manually for models outside the performance database.

​
Placement solvers

tandemn plan and tandemn deploy can show recommendations from two built-in solvers: If the advisor is unavailable, Tandemn falls back to the roofline solver. If KOI_SERVICE_URL is set, Tandemn can also show an optional Koi recommendation. The LLM Advisor requires ANTHROPIC_API_KEY and the performance database. Without those, tandemn plan and tandemn deploy can still use the roofline solver.

​
Performance database

The LLM placement advisor uses a performance database of profiled vLLM runs. Download it into the server repository:
The dataset contains 103K profiled vLLM runs across A100, H100, H200, B200, GB200, and L40S GPUs, sourced from NVIDIA Dynamo AIConfigurator.

​
Profiled models

For models not in the database, the advisor estimates throughput by matching model family, size, and I/O profile. Use --gpu, --tp, and --pp when you want to override placement manually.

​
Routing goals

Tandemn’s routing layer is designed to reduce the amount of manual placement work users need to do. Instead of asking each user to pick a specific machine, Tandemn can evaluate the job and choose an appropriate hardware mix.

​
What affects placement

The exact placement decision depends on deployment-specific configuration, but these are the common inputs to reason about:
  • Model size and runtime requirements
  • Prompt file size
  • Requested SLO
  • Available GPUs
  • Current cluster load
  • AWS quota and capacity
  • Spot or on-demand launch mode
  • Tensor and pipeline parallelism settings

​
Supported hardware

The solver searches across GPU types and parallelism configurations to find a placement that fits the model in memory and meets the requested deadline.
If a model cannot be scheduled, the first thing to check is whether the model is enabled in the deployment and whether compatible resources are available.