# Configuration Source: https://docs.tandemn.com/admin/configuration How to approach Tandemn server configuration. Tandemn server configuration starts from the `.env` file created by `setup.sh`. Use the repository's example environment values as the source of truth for the version you are running. ## Configure from the example ```bash theme={null} bash setup.sh ``` Open the generated `.env`, review every value, and update it for your deployment. The exact fields can change between releases, so use the repository's example environment values as the source of truth for that release. ## Configuration principles * Keep deployment-specific values in `.env`. * Do not commit `.env` to source control. * Use separate `.env` files for separate environments. * Record which server URL users should set in `TD_SERVER_URL`. * Set `S3_UPLOAD_BUCKET` to an existing bucket in your AWS account. * Set `HF_TOKEN` when running gated HuggingFace models. * Set `TD_API_KEY` when the control plane is exposed outside a private network. * Leave `KOI_SERVICE_URL` unset unless you intentionally enable optional Koi integration. * Keep model availability and cluster assumptions documented for users. ## User-facing value Users need one configuration value to start: ```bash theme={null} export TD_SERVER_URL= ``` If API key authentication is enabled, users also need: ```bash theme={null} export TD_API_KEY= ``` After that, they can verify access with: ```bash theme={null} tandemn check ``` ## Configuration handoff When onboarding a new user, give them: * The server URL * Supported model identifiers * Expected input file format * Expected SLO conventions * Whether `TD_API_KEY` is required * A support contact for failed jobs or connectivity issues # Environment variables Source: https://docs.tandemn.com/admin/environment-variables Environment variables used by Tandemn users and administrators. This page lists the environment variables users and administrators commonly touch. Server-side values live in the control plane `.env` file created by `setup.sh`. ## CLI variables | Variable | Required | Description | | --------------- | ---------- | ------------------------------------------------------------------------------------ | | `TD_SERVER_URL` | Yes | URL of the Tandemn server that the CLI should contact. | | `TD_API_KEY` | If enabled | API key for deployments that require authenticated client and replica communication. | ```bash theme={null} export TD_SERVER_URL= export TD_API_KEY= ``` Verify the value with: ```bash theme={null} echo $TD_SERVER_URL tandemn check ``` ## Server variables Server variables are release-specific. After running `bash setup.sh`, inspect `.env`. | Variable | Required | Description | | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------- | | `S3_UPLOAD_BUCKET` | Yes | Existing S3 bucket for uploaded workloads, chunk outputs, assembled output, and metrics. | | `HF_TOKEN` | For gated models | HuggingFace token for models such as Llama and Gemma. | | `TD_API_KEY` | Recommended | Shared API key for control plane authentication. | | `KOI_SERVICE_URL` | Optional | URL for optional Koi recommendations and lifecycle callbacks. Leave unset for standalone Orca behavior. | | `ANTHROPIC_API_KEY` | For LLM advisor | Enables the LLM placement advisor when the performance database is installed. | | `ORCA_URL` | Koi service only | URL Koi uses to call back to Orca/Tandemn System. | > Do not expose secrets, credentials, tokens, or private hostnames in public documentation or committed files. ## IAM requirements At minimum, the server's AWS credentials need permission to launch and terminate the instance types you plan to use, read and write objects in `S3_UPLOAD_BUCKET`, and read service quota values for capacity tracking. The server repository includes a least-privilege IAM policy example at `docs/iam-policy.json`. ## Common mistakes * Setting `TD_SERVER_URL` in one shell and running `tandemn` from another shell. * Using a server URL that is reachable from the admin machine but not from the user's network. * Leaving `S3_UPLOAD_BUCKET` pointed at a bucket that does not exist. * Setting `KOI_SERVICE_URL` accidentally when you want standalone behavior. * Committing `.env` by accident. # Administration Source: https://docs.tandemn.com/admin/index Operate the Tandemn server and help users troubleshoot setup and job submission. This section is for the person responsible for running the Tandemn server and supporting users. Bring up the Tandemn server and verify client connectivity. Manage release-specific settings with `.env`. Understand user-facing and server-side environment variables. Debug connectivity, control plane, and job submission issues. ## Administrator responsibilities * Keep the server reachable for CLI users. * Maintain deployment-specific configuration. * Share supported model identifiers and SLO conventions. * Help users diagnose failed connectivity or rejected jobs. # Security Source: https://docs.tandemn.com/admin/security Secure network access and API authentication for Tandemn System. The Tandemn System control plane accepts connections from EC2 replicas over the network. If the endpoint is exposed through a tunnel or public URL, set an API key. ## API key authentication Set `TD_API_KEY` in the server `.env` file: ```bash theme={null} TD_API_KEY=your-secret-key ``` Replica-to-server communication uses this token. The key is distributed to replicas automatically at launch. CLI users also need the key when the deployment requires authenticated requests: ```bash theme={null} export TD_API_KEY= ``` ## Network options | Option | Use case | | ----------------------- | ---------------------------------------------- | | Cloudflare Quick Tunnel | Local development with an ephemeral URL. | | Tailscale | Stable mesh VPN access for teams. | | Same-VPC EC2 | Production deployment without public exposure. | ## Cloudflare Quick Tunnel ```bash theme={null} curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared chmod +x cloudflared ./cloudflared tunnel --url http://localhost:26336 ``` Then pass the generated URL when starting the server or set it in `.env`: ```bash theme={null} TD_SERVER_URL=https://your-tunnel.trycloudflare.com ``` For production, deploy the control plane on a small EC2 instance in the same VPC as the inference replicas. This avoids public exposure and tunnel latency. # Server deployment Source: https://docs.tandemn.com/admin/server-deployment Operational guidance for running the Tandemn server. The Tandemn server should be treated as shared infrastructure. It receives job requests from users and coordinates execution across accelerated resources. ## Deployment checklist Pick a machine in your AWS environment that CLI users and EC2 inference replicas can reach. Configure AWS credentials, IAM permissions, S3 bucket access, and quota visibility. Fetch the Tandemn server repository and submodules. Install dependencies, verify AWS and Redis connectivity, and create `.env`. Run Redis for chunked multi-replica jobs. Run `python server.py --url `. Run `tandemn check` from a separate machine using the public server URL. ## Commands ```bash theme={null} git clone --recurse-submodules https://github.com/Tandemn-Labs/Tandemn-server.git cd Tandemn-server bash setup.sh docker run -d -p 6379:6379 redis python server.py --url https://your-public-url.example.com ``` ## Manual installation Use this path when you do not want to run `setup.sh`. ```bash theme={null} uv venv .venv --python 3.12 --seed source .venv/bin/activate uv pip install -r requirements.txt sky check ``` Create a `.env` file in the project root: ```bash theme={null} S3_UPLOAD_BUCKET=your-s3-bucket HF_TOKEN=hf_your_token_here TD_API_KEY=your-secret-key ANTHROPIC_API_KEY=sk-ant-... KOI_SERVICE_URL=http://localhost:8090 ``` `KOI_SERVICE_URL` is optional. Leave it unset for standalone Orca behavior. ## Operating model The administrator owns the server deployment, AWS credentials, S3 bucket, Redis, environment configuration, network exposure, and cluster access. Users own their prompt files, model choices, SLOs, and CLI environment. > Document the server URL, supported models, and expected SLO values for your users before sharing access. ## Network model For local development, the control plane can be exposed through a tunnel or VPN. For production, deploy the control plane on a small EC2 instance in the same VPC as the inference replicas to avoid public exposure and tunnel latency. ## Updates When a new Tandemn server version is available, review the release notes, back up deployment-specific configuration, and update in a controlled maintenance window. # Troubleshooting Source: https://docs.tandemn.com/admin/troubleshooting Fix common Tandemn setup and job submission issues. Start with connectivity. Most first-run issues happen before the job reaches the orchestration layer. ## `tandemn check` fails Check the server URL: ```bash theme={null} echo $TD_SERVER_URL ``` Then confirm the CLI can reach the server from the same terminal session: ```bash theme={null} tandemn check ``` If it still fails: * Confirm the server is running. * Confirm the server URL includes the correct host and port. * Confirm your network can reach the server. * Confirm any firewall or VPN rules allow the connection. * Confirm `TD_API_KEY` is set if the server requires authentication. ## Control plane does not start On the server host, check Python and setup output: ```bash theme={null} python --version sky check ``` Confirm Redis is running if you use chunked execution: ```bash theme={null} docker run -d -p 6379:6379 redis ``` ## A job is rejected Common causes: * The model is not supported by the deployment. * The input file path is wrong. * The prompt file is not valid JSONL. * The requested SLO is outside the conventions used by your deployment. * The cluster does not currently have compatible capacity. * `S3_UPLOAD_BUCKET` is unset or points at a bucket the control plane cannot write to. * AWS credentials lack EC2, S3, or service quota permissions. * The performance database is missing and the advisor is unavailable. ## A prompt file fails Validate that each line is a complete JSON object. ```json theme={null} {"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"model":"placeholder","messages":[{"role":"user","content":"Hello"}],"max_tokens":256}} ``` Avoid trailing commas and multi-line JSON objects in JSONL files. ## Still blocked Send your administrator: * The command you ran * The value of `TD_SERVER_URL`, without secrets * The model name * The first few non-sensitive lines of the JSONL file * The error message from the CLI # Analytics Source: https://docs.tandemn.com/cli/analytics Inspect completed Tandemn runs and scheduler timeseries. Analytics commands read completed run data recorded by the control plane. ## History ```bash theme={null} tandemn history tandemn history --model Qwen/Qwen2.5-7B-Instruct tandemn history --gpu A100 ``` | Flag | Description | | ----------- | ---------------------------------- | | `--model X` | Filter completed runs by model. | | `--gpu Y` | Filter completed runs by GPU type. | ## Inspect a run ```bash theme={null} tandemn inspect tandemn inspect --replicas ``` | Command | Description | | ------------------------------------- | -------------------------------------------------------------------- | | `tandemn inspect ` | Full run report with latency, throughput, cost, and GPU utilization. | | `tandemn inspect --replicas` | Show per-replica summaries side by side. | ## Scheduler timeseries ```bash theme={null} tandemn timeseries ``` Use `tandemn timeseries` when you need post-run scheduler detail for debugging throughput, chunk progress, or replica behavior. # tandemn check Source: https://docs.tandemn.com/cli/check Verify that the Tandemn CLI can reach the server. Use `tandemn check` before submitting jobs. It confirms that your CLI configuration points at a reachable Tandemn server. ## Usage ```bash theme={null} export TD_SERVER_URL= tandemn check ``` If the deployment requires API key authentication, set `TD_API_KEY` in the same shell session before running `tandemn check`. ```bash theme={null} export TD_API_KEY= tandemn check ``` ## When to run it * After installing the CLI * After changing `TD_SERVER_URL` * After changing `TD_API_KEY` * Before submitting a large workload * When debugging failed job submission ## If it fails Check: * `TD_SERVER_URL` is set in the same terminal session. * The server URL is correct. * Your machine can reach the server network. * The server is running. * `TD_API_KEY` is set if your administrator requires it. See [Troubleshooting](/admin/troubleshooting) for a fuller checklist. # plan and deploy Source: https://docs.tandemn.com/cli/deploy Preview placement plans and submit inference jobs to Tandemn System. Use `tandemn plan` to preview placement without launching, and `tandemn deploy` to submit a batch inference job. Both commands take a model name, an input JSONL file or S3 URI, and an SLO deadline. ## Usage ```bash theme={null} tandemn plan [options] tandemn deploy [options] ``` ## Examples ```bash theme={null} tandemn plan Qwen/Qwen2.5-7B-Instruct examples/workloads/demo_batch.jsonl --slo 4 tandemn deploy Qwen/Qwen2.5-7B-Instruct examples/workloads/demo_batch.jsonl --slo 4 ``` The solver automatically chooses a GPU type, tensor parallelism, pipeline parallelism, region, and launch mode unless you override those values. Manual placement example: ```bash theme={null} tandemn deploy Qwen/Qwen2.5-7B-Instruct input.jsonl --gpu A10G --tp 1 ``` Multi-replica chunked execution example: ```bash theme={null} tandemn deploy Qwen/Qwen2.5-7B-Instruct input.jsonl --gpu A10G --tp 1 --replicas 2 --chunk-size 100 ``` ## Arguments | Argument | Required | Description | | -------- | -------- | --------------------------------------------------------------------------------------------------------------- | | `model` | Yes | Model identifier to run. Use a model supported by your Tandemn deployment. | | `input` | Yes | Local JSONL file or `s3://...` URI containing the batch workload. Local files are uploaded to S3 automatically. | ## Options | Flag | Description | Default | | ----------------------- | ----------------------------------------------------------------------------------- | --------------- | | `--slo ` | Deadline. Accepts plain hours (`4`), fractional hours (`0.5h`), or minutes (`30m`). | `4` | | `--max-output-tokens N` | Maximum tokens per response. | `1024` | | `--gpu ` | Override GPU type, such as `A100`, `H100`, `L40S`, or `A10G`. | Solver-selected | | `--tp N` | Override tensor parallelism. | Solver-selected | | `--pp N` | Override pipeline parallelism. | Solver-selected | | `--replicas N` | Number of replica clusters. | `1` | | `--chunk-size N` | Lines per chunk for multi-replica jobs. | `1000` | | `--no-advisor` | Skip the LLM advisor and use the roofline solver only. | Disabled | | `--skip-dangerously` | Skip the interactive solver choice and auto-pick the advisor recommendation. | Disabled | | `--force` | Skip feasibility checks and launch anyway. | Disabled | | `--persist` | Keep clusters alive after the job completes. | Disabled | | `--on-demand` | Use on-demand instances instead of spot instances. | Disabled | ## Placement behavior Two placement solvers can run when you call `tandemn plan` or `tandemn deploy`: * LLM Advisor: architecture-aware recommendation over the performance database, with an LLM reasoning layer that ranks candidates by cost, throughput, and SLO feasibility. * Roofline solver: deterministic analytical placement based on GPU bandwidth, TFLOPS, memory, and model constraints. Both recommendations can be shown side by side. If the advisor is unavailable, Tandemn falls back to the roofline solver. If `KOI_SERVICE_URL` is set, Tandemn can also show an optional Koi recommendation alongside the built-in solvers. ## Supported overrides Use `--gpu`, `--tp`, and `--pp` when you want to force a specific hardware plan. This is useful for models outside the performance database or when you already know the target fleet. ```bash theme={null} tandemn deploy input.jsonl --gpu A10G --tp 1 ``` ## Prompt file format Use OpenAI-style batch JSONL. See [Input format](/cli/input-format) for the full schema. ```json prompts.jsonl theme={null} {"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"model":"placeholder","messages":[{"role":"user","content":"What is Tandemn?"}],"max_tokens":256}} ``` ## Before submitting * Run `tandemn check`. * Confirm the prompt file exists. * Confirm the file is valid JSONL. * Confirm the model is supported by your deployment. * Confirm the S3 upload bucket is configured on the server. * Start with a small file before scaling up. # CLI Reference Source: https://docs.tandemn.com/cli/index Command reference for the Tandemn CLI. Tandemn is operated by users through the CLI. Start by checking connectivity, then submit jobs with `tandemn deploy`. Install, configure, and use the Tandemn CLI. Verify that your CLI can reach the Tandemn server. Plan or submit a model inference job with a JSONL input file. Add, kill, or hot-swap replicas while a job is running. Watch progress, metrics, logs, clusters, and the web dashboard. Inspect completed runs, per-replica summaries, and scheduler timeseries. Prepare OpenAI-style batch JSONL workloads. ## Minimal workflow ```bash theme={null} pip install tandemn export TD_SERVER_URL= tandemn check tandemn plan Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 tandemn deploy Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 tandemn progress ``` # Input format Source: https://docs.tandemn.com/cli/input-format Prepare OpenAI-style batch JSONL workloads for Tandemn System. Tandemn System accepts OpenAI-style batch JSONL. Each line is one request object. ```json theme={null} { "custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": { "model": "placeholder", "messages": [ { "role": "user", "content": "Your prompt here" } ], "max_tokens": 256 } } ``` The `body.model` value can be a placeholder because the model is selected by the `tandemn deploy ` command. ## Minimal JSONL example ```json prompts.jsonl theme={null} {"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"model":"placeholder","messages":[{"role":"user","content":"Summarize what batch inference is in one sentence."}],"max_tokens":256}} {"custom_id":"req-2","method":"POST","url":"/v1/chat/completions","body":{"model":"placeholder","messages":[{"role":"user","content":"Give me three reasons to use heterogeneous GPUs."}],"max_tokens":256}} ``` ## Local files and S3 URIs Local files are uploaded to S3 automatically by the control plane. S3 URIs are passed through directly. ```bash theme={null} tandemn deploy Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 tandemn deploy Qwen/Qwen2.5-7B-Instruct s3://your-bucket/path/prompts.jsonl --slo 4 ``` ## Sample workloads The server repository includes sample workloads under `examples/workloads/`. | File | Size | Purpose | | ----------------------------- | -------------- | --------------------------------- | | `demo_batch.jsonl` | 30 requests | Quick smoke test. | | `sharegpt-numreq_200-*.jsonl` | 200 requests | Realistic ShareGPT conversations. | | `stress_5000.jsonl` | 5,000 requests | Load and stress testing. | Generate a larger workload from a sample: ```bash theme={null} python examples/workloads/make_long_workload.py \ examples/workloads/sharegpt-numreq_200-avginputlen_956-avgoutputlen_50.jsonl \ 25 /tmp/stress_5k.jsonl ``` # Monitoring and operations Source: https://docs.tandemn.com/cli/monitoring Monitor Tandemn jobs, metrics, clusters, logs, and cleanup. Use the monitoring commands after a job has been submitted with `tandemn deploy`. ## Progress and dashboard ```bash theme={null} tandemn progress tandemn progress tandemn web ``` | Command | Description | | --------------------------- | ---------------------------------------------------- | | `tandemn progress` | Live progress bar for the active or most recent job. | | `tandemn progress ` | Live progress bar for a specific job. | | `tandemn web` | Open the real-time web dashboard in a browser. | The dashboard shows workload details, chunk progress, replica phase state, cost, ETA, throughput, quota usage, event logs, and metrics charts. It uses server-sent events with polling fallback. ## Job and cluster state ```bash theme={null} tandemn status tandemn clusters tandemn logs [cluster] ``` | Command | Description | | ------------------------ | ------------------------------------- | | `tandemn status` | List jobs known to the control plane. | | `tandemn clusters` | Show active SkyPilot clusters. | | `tandemn logs [cluster]` | Stream logs from a SkyPilot cluster. | ## Metrics ```bash theme={null} tandemn metrics tandemn metrics --watch tandemn metrics --replica tandemn metrics --compare tandemn stream ``` | Command | Description | | ------------------------------------------ | ----------------------------------------------------- | | `tandemn metrics ` | Latest vLLM metrics snapshot. | | `tandemn metrics --watch` | Refresh metrics every two seconds. | | `tandemn metrics --replica ` | Show metrics for a specific replica. | | `tandemn metrics --compare` | Show aggregated and per-replica metrics side by side. | | `tandemn stream ` | Stream live metrics at roughly one event per second. | Metrics can include throughput, queue depth, KV cache utilization, scheduler state, GPU utilization, latency, and completion counters, depending on the replica state and server configuration. ## Cleanup ```bash theme={null} tandemn destroy tandemn destroy --all ``` | Command | Description | | -------------------------- | ----------------------------------------------- | | `tandemn destroy ` | Tear down clusters and Redis state for one job. | | `tandemn destroy --all` | Tear down all `tandemn` clusters. | Clusters are destroyed by default after job completion. Use `--persist` with `tandemn deploy` when you want to keep clusters alive. # CLI overview Source: https://docs.tandemn.com/cli/overview The Tandemn System command line workflow. The Tandemn CLI is the main interface for Tandemn System. Use it to check server connectivity, run placement planning, submit batch inference jobs, scale replicas, monitor progress, and inspect completed runs. Tandemn System is self-hosted. You run the control plane in your own AWS account, point the CLI at that server, and submit workloads from your environment. Your data stays inside your infrastructure. > Naming note: the product is Tandemn System. Parts of the repository, CLI output, and environment variables may still reference the legacy codename `orca`. ## Install ```bash theme={null} pip install tandemn ``` ## Configure ```bash theme={null} export TD_SERVER_URL= ``` If the server requires API key authentication, also set: ```bash theme={null} export TD_API_KEY= ``` ## Core commands Verify that the CLI can reach the Tandemn server. Preview or launch a job with a model, input file, and deadline. Add, kill, or hot-swap replica clusters while work is queued. Watch progress, metrics, logs, clusters, and the dashboard. Review completed runs and scheduler timeseries. Build OpenAI-compatible JSONL batch files. ## Typical workflow ```bash theme={null} pip install tandemn export TD_SERVER_URL= tandemn check tandemn plan Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 tandemn deploy Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 tandemn progress ``` ## Command groups | Command | Purpose | | -------------------------------- | ------------------------------------------------------------------------------- | | `tandemn check` | Verify CLI connectivity to the control plane. | | `tandemn plan ` | Show a placement plan without launching. | | `tandemn deploy ` | Run a batch job. The solver picks GPU and parallelism unless you override them. | | `tandemn add N` | Add replicas to a running job. | | `tandemn kill ` | Kill one or more replicas. | | `tandemn swap ` | Hot-swap all replicas to a new GPU configuration. | | `tandemn progress [job_id]` | Show a live progress bar. | | `tandemn web` | Open the real-time dashboard. | | `tandemn status` | List jobs. | | `tandemn metrics ` | Show vLLM metrics. | | `tandemn stream ` | Stream live metrics events. | | `tandemn logs [cluster]` | Stream SkyPilot cluster logs. | | `tandemn clusters` | Show active clusters. | | `tandemn destroy ` | Tear down job clusters and Redis state. | | `tandemn history` | Browse completed runs. | | `tandemn inspect ` | Show a full run report. | | `tandemn timeseries ` | Show scheduler timeseries for a completed run. | ## Responsibilities The CLI user is responsible for: * Installing the CLI * Pointing it at the correct Tandemn server * Preparing valid input files * Choosing a supported model * Sharing useful error details with the administrator when something fails # Replica operations Source: https://docs.tandemn.com/cli/replicas Add, kill, and hot-swap Tandemn replica clusters. Replica commands operate on running jobs that use chunked execution. New replicas join the same Redis chunk queue, and killed replicas have in-flight chunks reclaimed and returned to pending. ## Add replicas ```bash theme={null} tandemn add 2 tandemn add 3 --gpu L40S --tp 4 ``` The first command adds two replicas that inherit the job's current GPU configuration. The second command adds three L40S replicas, creating a heterogeneous fleet. | Argument or flag | Description | | ---------------- | ----------------------------------------------------------- | | `job_id` | Running job to scale. | | `N` | Number of replicas to add. | | `--gpu ` | Optional GPU type override for the new replicas. | | `--tp N` | Optional tensor parallelism override. | | `--pp N` | Optional pipeline parallelism override. | | `--on-demand` | Launch new replicas on on-demand instances instead of spot. | ## Kill replicas ```bash theme={null} tandemn kill --replica tandemn kill --replica r0 --replica r1 ``` Use `tandemn kill` to terminate specific replicas. Any chunk leased by a killed replica is reclaimed and returned to the queue. | Flag | Description | | ----------------- | -------------------------------------------------------------- | | `--replica ` | Replica ID to kill. Repeat the flag to kill multiple replicas. | ## Hot-swap replicas ```bash theme={null} tandemn swap --gpu A100 --tp 4 --replicas 2 tandemn swap --gpu L40S --tp 1 --ready-threshold 2 --on-demand ``` Hot-swap replaces all replicas with a new GPU configuration mid-job. Tandemn launches the replacement replicas first, waits for them to begin processing, and then tears down the old replicas. | Flag | Description | | --------------------- | -------------------------------------------------------------------------- | | `--gpu ` | GPU type for the replacement fleet. | | `--tp N` | Tensor parallelism for replacement replicas. | | `--pp N` | Pipeline parallelism for replacement replicas. | | `--replicas N` | Number of replacement replicas. | | `--ready-threshold N` | Number of new replicas that must be ready before old replicas are removed. | | `--on-demand` | Use on-demand instances for the replacement fleet. | ## Operational notes * Replica operations require Redis-backed chunk coordination. * Killed replicas do not lose completed chunks. * Hot-swap is designed to avoid dropped chunks by keeping the shared queue intact. # Architecture Source: https://docs.tandemn.com/concepts/architecture How the Tandemn server, CLI, users, and GPU resources work together. Tandemn System uses a client-server architecture. Administrators operate the self-hosted control plane, and users submit inference jobs through the CLI. ## Components The server that receives job requests, runs placement planning, tracks quota, launches replicas, and coordinates orchestration. The command line interface users install locally to check connectivity and submit workloads. AWS instances launched by SkyPilot to execute inference workloads. OpenAI-style JSONL files that describe the work users want Tandemn to run. ## Request flow The user runs `tandemn deploy` with a model, JSONL input file, and SLO deadline. The control plane validates the request, uploads local input to S3, and evaluates available resources. The placement solver chooses GPU type, region, tensor parallelism, pipeline parallelism, replica count, and launch mode. SkyPilot launches replicas, Redis coordinates chunks, vLLM runs inference, and outputs are written to S3. ## Execution model Input is split into chunks and queued in Redis. Replica clusters pull chunks, run inference through vLLM, upload per-chunk outputs, and mark chunks complete. If a replica dies, its in-flight chunks can be reclaimed and returned to the queue. `tandemn swap` keeps the same queue while launching replacement replicas with a different GPU configuration. Old replicas are removed after the new replicas begin processing. ## Optional Koi Orca standalone is the default path. Leave `KOI_SERVICE_URL` unset and the control plane handles placement, launch, chunked execution, monitoring, recovery, and output assembly itself. If `KOI_SERVICE_URL` is set, Tandemn can call Koi for an additional recommendation and send lifecycle callbacks. If Koi is unset, unavailable, times out, or returns bad data, Tandemn continues through the standalone path. ## Why this model matters This separation lets infrastructure teams manage hardware and cluster policy centrally while application and ML teams get a simpler job submission interface. > Treat the Tandemn server as shared infrastructure. Keep ownership, deployment, and monitoring responsibilities clear before inviting users onto the cluster. # Batch inference Source: https://docs.tandemn.com/concepts/batch-inference How Tandemn thinks about queued inference workloads. Batch inference is the process of running many inference requests as a workload instead of serving one interactive request at a time. In Tandemn, a batch job usually starts as a prompt file and a model selection. The CLI sends that job to the server, and the server decides how to run it across the available accelerated infrastructure. ## When batch inference is a good fit * Offline evaluation jobs * Dataset labeling or enrichment * Scheduled summarization, extraction, or classification tasks * Experiments that can tolerate queueing in exchange for lower cost or better hardware utilization ## Why heterogeneous GPUs help Not every workload needs the newest, largest GPU. Some jobs can run efficiently on smaller or less utilized accelerators. Tandemn is designed to make that resource selection part of the orchestration layer instead of a manual decision for every user. ## What users provide Users typically provide: * A model identifier * A JSONL input file * A service-level objective ```bash theme={null} tandemn deploy Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 ``` ## What Tandemn handles Tandemn receives the job, evaluates the available resources, and chooses an execution plan based on the deployment's cluster state and workload requirements. # Concepts Source: https://docs.tandemn.com/concepts/index Understand the core ideas behind Tandemn's CLI-first inference orchestration model. Use these pages to understand how Tandemn works before operating it in a shared environment. See how the CLI, server, and GPU resources fit together. Learn why Tandemn focuses on queued inference workloads. Follow a job from input file to execution. Understand how model choice and hardware placement relate. ## CLI-first design Tandemn is documented as a CLI-first product. Users submit work with `tandemn deploy`; administrators operate the server and cluster environment behind that interface. # Job lifecycle Source: https://docs.tandemn.com/concepts/job-lifecycle What happens after a user submits a Tandemn job. A Tandemn job moves from user submission to server-side scheduling and execution. The user prepares a JSONL prompt file and chooses a model that the deployment can run. The user runs `tandemn deploy` from a machine that can reach the Tandemn server. The server checks the request and determines whether the workload can be accepted. Tandemn evaluates the available GPU pool and chooses where to place the workload. The workload runs on the selected resources. The user checks the CLI output and any job artifacts exposed by the deployment. ## Practical advice * Start with a small prompt file before submitting a large workload. * Confirm the model is available in your deployment. * Keep `TD_SERVER_URL` visible in your terminal when debugging. * Ask your administrator for the supported SLO conventions in your environment. # Models and routing Source: https://docs.tandemn.com/concepts/models-and-routing How model selection and hardware routing fit into Tandemn. Tandemn users submit jobs with a model identifier, a JSONL workload, and a deadline. The control plane uses the deployment's available resources and configuration to decide how that workload should run. ## Model identifiers The quickstart uses a Hugging Face style model identifier: ```bash theme={null} tandemn deploy Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 ``` Use model identifiers that your Tandemn deployment supports. Tandemn System can run HuggingFace models compatible with vLLM, and you can override placement manually for models outside the performance database. ```bash theme={null} tandemn deploy input.jsonl --gpu A10G --tp 1 ``` ## Placement solvers `tandemn plan` and `tandemn deploy` can show recommendations from two built-in solvers: | Solver | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------- | | LLM Advisor | Uses the performance database and an LLM reasoning layer to rank placements by cost, throughput, and SLO feasibility. | | Roofline solver | Uses GPU bandwidth, TFLOPS, memory, and model constraints. No API key is required. | If the advisor is unavailable, Tandemn falls back to the roofline solver. If `KOI_SERVICE_URL` is set, Tandemn can also show an optional Koi recommendation. The LLM Advisor requires `ANTHROPIC_API_KEY` and the performance database. Without those, `tandemn plan` and `tandemn deploy` can still use the roofline solver. ## Performance database The LLM placement advisor uses a performance database of profiled vLLM runs. Download it into the server repository: ```bash theme={null} curl -L https://github.com/Tandemn-Labs/LLM_placement_solver/releases/download/aiconfigurator-v1/data.csv \ -o LLM_placement_solver/llm_advisor/data/aiconfigurator/data.csv ``` The dataset contains 103K profiled vLLM runs across A100, H100, H200, B200, GB200, and L40S GPUs, sourced from NVIDIA Dynamo AIConfigurator. ## Profiled models | Model | Params | Type | Profiled GPUs | | -------------------------- | ------ | -------------------- | ----------------------------------- | | Meta-Llama-3.1-8B | 8B | Dense | A100, H100, H200, B200, GB200, L40S | | Meta-Llama-3.1-70B | 70B | Dense | A100, H100, H200, B200, GB200, L40S | | Llama-3.1-70B-Instruct-FP8 | 70B | Dense FP8 | A100, H100, H200, B200, GB200, L40S | | Meta-Llama-3.1-405B | 405B | Dense | A100, H100, H200, B200, GB200 | | Nemotron-Super-49B | 49B | Dense | A100, H100, H200, B200, GB200, L40S | | Nemotron-H-56B | 56B | Dense | A100, H100, H200, B200, GB200, L40S | | Qwen3-8B | 8B | Dense | A100, H100, H200, B200, GB200, L40S | | Qwen3-32B | 32B | Dense | A100, H100, H200, B200, GB200, L40S | | Qwen3-32B-FP8 | 32B | Dense FP8 | A100, H100, H200, B200, GB200, L40S | | Qwen3-30B-A3B | 30B | MoE, 3B active | A100, H100, H200, B200, GB200, L40S | | Qwen3-235B-A22B | 235B | MoE, 22B active | A100, H100, H200, B200, GB200 | | Qwen3-235B-A22B-FP8 | 235B | MoE, 22B active, FP8 | A100, H100, H200, B200, GB200 | | Qwen3-235B-A22B-NVFP4 | 235B | MoE, 22B active, FP4 | A100, H100, H200, B200, GB200 | | Qwen3-Coder-480B-A35B | 480B | MoE, 35B active | H100, H200, B200, GB200 | | Nemotron-3-Nano-30B-A3B | 30B | MoE, 3B active | A100, H100, H200, B200, GB200, L40S | For models not in the database, the advisor estimates throughput by matching model family, size, and I/O profile. Use `--gpu`, `--tp`, and `--pp` when you want to override placement manually. ## Routing goals Tandemn's routing layer is designed to reduce the amount of manual placement work users need to do. Instead of asking each user to pick a specific machine, Tandemn can evaluate the job and choose an appropriate hardware mix. ## What affects placement The exact placement decision depends on deployment-specific configuration, but these are the common inputs to reason about: * Model size and runtime requirements * Prompt file size * Requested SLO * Available GPUs * Current cluster load * AWS quota and capacity * Spot or on-demand launch mode * Tensor and pipeline parallelism settings ## Supported hardware | GPU | AWS instance | VRAM | | --------- | ---------------------------------------------- | -------------------- | | A100 80GB | `p4d.24xlarge`, `p4de.24xlarge` | 8 x 80GB | | H100 80GB | `p5.48xlarge` | 8 x 80GB | | L40S 48GB | `g6e.12xlarge`, `g6e.24xlarge`, `g6e.48xlarge` | 4 x / 4 x / 8 x 48GB | | A10G 24GB | `g5.12xlarge`, `g5.48xlarge` | 4 x / 8 x 24GB | The solver searches across GPU types and parallelism configurations to find a placement that fits the model in memory and meets the requested deadline. > If a model cannot be scheduled, the first thing to check is whether the model is enabled in the deployment and whether compatible resources are available. # Getting started Source: https://docs.tandemn.com/getting-started/index Prepare your environment, install Tandemn, and submit a first inference job. Use this section when you are setting up Tandemn for the first time. Start with the requirements, then install the server and CLI, then submit a small test job. Check AWS, Python, Redis, S3, and network requirements before installing anything. Start the Tandemn server on a machine users can reach. Install the `tandemn` package and point it at the server. Submit a small JSONL prompt file and confirm the end-to-end workflow. Review what administrators and CLI users need before setup. Install the control plane, configure AWS, and start the server. Set up a local command line environment. Submit a model and JSONL prompt file to Tandemn. # Install the CLI Source: https://docs.tandemn.com/getting-started/install-cli Install the Tandemn CLI and connect it to a Tandemn server. The Tandemn CLI is the user-facing tool for checking connectivity and submitting inference jobs. ## Install Install the package in your preferred Python 3.12+ environment. ```bash pip theme={null} pip install tandemn ``` ```bash venv theme={null} python -m venv .venv source .venv/bin/activate pip install tandemn ``` ## Connect to the server Set `TD_SERVER_URL` to the URL provided by your Tandemn administrator. ```bash theme={null} export TD_SERVER_URL= ``` If your deployment requires API key authentication, also set `TD_API_KEY`. ```bash theme={null} export TD_API_KEY= ``` Persist the variable in your shell profile if you use the same server regularly. ```bash theme={null} echo 'export TD_SERVER_URL=' >> ~/.zshrc ``` ## Check connectivity ```bash theme={null} tandemn check ``` Successful output means the CLI can reach the Tandemn server and you can submit jobs. > If your organization uses multiple Tandemn environments, keep separate shell profiles or terminal sessions for development, staging, and production. ## Next step Submit your first inference job in [Run your first job](/getting-started/run-first-job). # Install the server Source: https://docs.tandemn.com/getting-started/install-server Install and start the Tandemn System control plane. The Tandemn System control plane receives job requests from users, runs placement planning, launches replicas through SkyPilot, coordinates chunks through Redis, and writes output to S3. Deploy it in your AWS environment on a host that CLI users and EC2 replicas can reach. ## Clone the repository ```bash theme={null} git clone --recurse-submodules https://github.com/Tandemn-Labs/Tandemn-server.git cd Tandemn-server ``` ## Run setup `setup.sh` installs Python dependencies, verifies AWS and Redis connectivity, and creates your `.env` file. ```bash theme={null} bash setup.sh ``` Review `.env` before starting the control plane. At minimum, confirm: * `S3_UPLOAD_BUCKET` * `HF_TOKEN`, if you use gated models * `TD_API_KEY`, if the control plane is reachable outside a private network * `KOI_SERVICE_URL`, only if you are enabling optional Koi integration > Do not commit `.env` files with secrets or deployment-specific credentials. ## Start Redis ```bash theme={null} docker run -d -p 6379:6379 redis ``` Redis is required for multi-replica chunked jobs. ## Download the performance database The LLM placement advisor uses a performance database that is not included in the repository. ```bash theme={null} curl -L https://github.com/Tandemn-Labs/LLM_placement_solver/releases/download/aiconfigurator-v1/data.csv \ -o LLM_placement_solver/llm_advisor/data/aiconfigurator/data.csv ``` Without this dataset, `tandemn plan` falls back to the roofline solver. ## Start the control plane Set the public URL that EC2 replicas will use to call back to the control plane. ```bash theme={null} python server.py --url https://your-public-url.example.com ``` For local development, you can use a tunnel or VPN endpoint. For production, run the control plane on a small EC2 instance in the same VPC as your inference replicas. ## Next steps Set up the user-facing command line tool. Understand what belongs in `.env`. # Requirements Source: https://docs.tandemn.com/getting-started/requirements What you need before deploying Tandemn or using the CLI. Tandemn System has two parts: a self-hosted control plane operated by an administrator and a CLI used by the people submitting inference jobs. The current deployment target is AWS. ## Server requirements | Requirement | Notes | | ------------------- | ---------------------------------------------------------------------------------- | | Python 3.12+ | Required for the control plane. | | `uv` | Required for setup and dependency management. | | SkyPilot | Installed by `setup.sh`; run `sky check` to verify cloud access. | | AWS credentials | Use `aws configure` or `~/.aws/credentials`. | | AWS IAM permissions | EC2 launch/terminate, S3 read/write to your bucket, and service quota read access. | | S3 bucket | Must exist in your AWS account and be set as `S3_UPLOAD_BUCKET` in `.env`. | | Redis | Required for multi-replica chunked jobs. | | HuggingFace token | Required for gated models such as Llama and Gemma; set `HF_TOKEN` in `.env`. | At minimum, AWS credentials need access to the EC2 instance types you plan to use, `s3:GetObject` and `s3:PutObject` on your bucket, and `servicequotas:GetServiceQuota` for quota tracking. Install `uv` before running server setup: ```bash theme={null} curl -LsSf https://astral.sh/uv/install.sh | sh ``` ## CLI requirements Users need a Python environment and the URL of the Tandemn server. * Python 3.12 or newer * `pip` * The `tandemn` Python package * `TD_SERVER_URL` set to the server URL ```bash theme={null} pip install tandemn export TD_SERVER_URL= tandemn check ``` ## Recommended first setup For an initial evaluation, use one administrator-managed server, one CLI user, and a small JSONL workload. Verify connectivity with `tandemn check` before submitting real workloads. Install and start the control plane. Set up a local CLI environment. # Run your first job Source: https://docs.tandemn.com/getting-started/run-first-job Submit a first batch inference job with Tandemn. After the server is running and the CLI is connected, submit a job with a model, an input file, and an SLO. ## Prepare a prompt file Use OpenAI-style batch JSONL where each line is one request payload. Keep the first test small so you can confirm the end-to-end flow quickly. ```json prompts.jsonl theme={null} {"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"model":"placeholder","messages":[{"role":"user","content":"Summarize what batch inference is in one sentence."}],"max_tokens":256}} {"custom_id":"req-2","method":"POST","url":"/v1/chat/completions","body":{"model":"placeholder","messages":[{"role":"user","content":"Give me three reasons to use heterogeneous GPUs."}],"max_tokens":256}} ``` ## Preview the placement ```bash theme={null} tandemn plan Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 ``` ## Submit the job ```bash theme={null} tandemn deploy Qwen/Qwen2.5-7B-Instruct prompts.jsonl --slo 4 ``` The command sends the workload to the Tandemn server. Tandemn then chooses an execution plan across the available accelerated resources. ## What the arguments mean * `Qwen/Qwen2.5-7B-Instruct` is the model identifier. * `prompts.jsonl` is the batch input file. * `--slo 4` is a four-hour deadline. ## Monitor progress ```bash theme={null} tandemn progress tandemn web ``` > Use a model that your Tandemn deployment is configured to run. If a model cannot be scheduled, ask your administrator which models are currently available. ## If something fails Start with the basics: * Run `tandemn check`. * Confirm `TD_SERVER_URL` points at the right server. * Confirm the JSONL file exists and is readable. * Confirm the model is available in your environment. See [Troubleshooting](/admin/troubleshooting) for more setup checks. # Introduction Source: https://docs.tandemn.com/introduction Learn what Tandemn does, who it is for, and where to start. Tandemn System helps teams run inference workloads across accelerated infrastructure without hand-tuning every model, GPU, and batch job. It provides a self-hosted control plane that manages orchestration and a CLI that users can use to submit inference jobs. Instead of sending every workload to the newest and most expensive accelerator, Tandemn can route jobs across heterogeneous GPU pools and use the hardware that best matches the workload's latency and throughput requirements. Install the server, connect the CLI, and submit a model inference job. Learn how the server, users, GPU workers, and job scheduler fit together. Install the control plane, configure AWS, and start the server. Check connectivity and submit jobs from a local Python environment. ## What Tandemn is for Tandemn System is designed for teams that already have, or plan to operate, accelerated compute and want a simpler way to run inference workloads across that capacity. * **Infrastructure teams** can expose a single service for users instead of asking each team to manage hardware placement. * **ML and application teams** can submit jobs through the CLI without deciding which machine or GPU should run them. * **Organizations with mixed GPU supply** can use available capacity more efficiently across different machines and accelerator types. ## How the workflow fits together The control plane is deployed on a machine that users and EC2 replicas can reach over the network. It manages cluster state and receives inference job requests. Users install the `tandemn` Python package, set `TD_SERVER_URL`, and check connectivity. A user provides a model, a JSONL prompt file, and a service-level objective. Tandemn schedules the job across available accelerated resources. The orchestration layer selects an efficient hardware mix for the workload so users can focus on the job rather than the cluster. ## Next step Start with the [Quickstart](/quickstart) if you want the shortest path from a new environment to a submitted inference job. # FAQ Source: https://docs.tandemn.com/learn-more/faq Answers to common questions about Tandemn. ## What is Tandemn? Tandemn is an infrastructure layer for running inference workloads across accelerated clusters. It gives administrators a server to operate and users a CLI to submit jobs. ## Who should use Tandemn? Tandemn is built for teams that run inference workloads and want to reduce manual infrastructure decisions. It is especially useful when an organization has mixed GPU supply or idle accelerated capacity. ## What problem does Tandemn solve? Inference infrastructure often requires users to choose hardware, size jobs, and manage placement manually. Tandemn moves those decisions into an orchestration layer so users can submit work through a simpler interface. ## Is Tandemn only for batch inference? The current docs focus on batch inference jobs submitted through the CLI. Batch workflows are a natural fit when jobs can be queued, planned, and executed across available resources. ## What does a user need to get started? A user needs the Tandemn CLI, the server URL from an administrator, and a valid OpenAI-style JSONL input file. ```bash theme={null} pip install tandemn export TD_SERVER_URL= tandemn check ``` ## What does an administrator need to get started? An administrator needs Python 3.12+, AWS credentials, IAM access for EC2/S3/service quotas, an S3 bucket, Redis, and a network-reachable host for the self-hosted control plane. ## How do I know which models are available? Model availability is deployment-specific. Ask your administrator which model identifiers are supported in your Tandemn environment. ## Where should I start? Use the [Quickstart](/quickstart) for an end-to-end setup, or use [Requirements](/getting-started/requirements) to understand what you need before deploying Tandemn. # Learn more Source: https://docs.tandemn.com/learn-more/index Find answers, support guidance, and official Tandemn links. Use this section when you want product context, support guidance, or official Tandemn links. Answers to common Tandemn questions. What to include when asking for help. Official Tandemn website, GitHub, social, and contact links. # Links Source: https://docs.tandemn.com/learn-more/links Official Tandemn links and resources. Learn more about Tandemn. View Tandemn's open source repositories. Follow product updates. Contact the Tandemn team. # Support Source: https://docs.tandemn.com/learn-more/support How to get help with Tandemn. If you are blocked, include enough context for someone to reproduce the issue. ## For CLI users Send your administrator: * The command you ran * The error message * Your `TD_SERVER_URL` value, if it is not sensitive * The model name * A small non-sensitive sample of the prompt file ## For administrators Check: * The server containers are running * The `.env` file is configured for the current deployment * Users can reach the server URL over the network * The requested model is supported * The cluster has compatible capacity ## Contact Tandemn For product or account help, email [contact@tandemn.com](mailto:contact@tandemn.com). # Quickstart Source: https://docs.tandemn.com/quickstart Deploy Tandemn server, connect the CLI, and submit your first inference job. ## Pre-requisites Server: * Docker with Docker Compose support CLI: * Python >= 3.12 ## Server The following steps are for the sysadmin to bring up Tandemn's server in order to manage the cluster and handle inference job requests from users. The servers consists of 3 containers, and are bundled in one `docker-compose.yml`. ### Download the release The server is released as a .tar.xz tarball, bundling a `docker-compose.yml` and a `.env.example`. ```bash theme={null} curl -L -O "https://github.com/Tandemn-Labs/tandemn-system/releases/download/0.0.1/tandemn-0.0.1.tar.gz" tar -xzf tandemn-0.0.1.tar.gz cd tandemn-0.0.1 ``` ### Populate the .env file Copy the `.env.example` file into `.env` and edit the respective parameters. ```bash theme={null} cp .env.example .env ``` ### Bring up the containers ```bash theme={null} docker compose pull docker compose up ``` ## Client (CLI) ### Install the Tandemn CLI You can do the following in any Python installation or virtual environment ```bash theme={null} pip install tandemn ``` ### Set server URL ``` export TD_SERVER_URL= ``` ### Check connectivity ``` tandemn check ``` ### Start using the CLI to launch jobs ``` tandemn deploy Qwen/Qwen2.5-7B-Instruct --slo 4 ``` You can look in the CLI documentation section to see details about the commands. ## Where to go next Review the deployment flow in more detail. Learn what the deploy command expects. Understand the client-server model. Debug the most common setup issues. # REST API Source: https://docs.tandemn.com/reference/rest-api Control plane endpoints exposed by Tandemn System. The Tandemn System control plane exposes a REST API at `http://localhost:26336` by default. Most users should use the CLI; use the REST API for integrations, dashboards, and operational tooling. ## Endpoint reference | Endpoint | Method | Description | | ---------------------------------- | ------ | ----------------------------------------------------------- | | `/submit/batch` | POST | Submit a batch inference job. | | `/test/placement` | POST | Run solver only without launching. | | `/jobs` | GET | List all jobs. | | `/job/{id}` | GET | Job status and progress. | | `/job/{id}/phase` | POST | Update job lifecycle phase. | | `/job/{id}/metrics` | GET | Latest aggregated metrics snapshot. | | `/job/{id}/metrics/stream` | GET | SSE metrics stream. | | `/job/{id}/metrics/ingest` | POST | Sidecar metrics ingest from replicas. | | `/job/{id}/metrics/summary` | POST | Per-replica build metrics summary. | | `/job/{id}/throughput` | GET | Sustained throughput over the rolling window. | | `/job/{id}/replicas` | GET | Per-replica state, phase, region, and metrics availability. | | `/job/{id}/replicas/{rid}/metrics` | GET | Metrics for a specific replica. | | `/job/{id}/replicas/summaries` | GET | Per-replica completion summaries. | | `/job/{id}/scale` | POST | Add replicas to a running job. | | `/job/{id}/kill` | POST | Kill specific replicas. | | `/job/{id}/swap` | POST | Hot-swap replicas to a new GPU configuration. | | `/job/{id}/chunks/progress` | GET | Chunk-level progress. | | `/job/{id}/chunks/pull` | POST | Pull the next chunk. Replica-facing. | | `/job/{id}/chunks/complete` | POST | Mark a chunk complete. | | `/job/{id}/chunks/renew` | POST | Renew a chunk lease. | | `/dashboard` | GET | Web dashboard HTML. | | `/dashboard/poll` | GET | Dashboard JSON payload for polling fallback. | | `/dashboard/stream` | GET | Real-time dashboard SSE stream. | | `/analytics/runs` | GET | List completed runs. | | `/analytics/runs/{id}` | GET | Full completed run report. | | `/analytics/runs/{id}/timeseries` | GET | Scheduler timeseries for a run. | | `/quota/status` | GET | Quota usage across AWS regions. | | `/resources` | GET | Instance catalog and quota pools. | ## Example responses ### `GET /job/{id}/metrics` ```json theme={null} { "job_id": "mo-qwen7b-a1b2", "timestamp": 1711612800.0, "avg_generation_throughput_toks_per_s": 1450.5, "avg_prompt_throughput_toks_per_s": 320.0, "gpu_cache_usage_perc": 0.42, "num_requests_running": 64, "num_requests_waiting": 0, "generation_tokens_total": 2800000, "prompt_tokens_total": 350000, "gpu_sm_util_pct": 95.2, "gpu_mem_bw_util_pct": 61.0, "ttft_ms_p50": 45.0, "ttft_ms_p95": 120.0, "tpot_ms_p50": 8.5, "tpot_ms_p95": 15.0 } ``` ### `GET /job/{id}/chunks/progress` ```json theme={null} { "total": 10, "pending": 3, "inflight": 2, "completed": 5, "failed": 0, "all_done": false } ``` ### `POST /job/{id}/scale` ```json theme={null} { "count": 2, "gpu_type": "L40S", "tp_size": 4, "pp_size": 1, "on_demand": false, "force": false } ``` `gpu_type`, `tp_size`, and `pp_size` are optional. If omitted, Tandemn inherits them from the existing job. ### `POST /job/{id}/swap` ```json theme={null} { "gpu_type": "H100", "tp_size": 4, "num_replicas": 2, "ready_threshold": 1, "force": false } ``` New replicas launch first. Old replicas are killed after `ready_threshold` new replicas begin processing. # Roadmap Source: https://docs.tandemn.com/roadmap Features that are in the works ## Observability / CLI * Add individual user level observability * Deployment of jobs within individual user's cluster limit * Dashboard view of only individual user's job * Integration with enterprise login solutions like SSO/RBAC/SAML * Add administrator controls to cluster * Deletion / termination of jobs from admin level ## Control plane * Adding suport for cloud + on-premise hybrid clusters * Unified resource views across cloud and on-premise machines * Scheduling of jobs across hybrid cluster scenarios * Adding support for GCP & MS Azure * Mapping instance types for same GPU types across cloud providers ## Data plane * Configuring fast inter-instance networking in GCP & Azure * GCP: GPUDirect-TCPX/TCPXO * Azure: InfiniBand/RDMA * Adding online inference based on Nvidia Dynamo * Nvidia Dynamo has important features like KV-aware routing, throughput/load-based autoscaling, multinode inference primitives