Operating foundation-model infrastructure is rarely a matter of submitting one job. Teams must create networks and control planes, attach accelerator capacity, install dependencies, configure identity and storage, recover from hardware failures, deploy model servers and monitor the resulting system. HyperPod InstantStart addresses the difficult handoffs between those steps by placing them behind a shared, stateful control plane.
The open-source project combines the user-managed orchestration surface of Amazon Elastic Kubernetes Service with managed Amazon SageMaker HyperPod capabilities. Its web interface and AI agent do not implement separate automation paths: both use the same backend APIs, validation rules and persisted operation state. The result is a system intended to turn cluster bootstrap, capacity management, training, inference and storage into guarded, retryable operations.
Solution overview
InstantStart runs in an AWS account as a single out-of-band management container. It calls AWS service APIs and the Kubernetes API, but it does not sit in the data path for training jobs or inference requests. Resources it creates remain standard AWS and Kubernetes objects that operators can inspect with the AWS CLI and kubectl.
The responsibility boundary is important. Amazon EKS remains user-managed and contains the Kubernetes API, EKS add-ons, and custom resources such as HyperPodPyTorchJob and InferenceEndpointConfig. SageMaker HyperPod supplies the AWS-managed capabilities around infrastructure health, automatic recovery, capacity provisioning, training recovery, checkpointing, inference routing and caching. Kubernetes schedules workloads onto HyperPod instance groups, while HyperPod manages those groups.
InstantStart brings the environment together in four broad layers:
- Infrastructure: staged EKS creation or import, network layout, dependency reconciliation and multi-cluster state, built on CloudFormation and EKS.
- Capacity and resilience: instance-group workflows and managed-feature configuration backed by HyperPod health monitoring, automatic node recovery, continuous provisioning and managed Karpenter.
- Workloads and data: training recipes, two inference paths, model downloads, storage workflows and MLflow integration.
- Interfaces: a web UI, REST APIs, MCP tools and agent skills, while the underlying AWS and Kubernetes resources remain directly inspectable.
A central design choice is that the MCP tools wrap the same REST endpoints used by the browser. A validation or reconciliation rule added to the backend therefore protects human-initiated and agent-initiated operations alike.
Prerequisites and deployment precautions
Deployment and ongoing operation require least-privilege IAM roles. S3 access should be restricted to the designated project bucket, and teams must account for CloudFormation and HyperPod permissions. Capacity planning can also require lead time: the source advises requesting the SageMaker Cluster Usage quota for every intended instance type and purchasing a SageMaker Flexible Training Plan for high-end accelerators when reserved capacity is needed. VPC quotas matter because staged provisioning creates one VPC per cluster by default.
The project supplies a CloudFormation template that creates a management environment, a shared S3 bucket and supporting IAM roles. On the resulting instance, operators clone the repository and run ui-panel/start-prod.sh. That script pulls a prebuilt image from public Amazon ECR, mounts AWS and kubectl credentials, and serves the interface on port 3099. AWS Systems Manager port forwarding is recommended instead of exposing that port publicly. The template security group permits public access for convenience, so that rule should be restricted before real use.
The agent path requires an installed and authenticated Kiro CLI. The MCP server and agent skills are included in the same container, avoiding another local tool installation.
Creating a cluster through staged operations
InstantStart separates cluster creation into six workflow stages: create the EKS cluster, select and verify it, install dependencies, create the HyperPod cluster, configure S3 storage and perform final verification. EKS control-plane creation takes roughly 8–12 minutes. Later stages persist their own status and can be retried independently, so a late failure does not undo a successfully created control plane.
In the source’s example session, the agent first found 2 existing clusters. It then created an EKS cluster, checked progress every 2 minutes using a 120-second wait, and reported that creation was still underway after about 4 minutes. Only after dependencies were installed did it request decisions about Availability Zone, accelerator type, instance count and capacity type. The example ultimately produced one ml.g6.4xlarge On-Demand node in us-west-2c, marked Ready and Schedulable, with the s3-claim storage mount present.
Three workflow rules make that interaction more than conversational convenience. The agent polls every asynchronous operation to a terminal state; it asks users only for consequential choices; and it inspects existing resources and valid account-specific options before creating anything. Networking details remain control-plane responsibilities. EKS control-plane subnets are separated from HyperPod compute subnets, with compute subnets sized at /20. A shared ensureComputeSubnet() path uses an explicitly selected subnet, reuses a compatible subnet in the chosen Availability Zone or creates one with a route table and S3 gateway endpoint association.
Capacity choices and resilient defaults
InstantStart enables automatic node recovery when it creates a HyperPod cluster. HyperPod can reboot or replace faulty nodes using its monitoring agent and health checks. Optional deep health checks stress GPUs and Elastic Fabric Adapter connectivity before nodes accept workloads. Findings are also reflected in Kubernetes labels, taints and annotations.
Instance-group creation captures decisions that cannot safely be deferred:
- Capacity type: On-Demand, Spot for fault-tolerant work or reserved capacity through a SageMaker training plan. Training plans are tied to particular Availability Zones, and the control plane reconciles the selected zone with the plan. Capacity type cannot change during the group’s life.
- Network interface mode: supported multi-card instances can use EFA-only interfaces to conserve VPC addresses. This choice is also immutable after creation.
- Subnet placement: groups normally share a per-AZ compute subnet, while large groups can request a dedicated subnet that survives the group and can be reused.
When a group is resubmitted, InstantStart normalizes it through an allowlist instead of rebuilding an update request ad hoc. Fields such as OnStartDeepHealthChecks and NetworkInterface are preserved during unrelated scaling changes.
Managed Karpenter: autoscaling HyperPod groups
HyperPod managed Karpenter adjusts how much capacity runs within a static ownership boundary. AWS operates the controller, and nodes launch from HyperPod instance groups that can scale from zero. The nodes consequently inherit HyperPod monitoring and recovery. InstantStart provisions a standard Karpenter NodePool connected to a HyperpodNodeClass, including consolidation settings that reduce idle groups toward zero. Its scope is limited to HyperPod instance groups, not general-purpose EC2 capacity.
Managed capabilities as reconciled state
The Advanced Features panel exposes the training operator, inference operator, managed tiered checkpointing and managed autoscaling as dependency-aware operations. For tiered checkpointing, InstantStart builds the complete identity chain: Kubernetes service account, IAM role and policy, OIDC trust relationship and binding annotation. Disabling the capability removes that same chain.
An earlier implementation revealed why explicit intent matters. Enabling the inference operator installed cert-manager, but a stale form value could remove it during a later unrelated submission. InstantStart corrected this with an explicit-diff contract: clients send only fields a user touched, while the backend reads actual state and performs no operation when requested and current state already match. The rule applies equally to UI and MCP requests.
Training: two submission paths and a recipe layer
The task layer: training operator or KubeRay
The HyperPod training operator provides process-level recovery, log-pattern-based hang detection and outlier detection. InstantStart installs it as an EKS add-on and submits HyperPodPyTorchJob resources. The documented example sets jobMaxRetryCount to 5, allows 3 process restarts before a full restart, uses an evaluation period of 21,600 seconds—six hours—and permits 1 full-job restart. Containers use hyperpodrun rather than torchrun, while the operator injects topology values including NNODES and NPROC_PER_NODE.
KubeRay is the second task path. It suits Ray-native workloads such as reinforcement learning, where a head node coordinates rollout and training workers. Ray jobs run on the same HyperPod nodes and use the same storage and monitoring surfaces rather than requiring a separate control plane.
The recipe layer: frameworks as configuration
Recipes are available for plain PyTorch, LLaMA-Factory, MS-Swift and VERL reinforcement learning; VERL uses the KubeRay path. A common data contract mounts the same S3 bucket at ~/workspace/s3 in development and /s3</code inside workload pods, allowing scripts or dataset definitions to change without rebuilding an image. Browser logs stream over WebSocket, and recipes can report metrics to managed MLflow on SageMaker AI. InstantStart wires the IAM and service-account path and displays run history, including cross-account sharing governed by fine-grained IAM.
Serving models two ways
The managed inference path gives endpoint lifecycle to the HyperPod inference operator. A declarative configuration specifies the S3 model location, image, invocation port, GPU resources and replicas. Managed tiered KV caching can place L1 cache data in CPU memory and use Redis or SageMaker managed tiered storage for L2. Routing options include prefix-aware, KV-aware, session and round-robin strategies.
The self-managed path deploys a chosen serving container, including vLLM, SGLang or a custom image, as a Kubernetes deployment. It supports an external load balancer, an internal service or a model pool. Warm GPU workers in a pool carry the label business=unassigned; reassignment changes the label instead of recreating the pod and loading weights again. Scale-in refuses to remove assigned workers. Multi-replica SGLang deployments can add cache-aware routing and KEDA autoscaling, with model workers—not the metrics-providing router—as the scale target.
The agent guardrails require explicit GPU allocation rather than deriving it from tensor parallelism. In the source example, a vLLM command used Qwen3-0.6B, a maximum model length of 1280 and tensor-parallel size 1. The agent confirmed that the cluster had one ml.g6.4xlarge node with 1 GPU, requested 1 GPU for the deployment, used a cluster-internal service and tested the model before declaring success.
Storage supports both inference paths. Mountpoint for Amazon S3 is used for read-mostly artifacts, while FSx for Lustre handles high-throughput read/write training data and checkpoints. Hugging Face downloads run as CPU-only Kubernetes jobs, staging files on instance NVMe before copying them to object storage so a GPU does not remain idle during transfer.
What makes the control plane agent-ready
Agent skills encode complete, reviewable workflows, including the six-stage cluster sequence, GPU confirmation and training-plan Availability Zone reconciliation. MCP provides 38 tools spanning cluster lifecycle, instance groups, managed features, storage, downloads, inference, jobs and node operations. Mutating tools identify the status operation used to determine completion.
Because those tools reuse the backend API, agent actions inherit subnet selection, normalization, validation and status persistence. Operations record their phase before polling, retries do not replay mutations, the API response remains the authority for success or failure, and explicit diffs prevent one feature request from expressing unintended changes to others.
Diagnostic skills have a narrower permission policy than provisioning workflows. They investigate without making changes, present state-changing commands for approval and escalate from investigation to reboot and then replacement. IAM, Kubernetes authorization, network controls and backend validation remain the real security boundaries; the agent does not gain privileges merely because it offers another interface.
Observability
The Monitoring page reports node health, total and available GPUs, and live Kubernetes objects including pods, services, deployments, InferenceEndpointConfig and HyperPodPyTorchJob. Agent status tools read the same operational state. At fleet level, HyperPod sends metrics to Amazon Managed Service for Prometheus, with Amazon Managed Grafana dashboards supplied through the observability add-on. Operators can still inspect generated resources, deployments and node metadata directly through AWS and Kubernetes tools.
Considerations
InstantStart reduces integration work, but it does not eliminate architectural decisions or service constraints.
- EKS access, workload authorization, egress and IAM remain the customer’s security responsibilities and should follow least privilege for human and agent credentials.
- Agent skills can affect cost, capacity and availability, so they should be versioned, reviewed and tested as operational code.
- Elastic training currently cannot be combined with Spot Instances, managed tiered checkpointing or checkpointless training.
- Costs can arise across EKS, HyperPod instances, storage, load balancing, managed observability and managed MLflow.
- Cluster-usage quotas and high-end accelerator reservations may need to be arranged before deployment.
- Managed Karpenter scales HyperPod instance groups only, and neither the control plane nor its agent removes the need to understand ownership boundaries.
Conclusion
HyperPod InstantStart’s main contribution is not simply a conversational interface. It is a shared control-plane contract beneath the agent, browser and REST API. Networking rules, dependency reconciliation, field preservation, state persistence and completion checks live in one backend, allowing every interface to inherit the same guardrails. That architecture makes agent-driven operations more dependable while keeping AWS and Kubernetes resources visible to conventional tooling.
About the authors
The source article is credited to Hao Zheng, Ying Hou, PhD, and Anoop Saha of Amazon Web Services, whose work described the project’s architecture, workflows and operational boundaries.
Source: AWS Machine Learning Blog — Run agent-driven Amazon SageMaker HyperPod operations with InstantStart.
Definition. HyperPod InstantStart is an open-source control plane for managing EKS and SageMaker HyperPod infrastructure, capacity, workloads and storage through shared, validated workflows.
| Managed inference | Self-managed inference |
|---|---|
| Lifecycle is handled by the HyperPod inference operator. | A serving container runs as a Kubernetes deployment. |
| Uses a declarative configuration for model location, image, port, GPU resources and replicas. | Supports vLLM, SGLang or a custom image with explicit GPU allocation. |
| Supports prefix-aware, KV-aware, session and round-robin routing. | Supports an external load balancer, internal service or model pool. |
| Can use managed tiered KV caching with CPU memory and Redis or SageMaker managed tiered storage. | Can keep warm GPU workers in a pool and reassign them by changing labels. |
Key takeaways
- The web interface, REST APIs, MCP tools and AI agent use the same backend validation and persisted state.
- Cluster creation is divided into six independently retryable stages, from EKS creation through final verification.
- Managed Karpenter scales capacity within HyperPod instance groups, including scaling groups toward zero.
- Training can use the HyperPod training operator or KubeRay, with recipes for PyTorch, LLaMA-Factory, MS-Swift and VERL.
- Inference supports both HyperPod-managed endpoints and self-managed Kubernetes deployments.
- IAM, Kubernetes authorization, networking and backend validation remain the effective security boundaries.
FAQ
What problem does HyperPod InstantStart solve?
It coordinates the handoffs involved in creating clusters, provisioning accelerator capacity, configuring dependencies and storage, running training, deploying inference and monitoring operations.
How does InstantStart guard AI agent operations?
Agent tools call the same REST endpoints as the web interface, so they inherit backend validation, reconciliation, field preservation, persisted state and completion checks.
Does InstantStart run in the training or inference data path?
No. It runs as an out-of-band management container and creates standard AWS and Kubernetes resources that operators can inspect directly.
How does staged cluster creation handle failures?
Each of the six stages persists its own status and can be retried independently, so a later failure does not undo a successfully created EKS control plane.
Which training paths are supported?
InstantStart supports the HyperPod training operator for HyperPodPyTorchJob resources and KubeRay for Ray-native workloads such as reinforcement learning.
What security responsibilities remain with customers?
Customers remain responsible for EKS access, workload authorization, egress, IAM and least-privilege credentials for both people and agents.
