Amazon Web Services has launched model caching for Amazon SageMaker Inference on HyperPod, preloading model weights and container images onto cluster nodes before inference pods need them. For teams scaling large-model deployments, AWS says pods on cache-ready nodes can start serving in seconds instead of waiting tens of minutes for network downloads.
Without caching, every new pod sequentially pulls its inference-server image from Amazon Elastic Container Registry and retrieves model weights from remote storage. AWS says multi-gigabyte vLLM or LMI images can take 5–7 minutes to pull. A 145 GB model stored in Amazon S3 can then take more than 20 minutes to download, depending on network conditions and available bandwidth, while a model larger than 600 GB, such as DeepSeek-R1, can take more than 30 minutes.
That sequence also delays autoscaling. A HorizontalPodAutoscaler might request five new pods within seconds, but AWS says the additional capacity can remain unavailable for 25–30 minutes or longer because every pod independently waits for its image and weights.
Separate caches for weights and images
HyperPod offers weights and image caching as independent options that can be enabled separately or together. The weights cache downloads model data from Amazon S3, Amazon FSx for Lustre, HuggingFace Hub or SageMaker JumpStart to local NVMe storage on each target node. The HyperPod Inference Operator marks nodes as cache-ready after their downloads finish and waits for all target nodes before creating the inference deployment. Pods can then read weights locally at a typical speed of approximately 7 GB/s, and the cache persists across pod restarts on the same node.
The image cache uses a DaemonSet to pre-pull the inference-server container onto target nodes. It does not delay creation of the inference deployment: a pod skips the ECR pull when its node already has the image, but pulls from ECR normally if caching there is incomplete. AWS says a warm image cache can save the cited 5–7-minute pull. Deployments using the same image share one cache resource, which remains until no deployment references it.
Both features use preferred rather than required scheduling. Pods favor nodes containing the relevant cache but are allowed to run elsewhere. If a rapid scale-out exceeds the number of warm nodes, affected pods fall back to the original weights source and ECR, incurring the ordinary download time instead of failing.
Configuration and lifecycle
Users enable caching by adding modelCacheConfig to an existing InferenceEndpointConfig or JumpStartModel resource. Separate weightsCache and imageCache settings control the capabilities, and AWS says no additional infrastructure setup is required. Weight caching can also use an optional hostPath override instead of the default /opt/dlami/nvme mount.
The operator creates and manages two custom resources. ModelDataCacheConfig handles weight downloads, readiness labels, health monitoring and file removal. ModelImageCache manages image pulls, per-node status and cleanup. When the model source or image reference changes in the specification, the operator creates a replacement cache, rolls out the updated deployment and removes the old cache afterward.
Both cache types support Amazon S3, Amazon FSx for Lustre, HuggingFace Hub, and gated and non-gated SageMaker JumpStart models. The feature is generally available in every region where SageMaker HyperPod is available.
AWS benchmark results
In AWS benchmarks covering models from 57 GB to 145 GB, weights caching produced around 60% faster scale-out. AWS also reports that image caching removed more than two minutes of cold image-pull time and reduced that time by as much as 97% compared with pulling a fresh image from ECR for every pod. These are AWS-reported results, and the practical gain depends on the model size, network conditions and whether the scheduler places a pod on a cache-ready node.
Storage and refresh limitations
The faster starts do not eliminate the initial transfer. Populating a weights cache for the first time still requires downloading the complete model from its remote source; the benefit begins only after that work finishes. Because every node stores its own copy, total NVMe consumption also increases with the number of target nodes.
Each instance must have enough local capacity for the model. AWS lists 250 GB of NVMe for ml.g5.xlarge, 3,800 GB for ml.g5.12xlarge, 7,600 GB for ml.g5.48xlarge, 8,000 GB for ml.p4d.24xlarge and 30,000 GB for ml.p5.48xlarge. A 300 GB model therefore cannot be cached on an instance with only 250 GB.
HyperPod also does not automatically detect weights replaced at the same S3 path. If the files change without a corresponding update to the InferenceEndpointConfig specification, the operator continues serving the cached version. Users must change the specification—for example, by using a new model path or version suffix—to trigger creation of a fresh cache.
Deleting the parent InferenceEndpointConfig or JumpStartModel causes the operator to remove its cached data, DaemonSets and node labels, freeing the associated NVMe storage.
Source: AWS Machine Learning Blog
Definition. SageMaker HyperPod model caching preloads model weights and inference-server images onto cluster nodes so pods can use local data instead of downloading it at startup.
| Weights cache | Image cache |
|---|---|
| Downloads model data to local NVMe storage | Pre-pulls the inference-server container onto target nodes |
| Supports Amazon S3, Amazon FSx for Lustre, HuggingFace Hub and SageMaker JumpStart | Uses a DaemonSet to manage image pulls |
| Waits for all target nodes before creating the inference deployment | Does not delay creation of the inference deployment |
| Persists across pod restarts on the same node | Can be shared by deployments using the same image |
| Falls back to the original weights source on an uncached node | Falls back to a normal ECR pull when caching is incomplete |
Key takeaways
- Weights and image caching are independent options that can be enabled separately or together.
- Cached weights are stored on each target node's local NVMe storage and persist across pod restarts on that node.
- Preferred scheduling lets pods use uncached nodes when warm capacity is exhausted, falling back to remote weights and ECR downloads.
- The first cache population still requires a complete model download, and every target node needs enough local storage.
- Weights replaced at the same S3 path are not detected automatically; users must update the specification to refresh the cache.
- The feature is generally available wherever SageMaker HyperPod is available.
FAQ
How does SageMaker HyperPod model caching reduce cold starts?
It downloads model weights and container images to cluster nodes before inference pods need them, allowing cache-ready pods to use local data instead of waiting for remote downloads.
Can weights caching and image caching be enabled separately?
Yes. Separate weightsCache and imageCache settings allow either capability to be enabled alone or both to be used together.
What happens when no cache-ready node is available?
Pods may run on other nodes and fall back to the original model source and Amazon ECR, incurring ordinary download time rather than failing.
Does caching eliminate the first model download?
No. The complete model must still be downloaded when the weights cache is initially populated; the acceleration applies afterward.
How are changed model weights refreshed?
HyperPod does not automatically detect weights replaced at the same S3 path. The specification must change, such as by using a new path or version suffix, to create a fresh cache.
What storage limitation applies to weights caching?
Each target instance must have enough local NVMe capacity for the model, and every node stores its own copy.
