AWS added model caching to SageMaker HyperPod and measured about 60% faster scale-out

Cold start is the most expensive part of serving a large model behind an autoscaler. Before a new pod answers anything it pulls a container image from ECR and model weights from S3 or FSx, and on a 145 GB model that runs into tens of minutes. Amazon SageMaker HyperPod now has model caching, which takes both downloads out of the start-up path. AWS announced it on September 11, 2026, generally available in every region where HyperPod runs.

Two caches, and a fallback that makes them optional

Model caching has two independent parts. The weights cache stores model weights on the node's local NVMe, so a pod reads them from local storage instead of pulling them over the network from S3 or FSx. The image cache pre-pulls the container image onto cluster nodes, so a starting pod skips the ECR download entirely.

Either cache can miss. AWS says a pod that lands on a node without a warm cache falls back to pulling from the original source automatically, so a miss costs the old start-up time rather than a stuck or failed pod. You enable the feature through the HyperPod Inference Operator, by adding a modelCacheConfig section to an InferenceEndpointConfig or a JumpStartModel resource.

What the numbers say, and what they leave out

AWS reports benchmarks across models from 57 GB to 145 GB: around 60% faster scale-out, and a 97% reduction in image-pull time that removes over two minutes of waiting. It also says the benefit grows with model size, which follows from what the cache replaces, because a download scales with the weights and a local read does not.

The announcement does not describe the test. It names no models, no instance types, no baseline in seconds, and no method for telling a cold node from a warm one, so the figures are a vendor measurement rather than an independent one. Pricing is absent too, so the announcement does not say what holding weights on local NVMe costs.

Bedrock knowledge bases stop depending on the transcript

Amazon Bedrock Managed Knowledge Base could already search media by turning audio and video into text and embedding that text. TwelveLabs Marengo 3.0, available in the managed knowledge base from the same day, instead encodes visual scenes, speech and video cues directly into multimodal embeddings, which AWS describes as capturing meaning that transcription alone cannot.

The vectors are 512-dimensional, and a result carries the start and end time of the matching segment, so an application can jump to the moment instead of to the file. Segmentation is configurable. AWS calls the retrieval accuracy state of the art without naming a benchmark, and the announcement lists neither regions nor prices.