Amazon SageMaker Inference has introduced prefix-aware routing for real-time endpoints, sending requests with matching prompt beginnings to the same instance so LLM applications can reuse cached computations instead of repeatedly processing shared context. For teams operating multi-instance endpoints, AWS says this can lower time to first token and improve throughput without requiring request tagging or changes to invocation APIs.
Prefix caching stores the key-value pairs already computed for a prompt prefix, allowing the model to process only the new portion of a subsequent request. That benefit can erode across a fleet when conventional routing sends similar prompts to different machines, leaving each instance to recompute the same instructions, retrieved document or conversation history. SageMaker’s new strategy examines the beginning of the request payload and uses it to keep matching traffic on an instance whose cache is warm.
Safeguards and routing choices
The endpoint can override affinity when the preferred instance reaches a configurable concurrency threshold, directing the request to a less busy machine. This protects the instance from overload at the cost of a possible cache miss. AWS also says the routing assignment remains mostly stable when instances are added or removed, limiting cache disruption during scaling.
Prefix-aware routing joins two existing strategies. The default random option distributes requests uniformly, while least-outstanding-requests selects the instance with the fewest requests in flight. The new option is intended for LLM workloads with repeated prompt beginnings and prefix caching enabled. Administrators select a strategy for each production variant and can change it through an updated endpoint configuration without redeploying the model.
AWS benchmark results
AWS compared prefix-aware routing with random routing using Llama 3.1 70B Instruct, vLLM with prefix caching enabled and seven ml.p5.48xlarge instances. The evaluation comprised 16 configurations spanning single-model and inference-component endpoints as well as the native Invoke API and OpenAI-compatible API; AWS reported a 100% success rate across the tests.
For workloads with an 8,000-token shared prefix sustained for one hour, AWS measured P50 time-to-first-token reductions of 71%–77% and P90 reductions of 33%–37%. KV-cache hit rates rose from approximately 25% to 82%, while throughput increased 15%–16%.
The gains were smaller in 30-minute tests using variable-length, ShareGPT-style conversations. P50 time to first token fell 13%–16%, P90 fell 24%–37%, cache hit rates moved from approximately 30% to 80%, and throughput rose 1.7%–2.0%. The comparison indicates that workloads with longer shared prefixes benefit more because each cache hit avoids more computation; these figures are AWS benchmark results for the stated configurations, not general performance guarantees.
AWS measured routing overhead of 1.3–1.9 milliseconds per request against model time to first token of 63–280 milliseconds. Traffic also remained distributed across the fleet: each of the seven instances handled 13.3%–15.4% of requests, within one percentage point of an even allocation.
Workloads and configuration
The strategy is aimed at patterns where requests repeatedly begin with the same material. In retrieval-augmented generation, several questions can share a retrieved document; multi-turn chats repeatedly include prior conversation history; templated assistants reuse instruction blocks; and code-completion requests can share file contents while a developer works in the same file.
Two endpoint settings govern the feature. PrefixLength, ranging from 1,024 to 65,536, specifies how much request content participates in routing. It counts raw request-body bytes for SageMaker’s native Invoke API but characters from extracted message text for the OpenAI-compatible API. ConcurrencyThreshold, ranging from 1 to 1,024, sets the maximum number of in-flight requests on the selected instance before overflow routing takes effect.
The routing layer does not require model-container changes, and existing InvokeEndpoint, InvokeEndpointWithResponseStream and OpenAI-compatible Chat Completion calls remain available. Prefix-aware routing also supports inference-component endpoints and dynamic LoRA adapters. For LoRA, prefix-based selection occurs among the instances in the adapter’s existing sticky set.
Operational caveats
The serving framework must have prefix caching enabled; routing related requests together does not itself store or reuse KV pairs. At least two instances are required for the routing choice to make a difference.
Native API users also need consistent request serialization because whitespace, JSON key order and formatting change the raw bytes used for routing. A prefix length that is too short can concentrate traffic and trigger overflow, while one that is too long can let minor payload differences separate requests that should share a cache. AWS recommends monitoring model-level cache-hit rates through SageMaker detailed observability to validate results for the actual workload.
For tenant isolation, native API clients can supply an X-Amzn-SageMaker-Prefix-Aware-Id header of up to 64 ASCII characters, while OpenAI API clients can use prompt_cache_key. SageMaker combines that identifier with the prefix so otherwise identical prompts from different tenants can be routed separately.
The feature is available on SageMaker real-time inference endpoints. Access to the RoutingStrategy and PrefixAwareRoutingConfig parameters requires an updated AWS SDK or CLI. Source: AWS Machine Learning Blog.
Definition. Prefix-aware routing directs requests with matching prompt beginnings to instances likely to hold reusable cached computations.
| Routing strategy | Behavior |
|---|---|
| Prefix-aware | Routes matching prompt beginnings to a likely warm-cache instance, with overflow at the configured concurrency threshold. |
| Random | Distributes requests uniformly across endpoint instances. |
| Least outstanding requests | Selects the instance with the fewest requests currently in flight. |
Key takeaways
- Prefix-aware routing keeps requests with matching beginnings on an instance whose cache is warm.
- AWS reported P50 time-to-first-token reductions of 71%–77% for workloads with an 8,000-token shared prefix.
- Variable-length conversation tests produced smaller throughput gains of 1.7%–2.0%.
- The serving framework must have prefix caching enabled, and at least two endpoint instances are needed for routing to matter.
- PrefixLength and ConcurrencyThreshold control prefix matching and overflow routing.
- Native API clients need consistent serialization because formatting differences change the request bytes used for routing.
FAQ
What is prefix-aware routing in Amazon SageMaker?
It examines the beginning of a request and sends matching traffic to an instance likely to have cached computations for that prefix.
Does prefix-aware routing enable prefix caching?
No. Prefix caching must already be enabled in the serving framework; the routing strategy only improves the chance that related requests reach a warm cache.
Which workloads benefit most?
Workloads with repeated prompt beginnings, such as retrieval-augmented generation, multi-turn chat, templated assistants and code completion, are the intended use cases.
What performance gains did AWS report?
In its 8,000-token shared-prefix tests, AWS reported P50 time-to-first-token reductions of 71%–77%, P90 reductions of 33%–37% and throughput gains of 15%–16%.
What settings control prefix-aware routing?
PrefixLength determines how much request content participates in routing, while ConcurrencyThreshold controls when traffic overflows to a less busy instance.
Can administrators change the routing strategy without redeploying the model?
Yes. Administrators can select a strategy per production variant and change it through an updated endpoint configuration.
