deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

AWS SageMaker inference router now fails over AI requests across regions and accounts

Cross-region inference routing is now generally available in SageMaker AI. The router shifts requests to endpoints in other AWS regions and accounts after faults, throttles or timeouts, as shown in Salesforce's failover test.

AWS SageMaker inference router now fails over AI requests across regions and accounts

What shipped

AWS made cross-region inference routing generally available in SageMaker AI on 18 November 2026, in every commercial region where the service runs real-time inference. According to a dev.to summary of an AWS Machine Learning Blog post, the inference router is a regional AWS resource that receives an application's inference calls and forwards each request to one of the endpoints registered for that model — and those endpoints can now sit in other regions and in other AWS accounts.

The launch customer is Salesforce's Agentforce Foundation team, which serves models from the region a request originates in, across more than thirty global regions and hundreds of accounts. The economics motivating the feature are straightforward, as dev.io describes them: each customer has its own peak window, those windows overlap, and provisioning for the combined peak costs more than the traffic justifies, while per-region provisioning leaves idle instances in one place as another runs short.

How a routing target is defined

Applications invoke the router with a routing-target name and the same body and content type they already send to InvokeEndpoint. A routing target names one logical model, such as a classifier, and lists by full Amazon Resource Name the endpoints and inference components that may serve it. One router holds one target per model, so a problem with one model's destinations leaves the others untouched, and a target can mix single-model endpoints with inference components.

Because destinations are identified by ARN, they can live in a separate account. The owning account attaches a resource-based policy allowing the router's account to invoke the endpoint, with a condition limiting access to one named router. The endpoint owner keeps the ability to revoke access, and the consumer never needs credentials that span the whole account.

When the router moves traffic, and when it does not

Health is judged from recent request outcomes. Platform faults, 5xx responses from the model container, throttles and timeouts lower a destination's standing, and as failures persist the router sends a growing share of new requests elsewhere — typically starting within about a minute, though a destination with little traffic can take several minutes. After recovery, traffic returns more slowly than it left, which keeps the system from oscillating.

Two exclusions carry most of the design intent, per the dev.to write-up. Client errors such as invalid input move nothing, because the same request would fail identically anywhere. Model latency alone also moves nothing, since a slow answer often reflects a long prompt rather than endpoint health. Where a destination never ran the model — a throttled request, for example — the documentation limits the router to one retry on another destination.

What Salesforce's failover test measured

Salesforce tested two configurations: same-region routing inside us-west-2, and a cross-account, cross-region setup where the us-west-2 router also listed a us-east-2 endpoint owned by a separate infrastructure account. Before sending production traffic, the team called DescribeInferenceRouter and confirmed both destinations reported a routable state.

The test saturated the local endpoint and watched with a separate probe. Probe p50 latency rose from 0.2 seconds to between 45 and 58 seconds over roughly five minutes, but traffic stayed in us-west-2 because slow-but-successful responses count as healthy. Only when requests exceeded the model container's 60-second handler timeout and began returning errors did the router act. The first requests reached us-east-2 about two minutes after the errors began, and within another minute roughly 78 percent of probe traffic was served there. The remaining 20 to 25 percent stayed local, supplying the request outcomes the router needs to detect recovery.

After the shift, p50 latency returned to 0.2 seconds, and the cross-region path added under 100 milliseconds at p50. Once the local endpoint recovered, traffic returned to us-west-2 gradually over about ten minutes, and the probe recorded no errors during failover or failback. Salesforce plans a longer saturation window, millisecond-precision latency measurement, and a closer look at cascading behaviour at the traffic cap before widening the rollout.

Observability and guardrails

Every response carries the invoked resource ARN, InvokedRoutedResourceArn, plus a routing reason, and the router publishes per-destination CloudWatch metrics — turning a traffic shift into something a team can explain after the fact. A CrossRegionTrafficConfig cap limits inbound remote traffic per endpoint, protecting the receiving region from being overloaded during an incident. Model deployment and capacity remain the platform team's responsibility: each region still needs the same artifacts and its own autoscaling headroom.

Why it matters

Cross-region failover for inference was previously something teams built and operated themselves. Salesforce ran its own internal gateway with health checks, retries and region-switching logic, had no capacity signal from either endpoint, and had to push cross-region traffic out through a NAT gateway over public networking. The router replaces that with a managed, policy-scoped primitive that comes with built-in observability and a ceiling on inbound failover traffic. The caveats are real: slow responses alone will not trigger a shift, recovery detection deliberately keeps a fifth of traffic on the impaired endpoint, and capacity planning does not disappear. But for production AI workloads where a regional fault or throttle storm would otherwise surface as user-facing errors, this is a practical reliability feature rather than a benchmark claim.

  • #aws
  • #sagemaker
  • #machine-learning
  • #reliability
  • #cloud

Related posts