Tracing Request Placement in SGLang's Rust Router

How the router uses engine state to place requests, and what changes when migrating from sgl-model-gateway.

Suppose three SGLang engines serve the same model. Engine A has cached most of an incoming prompt but is busy. Engine B has an empty queue but would need to prefill the entire prompt. Which one should receive the request?

SGLang's sgl-router is a Rust-based service that sits between clients and a fleet of inference engines serving the same model. It does more than forward HTTP requests. Depending on the routing policy, it considers engine load, queued requests, and cached KV prefixes to decide where a request should run. The engines can handle both prefill and decode, or operate as separate prefill and decode workers. The router handles request preparation, placement, and response forwarding, while each engine manages its own GPU scheduling and KV cache. In P/D disaggregation, KV data moves directly between engines rather than through the router.

I ran into the placement question above while preparing SGLang router migration guide PR #51773. The new Rust router changes more than CLI flags: it learns engine capabilities through introspection and uses reported load and KV cache events for placement. I traced a request through the reorg path to understand how those pieces fit together.

This article uses SGLang's November 2026 main implementation and compares it with sgl-model-gateway v0.5.23. Some options on main are not yet in published releases. All workload numbers below are illustrative, not benchmark results.

1. What information does the router need before a request arrives?

The router chooses where a request goes; the engine scheduler decides when it enters a batch and runs on the GPU. Keeping the two apart matters once engines differ in queue length and cached prefixes, because the router has to weigh those differences before the scheduler ever sees the request.

I found it helpful to separate three activities by when they happen:

WhenWhat happensRouter state produced
Startup or topology changeDiscover and register enginesWorker Registry
While servingEngines publish load and KV eventsLoad Table and Prefix Index
On each requestPrepare, select, and dispatchSelected Worker(s)

A client request normally uses existing state rather than triggering new discovery. The animation follows those three paths on the same deployment, so you can see when each message is sent.

Figure 1

Three flows: when, how often, and which direction

1 / 6

T0 · Worker discovered

Worker Registryempty—
Load tableempty—
Prefix indexempty—

T0Engine discovery

DiscoveryEvent::Added → register_one()
GET /model_info /server_info
model · role · DP · endpoints

T1State reporting

ZMQ LoadStat
ZMQ BlockStored / BlockRemoved

T2Request routing

POST /v1/chat/completions
Bucket → Policy + Admission reads the 3 stores
forward to Engine C
tokens stream back
Each worker URL becomes a DiscoveryEvent::Added. The Router knows only an address.
HTTPLoadStatKV events

Simplified from sgl-router (Nov 2026). Numbers are hypothetical, not measured.

Text version: all steps
  1. T0 · Worker discovered: Each worker URL becomes a DiscoveryEvent::Added. The Router knows only an address.
  2. T0 · Router asks, engine answers: The Router queries each engine once and saves its profile to the Registry.
  3. T1 · Engines publish load: Engines publish load continuously; the Router only subscribes.
  4. T1 · Engines publish KV events: A second stream carries block hashes and owners, never KV tensors.
  5. T2 · A request arrives: The request reads state that already exists. It does not trigger discovery.
  6. T2 · Dispatch and respond: Engine C runs it and streams tokens back. Load reports keep arriving.

Tip: focus the figure and use ← → to step.

2. How the router registers engines

Start with three static engine URLs:

sgl-router \
  --model-id Qwen/Qwen3-0.6B \
  --worker-urls http://engine-a:30001 http://engine-b:30001 http://engine-c:30001 \
  --chat-routing reorg --policy power_of_two \
  --host 0.0.0.0 --port 30000

The URLs identify where to connect, but not what each engine can do. Static discovery emits a DiscoveryEvent::Added for each worker; WorkerManager::register_one() then uses WorkerIntrospector to fetch /model_info and /server_info. Those responses identify the served model, plain/prefill/decode role, DP ranks, and optional load/KV publisher endpoints.

The Registry indexes workers by identity and model:

by_id: DashMap<WorkerId, Arc<Worker>>,
by_model: DashMap<ModelId, HashSet<WorkerId>>,

The registry answers only which engines serve the model. Which of them receives a particular request is decided when that request arrives, by bucket selection, admission and policy, using the load and KV state described in the next section.

The migration implication is immediate: the old gateway accepted --pd-disaggregation, --prefill, and --decode; the new router can learn roles and bootstrap information from the engines' /server_info. Those roles are fixed when each engine starts and the router only reads them, so the engines still need to be launched with the correct --disaggregation-mode.

Source: Worker Manager · Introspection

3. How load and KV events reach the router

After registration, engines can publish two distinct streams over ZeroMQ:

Engine sourceMessageRouter destinationUsed for
SchedulerLoadStatEngineReportedLoadTableComparing load, admission
KV cacheBlockStored, BlockRemoved, AllBlocksClearedPrefix Index / HashTreeFinding cached prefix owners

Both streams are published by the engines and subscribed to by the router, so the router doesn't have to query an engine while a request waits. A load report might contain waiting requests, uncached prefill work, and KV usage. The prefix index records only which engine reports owning a chain of cached blocks; the GPU KV tensors themselves stay on the engines and never move to the router.

Because these reports arrive asynchronously and may be missing or stale, the router also keeps its own count of in-flight requests it has dispatched but not yet seen complete.

Watch how a LoadStat updates the load table while a BlockStored event updates the prefix index. The two streams feed different parts of the same routing decision.

Figure 2

LoadStat and KV events: two separate ZMQ streams

1 / 6

Load · the scheduler measures

LoadStat

  1. ENGINE

    Scheduler

    waiting · KVA · 12 · 50%B · 4 · 70%C · 6 · 60%
  2. ZMQ PUB → SUB
  3. ROUTER

    EngineReportedLoadTable

    waiting · KVA · —B · —C · —

KV events

  1. ENGINE

    KV cache (A)

  2. ZMQ PUB → SUB
  3. ROUTER

    Prefix Index

Policy

No KV tensors pass through the Router.

Each scheduler knows its own queue and KV usage. The Router doesn't, yet.
KV tensor (GPU)hash in router

Simplified from sgl-router (Nov 2026). Numbers are hypothetical, not measured.

Text version: all steps
  1. Load · the scheduler measures: Each scheduler knows its own queue and KV usage. The Router doesn't, yet.
  2. Load · LoadStat over ZMQ: LoadStat goes Engine PUB → Router SUB and fills the load table.
  3. Cache · engine A stores a prefix: Engine A keeps 8 prefix blocks as KV tensors in GPU memory.
  4. Cache · BlockStored on a second stream: Only the hashes cross. The index records A as their owner.
  5. Cache · eviction and BlockRemoved: A evicts h7 and h8; BlockRemoved drops them from the index.
  6. Both streams feed the policy: Policy reads both: a load snapshot and prefix ownership.

Tip: focus the figure and use ← → to step.

Source: KV Event Index

4. Following a request through bucket selection and admission

Consider a chat request with a long shared system prompt:

{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [
    {"role": "system", "content": "A long shared knowledge base..."},
    {"role": "user", "content": "Explain KV cache architecture."}
  ],
  "max_tokens": 256,
  "stream": true
}

Assume the rendered prompt is 4,096 input tokens. The router builds a PreparedRequest containing model, token counts, expected context needs and, for cache-aware routing, token IDs and cache namespace information. This step runs on the router's CPU and only produces metadata for bucket selection, admission and policy; the GPU computes prefill for these 4,096 tokens later, after the request reaches the selected engine and its scheduler puts it in a batch.

BucketResolver first finds compatible engine groups using request length and any configured SLO preferences. Without a custom bucket file, our three plain engines belong to the same eligible group. Next, admission checks whether each engine can accept more work.

For example, configure --max-waiting-requests 10 --max-kv-usage 0.9 and assume the following reports:

EngineWaitingKV usageAdmission
A1250%Reject
B470%Allow
C660%Allow

With power_of_two, only admitted workers B and C are compared. If their estimated prefill queue times are 300 ms and 150 ms respectively, C wins this illustrative comparison. Admission only filters candidates using the latest reports. It doesn't reserve GPU memory or a queue slot on the engine, and because reports update asynchronously, C's load may have changed by the time the request arrives.

The step-through animation shows how a request moves from preparation to bucket, admission, policy and dispatch. Notice that rejected Engine A never enters the subsequent Power-of-Two comparison.

Figure 3

How one request ends up on Engine C

1 / 6

1. Prepare

  1. 1Prepare
  2. 2Bucket
  3. 3Candidates
  4. 4Admission
  5. 5Power-of-Two
  6. 6Dispatch

PreparedRequest

model
Qwen/Qwen3-0.6B
input
4,096 tok
output
256 tok
expected
4,352 tok
bucket
…

Admission limits--max-waiting-requests 10--max-kv-usage 0.9

Engine A

waiting
—
KV
—
est. queue
—

Engine B

waiting
—
KV
—
est. queue
—

Engine C

waiting
—
KV
—
est. queue
—
Parse and tokenize: 4,096 in + 256 out. No GPU work yet.

Simplified from sgl-router (Nov 2026). Numbers are hypothetical, not measured.

Text version: all steps
  1. Prepare: Parse and tokenize: 4,096 in + 256 out. No GPU work yet.
  2. Bucket: BucketResolver finds a compatible bucket: A, B and C.
  3. Candidates: Each candidate comes with its latest reported load.
  4. Admission: A has 12 waiting, over the limit of 10: rejected. Nothing is reserved.
  5. Power-of-Two: Sample two admitted engines (B, C) and compare estimated queue time.
  6. Dispatch: C's queue is shorter, so C gets the request.

Tip: focus the figure and use ← → to step.

Source: Reorg Selection · Power-of-Two

5. How cached prefixes change the decision

Replay the same 4,096-token request. This time all engines pass admission, but their cached prefixes differ. Assume a block size of 128 tokens:

EngineCached prefixBlocksNew prefill work
A3,072 tokens241,024 tokens
B004,096 tokens
C1,024 tokens83,072 tokens

The router hashes the request's token blocks using the same rules as the engines' KV events, then looks up the longest prefix that matches continuously from the start of the prompt; identical tokens that appear later in the prompt, after the first mismatch, don't count toward the match.

The key selection code in CacheAwarePolicy::pick() includes:

// Capture load after remote I/O; selection and admission share it.
let load = self.engine_load.capture_snapshot(Instant::now());
let uncached = uncached_by_url(signal.as_deref(), request.input_tokens);
let candidates = self.candidates(engines, request, &uncached, &load);
let mut rejections = Vec::new();
let admitted = self.admit(&candidates, &load, &mut rejections)?;

Here signal comes from a prefix lookup that may use a local index or a remote KV indexer. Load is captured after lookup, so a remote query doesn't leave the policy comparing an unnecessarily old load snapshot, and the same snapshot is then used for admission, so even the engine with the longest match is dropped if it exceeds the capacity limits. By default, cache-aware candidates require at least 1,024 matched tokens; an optional match-ratio threshold can narrow them further.

The final tradeoff depends on affinity mode. prefer selects the best admitted prefix owner. balanced also considers a load-aware alternative and switches only when the owner's effective load exceeds both a multiplicative threshold and an absolute gap.

For the default prefill-tokens comparison, suppose A has 10,000 waiting uncached tokens and B has 1,000:

Engine AEngine B
Existing waiting uncached work10,0001,000
New request's uncached work1,0244,096
Effective prefill work11,0245,096

With the default factor of 2 and gap of 1,024 tokens, A exceeds both thresholds relative to B. If B is the sampled alternative and both pass admission, balanced can choose B. A would prefill only 1,024 new tokens, but 10,000 tokens of work are already queued ahead of it, so the request may reach its first token sooner on B even though B recomputes the whole prompt.

The animation walks the request from prefix lookup to the final choice. It shows why prefer takes A, then how balanced computes effective work on both engines and checks the two thresholds before moving the request to B. The last step replays the request with only 2,000 tokens queued on A; neither condition holds, so the request stays on A.

Figure 4

Cache reuse vs. queueing

1 / 5

1. Prefix lookup

4,096 tokens = 32 blockshypothetical
Engine Acached 3,072
Engine Bcached 0
Engine Ccached 1,024
cached blockblock to prefill
The router hashes the request's 32 blocks and finds the longest prefix each engine holds from the start: A 3,072, C 1,024, B none.

Simplified from sgl-router (Nov 2026). Numbers are hypothetical, not measured. A worked example, not a full CacheAwarePolicy simulation.

Text version: all steps
  1. Prefix lookup: The router hashes the request's 32 blocks and finds the longest prefix each engine holds from the start: A 3,072, C 1,024, B none.
  2. prefer: A and C both reach the 1,024-token minimum; A has the longest match, so it becomes the owner. In prefer mode the request goes to A without looking at load.
  3. balanced: effective work: In balanced mode the router also samples a load-aware alternative, here B, and adds each engine's queued work to what this request would prefill there.
  4. balanced: both thresholds: A's 11,024 is more than twice B's 5,096 and more than 1,024 above it. Both conditions hold, so the request moves to B and gives up A's 3,072 cached tokens.
  5. Shorter queue on A: Replay the same request with only 2,000 tokens queued on A. A's 3,024 is below B's 5,096, so neither condition holds and the request stays on the cache owner.

Tip: focus the figure and use ← → to step.

Source: Cache-aware Policy · Affinity

6. Why can the router's prefix index lag behind the engine?

The router doesn't ask every engine which KV caches it holds for each incoming request. Instead, it maintains a Prefix Index using KV events published by the engines.

Because these events are asynchronous, the router's information may temporarily be out of date.

For example, Engine A may have already cached prefix P, but its BlockStored event hasn't reached the router yet. If a new request uses the same prefix, the router might not know that A has it and could send the request to Engine B instead, causing B to repeat the prefill computation.

The reverse can also happen. If Engine A evicts a cached prefix before its BlockRemoved event arrives, the router may still believe A has it.

This is called Eventual Consistency: the router's Prefix Index doesn't always reflect the engine's actual cache state immediately, but it updates as events arrive.

SGLang provides mechanisms to reduce these gaps, including Sequence Gap Replay for missing events, Pending-prefix Tracking for prefixes not yet confirmed by KV events, and Peer Bootstrap to help new router replicas initialize their Prefix Index.

The animation below shows how a request can miss a cache reuse opportunity when an engine already holds the prefix, but the router hasn't received the corresponding event yet. You can also switch to the router-restart scenario: even when the engines' caches are still warm, a new router's index starts empty.

Figure 5

KV event lag: what the engine has vs. what the router believes

1 / 5

t1 · Engine A stores prefix P

t1out of sync

Engine A KV cache (actual)P cached ✓
KV event streamBlockStored published
Router prefix index (view)P: no owner
Requests—

t2

Engine A KV cache (actual)·
KV event stream·
Router prefix index (view)·
Requests·

t3

Engine A KV cache (actual)·
KV event stream·
Router prefix index (view)·
Requests·

t4

Engine A KV cache (actual)·
KV event stream·
Router prefix index (view)·
Requests·

t5

Engine A KV cache (actual)·
KV event stream·
Router prefix index (view)·
Requests·
Engine A caches P and publishes BlockStored.
in syncout of sync

Simplified from sgl-router (Nov 2026). Numbers are hypothetical, not measured. No real delay is implied. Replay, pending-prefix tracking and peer bootstrap narrow some gaps.

Text version: all steps
  1. t1 · Engine A stores prefix P: Engine A caches P and publishes BlockStored.
  2. t2 · Event in transit: The event is in flight; the two views disagree.
  3. t3 · A matching request arrives: A request with P arrives; the lookup misses.
  4. t4 · Load-aware fallback: Load-aware fallback may pick B, which may recompute P.
  5. t5 · Index catches up: The event lands; later requests with P can match A.

Tip: focus the figure and use ← → to step.

Source: KV Event Subscriber · Reference

7. What happens after the router selects a worker?

For a plain engine, the router forwards the request to one worker; that worker schedules both prefill and decode. With prefill/decode (PD) disaggregation, the router selects a pair, which can be represented as:

SelectedWorkers {
    prefill: prefill_engine,
    decode: Some(decode_engine),
    // Other fields omitted
}

The router coordinates dispatch to the two workers, but KV data transfers directly between prefill and decode engines, through the configured transfer backend rather than through the router. The animation contrasts the unified and PD paths.

Figure 6

Unified vs. PD: KV data bypasses the Router

1 / 5

Roles come from the engines

Client
Routerplacement + dispatch
Prefill Pmode=prefill
Decode Dmode=decode
Roles come from --disaggregation-mode, read via /server_info.
request / tokens (via Router)KV tensors (P → D)

Simplified from sgl-router (Nov 2026). Numbers are hypothetical, not measured.

Text version: all steps
  1. Roles come from the engines: Roles come from --disaggregation-mode, read via /server_info.
  2. Router selects P and D: SelectedWorkers { prefill: P, decode: Some(D) }
  3. Coordinated dispatch: The request and bootstrap info go to both P and D.
  4. KV transfer, P → D directly: P prefills; the KV tensors move straight to D. The Router is idle.
  5. D decodes, tokens stream back: D decodes; tokens go D → Router → Client.

Tip: focus the figure and use ← → to step.

This also explains why new --worker-urls can replace old gateway flags like --pd-disaggregation, --prefill, and --decode. The router learns roles through introspection, while the engines themselves still require correct PD launch settings. Forwarding failures may be retried on other workers before a response reaches the client, subject to configured retry limits; an already-started streaming response isn't replayed.

Source: PD Disaggregation

8. Migration details that can change production behavior

Several differences affect deployments even when client requests stay the same:

ConfigurationGateway v0.5.23sgl-router
Default policycache_awarelegacy + round_robin; reorg defaults to power_of_two
Model and bind addressModel could be inferred; 0.0.0.0--model-id required; 127.0.0.1
Attempts / timeout5 / 1,800 s1 / 300 s
Body size limit512 MiB32 MiB
Client API-key validationGateway-enforcedNot enforced by router
Cache-aware informationGateway prefix approximationEngine KV events

Three items deserve explicit attention during cutover.

Authentication. --worker-api-key authenticates the router's own engine requests, such as /server_info. It doesn't protect router-owned endpoints or replace gateway client authentication. Forwarded inference requests retain the client's Authorization for engines that enforce an API key, so if clients must be authenticated, a trusted ingress or authentication layer in front of the router has to do it.

Cache-aware prerequisites. Engines need --kv-events-config; --load-publish-endpoint auto supplies optional load signals for fallback, balanced affinity and reported admission limits. Old --cache-threshold 0.3 is not equivalent to new --cache-affinity-min-match-ratio 0.3 because the cache signal and selection logic differ.

Reliability and operations. Review retries, timeouts, the body-size cap, metrics (smg_* to sgl_router_*), and health checks. Static worker URLs don't provide the gateway's active worker polling; the new router uses circuit breakers for certain failures. For the full before/after commands, DP-aware routing and unsupported endpoints, use migration guide PR #51773.

9. How I would validate the migration and measure placement

I'd validate registration first: /readyz, model names, PD roles, DP ranks, and publisher endpoints. Then I'd test regular, streaming, long-context and representative multimodal requests. For placement, I'd check which engines or ranks actually receive them, not just whether the Router CLI parses.

Next I'd test an unavailable engine, backpressure, timeout and failures after streaming begins. Authentication, ingress TLS, body limits and metrics scrape targets also need attention. PR #51773 reports source and CLI validation, not live GPU, RDMA or Kubernetes integration testing.

For performance, I want to compare power_of_two, cache-aware prefer, and cache-aware balanced under controlled prefix reuse and queue imbalance. I'd measure TTFT (p50/p95), throughput, uncached prefill work and actual reuse, then introduce KV event delay to see how stale index information affects placement. Every run needs the same tokenizer, chat template and cache namespace; otherwise prefix hashes differ and cache reuse can't be compared. These experiments are future work, and none of the numbers earlier in this article are measured results.

Tracing this request made the migration differences easier to understand. Discovery establishes which engines exist; load and KV events supply the state; bucket selection, admission and policy place the request; and the engine scheduler executes it. That separation is also useful beyond SGLang when reasoning about distributed inference.

References