Tracing a Request Through Mini-SGLang

Code Reading Notes #01 — From an HTTP Request to GPU Execution

As I've been learning about LLM inference, I've come across concepts such as prefill, decode, continuous batching, KV caching, and scheduling.

Understanding each concept in isolation is one thing. Understanding how they work together inside a real inference engine is another.

That led me to a simple question:

What actually happens after a user sends a prompt to an LLM serving engine, and how does the first generated token eventually reach the user?

Rather than starting with the full SGLang codebase, I began with the more approachable Mini-SGLang. Mini-SGLang is a relatively compact LLM serving engine that preserves important runtime mechanisms, including scheduling, KV cache management, attention backends, and model execution. Its smaller codebase makes it a useful place to build a mental model of an inference system.

In this post, I'll follow a text-only request through Mini-SGLang, from the HTTP API to the tokenizer, scheduler, GPU engine, and finally the streaming response.

I'll focus on a single request running with tensor parallelism set to one (TP=1). The discussion is based on source code inspection rather than an instrumented runtime trace or performance benchmark.

Architecture: Following a Request Across Processes

Let's start with the overall data flow:

User
  │ HTTP POST /generate
  ▼
API Server / FrontendManager
  │ TokenizeMsg
  ▼
Tokenizer Worker
  │ UserMsg(input_ids)
  ▼
Scheduler
  │ Batch Preparation / KV Allocation
  ▼
Engine / GPU
  │ Model Forward → Logits → Sampling
  ▼
Scheduler
  │ DetokenizeMsg(next_token)
  ▼
Detokenizer Worker
  │ UserReply(incremental_output)
  ▼
FrontendManager
  │ HTTP Streaming
  ▼
User

The first thing to notice is that LLM serving is not simply an HTTP handler calling model.forward() and returning the result. Mini-SGLang separates request handling, tokenization, scheduling, model execution, and detokenization. These components exchange messages through ZeroMQ (ZMQ).

Looking at server/launch.py, Mini-SGLang launches one scheduler worker per tensor-parallel rank. Each scheduler owns an Engine instance responsible for model execution on its GPU.

Meanwhile, with the default --num-tokenizer=0, tokenization and detokenization share a worker process.

This means the diagram represents logical responsibilities, not necessarily separate operating-system processes.

That distinction matters when reasoning about process boundaries, communication overhead, and CPU/GPU coordination.

Source: Architecture Documentation, launch.py.

The Frontend: Turning an HTTP Request into a Message

Consider a request to Mini-SGLang's /generate endpoint:

POST /generate
Content-Type: application/json

{
  "prompt": "Hello",
  "max_tokens": 8
}

The FastAPI endpoint in server/api_server.py receives the request and delegates its lifecycle management to FrontendManager.

One of the first things FrontendManager does is assign a unique request identifier, or uid.

It also creates the internal state needed to associate future responses with this request.

The frontend then constructs a message similar to:

TokenizeMsg(
    uid=uid,
    text=req.prompt,
    sampling_params=SamplingParams(
        max_tokens=req.max_tokens,
        ignore_eos=req.ignore_eos,
    ),
)

This message is sent to the tokenizer worker through ZMQ.

The first mental model here is that the frontend manages the request lifecycle and the response channel. It also sets up a waiting state for the uid, so the HTTP stream can later receive the results the backend produces step by step.

Source: api_server.py.

Tokenization: From Text to Input IDs

The tokenizer worker receives TokenizeMsg and converts the input text into token IDs.

Conceptually:

"Hello"
    ↓
Tokenizer
    ↓
input_ids = [...]

The exact token IDs depend on the model's tokenizer, so I'm intentionally leaving them unspecified.

In Mini-SGLang, the relevant path is:

tokenize_worker()
    ↓
TokenizeManager.tokenize()
    ↓
tokenizer.encode()
    ↓
UserMsg(uid, input_ids, sampling_params)

The worker uses a Hugging Face tokenizer. When the input consists of chat messages, it also applies the model's chat template before encoding the prompt.

The resulting input_ids are represented as an integer tensor and sent to the scheduler in a UserMsg.

At this point, we have transformed human-readable text into model input IDs, but we haven't executed the transformer yet.

This helped me separate two responsibilities that are easy to mentally combine:

Tokenization prepares the input. Scheduling and model execution determine how that input is processed.

Source: tokenizer/server.py, tokenizer/tokenize.py.

Inside the Scheduler: Deciding What the GPU Runs Next

Once a request reaches the scheduler, the scheduler must decide which requests are runnable, how to group their work, and whether the next batch should perform prefill or decode.

A simplified view of Mini-SGLang's normal_loop() looks like this:

Receive Messages
      ↓
Process Requests
      ↓
Schedule Next Batch
      ↓
Prepare Batch
      ↓
GPU Forward
      ↓
Process Results

When a new UserMsg arrives, the scheduler passes it to PrefillManager.

The key decision happens in _schedule_next_batch():

batch = (
    self.prefill_manager.schedule_next_batch(self.prefill_budget)
    or self.decode_manager.schedule_next_batch()
)

In this version, the scheduler first tries to construct a prefill batch. If no prefill batch can be scheduled, it tries to schedule decode work.

These phases have different purposes.

Prefill processes the prompt tokens that do not yet have computed KV states. It builds the required KV cache and produces the prediction for the first output token.

Decode processes newly generated tokens while reusing the KV cache from previous positions.

Even with only one request, the scheduler needs to track its state and determine how much work must be performed in each iteration.

With multiple concurrent requests, the same mechanisms become the foundation for batching, admission control, and GPU resource management.

Source: scheduler.py, prefill.py.

Request State: Connecting Prefill and Decode

In LLM inference, prefill and decode are two consecutive phases of the same request. Prefill processes the prompt tokens, builds the KV cache, and predicts the first output token. Decode then processes one newly generated token per step, using the KV cache built so far to keep generating.

From a GPU hardware perspective, prefill processes many tokens at once, so GEMMs can reuse each weight across many tokens, which tends to make it compute-bound. Decode with a small batch does little computation per step but still has to read the model weights and the historical KV cache every step, so it tends to be limited by memory bandwidth.

Since prefill and decode process different tokens at each step, how does the runtime know which tokens already have KV cache and which still need to be computed? That is exactly what cached_len, device_len, and extend_len are for.

In core.py, a Req contains several important fields:

cached_len
device_len
extend_len = device_len - cached_len

I think about them this way:

• [0, cached_len) represents token positions whose KV states are already available.
• [cached_len, device_len) represents token positions that still need to be processed in the current forward pass.
• extend_len tells us how many tokens need to be computed.

Let's walk through an example.

Suppose our prompt consists of three tokens, A B C, with no prefix cache hit.

First forward pass: Prefill

Before execution:

Tokens:       A   B   C
cached_len:   0
device_len:   3
extend_len:   3

The model processes A, B, and C, computes their KV states, and samples the next token from the resulting logits.

Let's call that token D.

The request now has the state:

KV already computed    Next token to process
 A   B   C         |          D

cached_len = 3
device_len = 4
extend_len = 1

This reveals a subtle but important distinction:

Sampling token D does not mean that D's own KV state has already been computed.

The prefill pass processed A, B, and C. Token D is the output selected from that computation.

D's KV state will be calculated in the next forward pass.

Second forward pass: Decode

The next iteration processes token D, using the cached K/V from A, B, and C as context.

The model then samples another token, E.

Conceptually, the state becomes:

KV already computed        Next token
 A   B   C   D         |       E

cached_len = 4
device_len = 5
extend_len = 1

Mini-SGLang advances this state using Req.complete_one():

self.cached_len = self.device_len
self.device_len += 1

This small function helped connect several ideas for me.

Prefill and decode are not unrelated operations. They are successive stages of the same autoregressive generation process, with request state and KV cache determining which token positions need computation.

Source: core.py, engine.py.

KV Cache Management: Allocation Is Not Computation

Once I understood request state, my next question was: Where are these KV states actually stored, and who manages their memory?

Mini-SGLang separates several responsibilities.

TableManager allocates a table index for an active request.

CacheManager manages physical KV slots and updates the page table when new storage is required.

MHAKVCache owns the GPU tensors that actually store the keys and values.

A simplified mapping looks like this:

Request
   ↓
table_idx + logical token position
   ↓
Page Table
   ↓
physical KV slot
   ↓
GPU KV Cache Tensor

In this implementation, the page table stores physical token locations rather than page IDs. The default page size is one token, although the cache manager also supports page-oriented allocation logic.

During batch preparation, the scheduler calls CacheManager.allocate_paged() and constructs the physical output locations used by the batch.

But allocating those locations does not populate them with computed K/V values.

That happens during the model's attention computation.

For example, the FlashInfer backend calls:

self.kvcache.store_kv(k, v, batch.out_loc, layer_id)

It then invokes the attention wrapper using the appropriate KV cache tensors and metadata.

The distinction is straightforward but fundamental:

The scheduler prepares where KV data will be stored. The attention computation produces and writes that data.

From a hardware perspective, this also explains why KV cache management is so important to serving capacity.

Model weights and KV buffers occupy GPU memory. As context length and concurrency increase, the required KV storage increases as well.

A scheduler cannot safely admit work based only on whether the model fits in VRAM. It must also account for the memory needed to process and retain requests' KV states.

Source: cache.py, table.py, mha_pool.py, fi.py.

GPU Forward: From Input IDs to the Next Token

We can now connect scheduling to actual model execution.

Mini-SGLang's scheduler contains a relatively small _forward() method:

batch.input_ids = self.token_pool[input_mapping]
forward_output = self.engine.forward_batch(batch, sample_args)
self.token_pool[output_mapping] = forward_output.next_tokens_gpu

These lines describe a crucial boundary.

First, the scheduler retrieves the token IDs required for the current batch.

Next, Engine.forward_batch() runs the model. Depending on the batch and the available execution path, the engine uses a regular model forward pass or CUDA Graph replay.

Inside the transformer, the attention layer processes Q, K, and V, applies the relevant positional transformations, and invokes the configured attention backend.

The attention implementation reads the required cached states and writes new K/V values.

The model then produces logits, which are used by the sampler to select the next token ID.

The distinction between these representations matters:

Input Token IDs
      ↓
Model Forward
      ↓
Attention + KV Cache
      ↓
Logits
      ↓
Sampler
      ↓
Next Token ID

Logits are not tokens, and token IDs are not text.

Logits represent scores over the vocabulary. Sampling converts those scores into a selected token ID. Detokenization will later convert the generated token sequence into readable text.

The newly selected token is also written into the token pool so the next decode iteration can continue.

To me, _forward() is especially useful because it marks the boundary between the scheduler's view of the world and GPU model execution.

The scheduler decides what needs to run. The engine performs the computation.

Source: scheduler.py — _forward, engine.py — forward_batch, AttentionLayer.

Streaming the Output Back to the User

After GPU execution, the scheduler obtains the generated token ID and checks whether the request should finish.

Termination may depend on conditions such as reaching the maximum number of generated tokens or producing an EOS token.

The scheduler sends a message containing the request ID, generated token, and completion status:

DetokenizeMsg(
    uid,
    next_token,
    finished
)

The detokenizer worker converts token IDs into incremental text.

An important detail is that one token does not necessarily correspond to one complete character or word.

For that reason, the detokenizer maintains decoding state for each request rather than independently converting every token into a final printable string.

It may hold back incomplete or unstable fragments until enough information is available to produce readable output.

The worker then sends a UserReply back to the frontend, including the uid, incremental text, and completion status.

FrontendManager uses the request ID to wake up the corresponding waiting HTTP response and stream the new text to the client.

For the /generate endpoint, this is implemented through a StreamingResponse using the text/event-stream media type. The stream eventually emits a [DONE] marker.

This helped me recognize that generating the next token and displaying the next piece of text are different events.

The entire streaming experience depends on multiple components cooperating across process boundaries.

Source: detokenize.py, scheduler/io.py, api_server.py — Streaming.

A Brief Note on Overlap Scheduling

So far, I've used the simpler normal_loop() to explain the control flow.

Mini-SGLang also implements overlap_loop(), which is used when overlap scheduling is enabled—the default configuration unless explicitly disabled.

The basic idea is to overlap GPU work for the current batch with CPU-side processing of a previous batch's results.

Instead of treating scheduling, GPU execution, and output processing as entirely sequential steps, the runtime attempts to reduce idle time by allowing appropriate work to overlap.

This is an interesting reminder that inference optimization does not stop at the GPU kernel.

CPU scheduling overhead, memory transfers, synchronization, and the coordination of different execution stages can all affect serving performance.

I won't go deeper into CUDA streams or synchronization here. Those mechanisms deserve their own investigation, ideally supported by profiling measurements.

Source: scheduler.py — normal_loop and overlap_loop.

What I Learned

Reading through Mini-SGLang helped me connect several concepts that previously felt disconnected.

The frontend owns request handling and response delivery. The tokenizer prepares the model input. The scheduler manages request state, batching, and resources. The cache manager prepares KV storage. The engine and attention backend perform GPU computation. The sampler selects the next token, and the detokenizer turns generated token IDs into readable text.

Three ideas stood out to me.

First, prefill and decode are stages of one continuous autoregressive process. Sampling a new token and computing that token's KV state are not the same operation.

Second, KV caching is more than storing a tensor. It also involves request-to-memory mapping, allocation, reuse, and resource lifecycles.

Third, LLM serving is a systems problem. GPU kernels are important, but scheduling, memory management, process communication, and CPU/GPU coordination are part of the same performance picture.

I haven't instrumented the runtime or measured the performance of these stages yet. Everything discussed here is based on reading and reasoning about the implementation.

My next step is to follow an actual request ID in a GPU environment, record the transitions between tokenization, scheduling, prefill, decode, and detokenization, and eventually investigate metrics such as TTFT and TPOT.

For now, I want to document what I've learned instead of waiting until I've mastered every part of the inference stack.

Read → Trace → Understand → Write → Measure → Improve.

References

• Mini-SGLang GitHub Repository
• Mini-SGLang Architecture Documentation
• Source Snapshot — Commit 9a91cfafe754

This article is based on source code reading, not an instrumented runtime trace or performance benchmark. Implementation details refer to the linked Git commit.