From Image Bytes to the First Token: Tracing a Qwen3-VL Request Through SGLang

Introduction

What actually happens inside SGLang when we send an image along with a text prompt? How does the request travel from raw image bytes to the first generated token?

Unlike text-only inference, multimodal inference introduces additional steps: decoding and preprocessing images, constructing visual token positions, transferring multimodal features between processes, running the vision encoder, and integrating visual embeddings into the language model.

In this article, I'll trace a Qwen3-VL-4B-Instruct request end to end through SGLang v0.5.23, from the OpenAI-compatible HTTP API to the first generated token. Along the way, we'll follow how image data is transformed, how it moves through the system, and how the scheduler determines which computations are necessary.

Rather than simply tracing the request through the codebase, I designed and ran a series of controlled experiments on a single NVIDIA A100 80GB GPU to understand how SGLang behaves in practice.

Some results were less obvious than the execution diagram suggested. Changing the order of an image and a question caused Radix prefix reuse to drop from 264/289 to 6/289 tokens, but with the VLM embedding cache enabled, median TTFT barely changed. In another experiment, switching the image-feature transport from cpu to cuda_ipc saved 15.3 ms for repeated-image requests, even though 283/284 input tokens were already matched in the Radix cache.

These observations raised a broader question: which parts of a multimodal request are actually reused, and which parts still execute before the first token is returned? To investigate, we'll connect the source code with runtime measurements, looking at what each component does, what data it passes onward, and where latency can remain even when expensive model computations are skipped.

This is the first article in a series investigating multimodal inference performance in SGLang. The following articles will examine cache reuse in more detail, benchmark image resolution and concurrency, and investigate bottlenecks through profiling and controlled comparisons.

The goal is not only to understand SGLang's multimodal execution path, but also to establish a reproducible, measurement-driven approach to studying inference systems.

Experiment Setup

To make the investigation reproducible, I pinned both the SGLang implementation and the model revision.

For the main example, I generated a synthetic PNG containing a red circle and the text SGLang 42. The request asks:

What shape and color is in the image, and what text is written? Answer briefly.

The corresponding API payload looks like this (with the base64 data abbreviated):

body = {
    "model": "Qwen/Qwen3-VL-4B-Instruct",
    "messages": [{
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "data:image/png;base64,..."},
            },
            {
                "type": "text",
                "text": (
                    "What shape and color is in the image, "
                    "and what text is written? Answer briefly."
                ),
            },
        ],
    }],
    "temperature": 0,
    "max_tokens": 32,
    "stream": True,
    "stream_options": {"include_usage": True},
}

The image comes before the text question. This ordering becomes important when we later discuss prefix caching.

We'll follow two execution scenarios:

• A new image request: An image without a reusable image-specific KV prefix, requiring vision processing and language-model prefill.
• A repeated image request: The same image and prompt sent again, allowing SGLang to reuse most of the KV prefix.

The first scenario reveals the full multimodal execution path. The second shows which operations can be skipped and which still need to run.

Beyond reading the code, I collected client-side TTFT, token usage, cached-prefix information, and server-side request timing statistics. For controlled comparisons, I ran independent benchmark repetitions rather than relying on one request.

From HTTP to Multimodal Preprocessing

The Request Enters SGLang

Our journey begins at SGLang's OpenAI-compatible /v1/chat/completions endpoint.

The HTTP server receives a ChatCompletionRequest and passes it to the OpenAI serving layer. OpenAIServingBase.handle_request() validates the request, while OpenAIServingChat._convert_to_internal_request() constructs the representation passed to TokenizerManager.

One useful implementation detail is that request conversion can run in a thread executor, so prompt preparation does not monopolize the HTTP event loop.

But the most important transformation happens when SGLang processes the message content.

The original OpenAI request contains an image URL and a text question in the same messages array. During conversion, SGLang separates them:

• The image data URI is preserved in an ImageData object.
• The chat template receives an image marker alongside the text, rather than the full base64 payload.

[
  {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0…"}},
  {"type": "text", "text": "What shape and color…?"}
]

ImageData(
  url="data:image/png;base64,…"
)
# kept as-is · not decoded yet

[{"type": "image"},
 {"type": "text", "text": "What shape…?"}]

<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>What shape…?<|im_end|>
<|im_start|>assistant

GenerateReqInput(
  text="<|im_start|>user\n<|vision_start|><|image_pad|>…",
  image_data=[ImageData(url="data:image/png;base64,…")],
  input_ids=None,  # filled later by the multimodal processor
)

For Qwen3-VL, the rendered prompt contains a single <|image_pad|> surrounded by vision markers:

<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>What shape and color is in the image, and what text is written? Answer briefly.<|im_end|>
<|im_start|>assistant

This prompt is derived from the pinned chat template and source code, rather than directly captured from the runtime prompt log.

At this point, the image has not yet been decoded into pixel tensors, and the placeholder does not contain visual information. It only indicates where the image belongs in the conversation.

SGLang then constructs GenerateReqInput, which carries the rendered prompt, image data, sampling parameters, and streaming settings.

Why TokenizerManager Does Not Finish Tokenization Yet

There is a subtle distinction between two checks:

• model_config.is_multimodal: Does the loaded model support multimodal inputs?
• contains_mm_input(): Does this particular request contain image, video, or audio data?

A model can support multimodal inference while serving an entirely text-only request. The latter does not need to enter the image-processing path simply because the model happens to be Qwen3-VL.

For Qwen-VL, TokenizerManager._tokenize_one_request() also does something that may initially seem surprising: it does not finalize the multimodal prompt's token IDs itself.

Qwen-VL sets generates_input_ids_from_raw_prompt=True, so TokenizerManager passes the rendered text and image data to process_mm_data_async(). That processor can expand the image marker only after it knows the processed image shape and the required visual positions.

This is our first important distinction: the representation produced by the chat template is not yet the final representation consumed by the language model.

The template renderer also has an LRU cache over template-ready messages. Since the image payload is kept separate, the same surrounding text can produce the same template representation even when the image changes. The source supports this behavior; I did not measure the template-cache hit rate, so I do not attribute specific latency savings to it.

Relevant source:

• HTTP entrypoint
• OpenAI request conversion
• Template content processing
• TokenizerManager

Image Bytes Become Model Inputs

Next, the request reaches Qwen-VL's multimodal processor through process_mm_data_async().

The processor checks whether the media artifact path is applicable. For the image-only request in this experiment, Qwen3-VL can use this path even when cross-request preprocessing caching is disabled. A cache miss inside the artifact path is not the same thing as taking the separate uncached fallback path.

The preparation work can include:

• Decoding the base64 image payload and computing a content identity.
• Looking up a reusable preprocessing artifact when that cache is enabled.
• Decoding the PNG and preparing the pixel data when processing is necessary.
• Resizing and normalizing the image.
• Constructing the processed image tensors and the metadata needed downstream.

One detail surprised me during the source-code investigation: not all image preprocessing happens on the CPU.

In the pinned configuration, Qwen3-VL's fast image processor uses the GPU for resize and normalization. With the default cpu multimodal feature transport, the resulting pixel data is then copied back to CPU memory and made available to the scheduler through shared memory.

This is an important distinction: GPU image preprocessing is not the same as running the Vision Encoder. Preprocessing prepares the numerical inputs; the Vision Encoder later computes learned visual features.

The multimodal processor produces information including input_ids, mm_items containing image features and metadata, the image grid information, and position information such as mrope_positions. These are passed onward to the scheduler.

Relevant source:

• Qwen-VL multimodal processor
• Media artifacts

Two Data Paths: Token IDs and Image Tensors

At this point, an important question arises: if the prompt already contains image positions, where is the actual image information?

The answer is that SGLang carries two kinds of information through the request.

The Token and Identity Path

The text prompt contains token IDs, including positions reserved for the image.

Before multimodal processing, the rendered prompt contains a single <|image_pad|> placeholder. After processing, that placeholder expands into the number of positions required by the processed image.

For our 512 × 512 synthetic image, the server reported 284 prompt tokens. The model configuration and recorded totals are consistent with approximately 28 text/template tokens and 256 visual positions.

For a simple square image that remains 512 px after processing, patch size 16 yields a 32 × 32 grid; a 2 × 2 spatial merge yields 16 × 16 = 256 visual positions. This is an illustration of the configuration, not a rule that raw image width always determines visual token count. The exact runtime image_grid_thw tensor was not captured in this trace, and Qwen3-VL can perform dynamic image processing.

These visual positions tell the system where computed image features will eventually be inserted. They are not the features themselves.

The Image Tensor Path

Meanwhile, the numerical image data follows another path: image bytes become decoded and normalized pixel data, then enter the Vision Encoder when visual computation is needed.

Three terms should not be confused:

• Image slots identify positions in the language model's input sequence.
• pixel_values contains numerical image data prepared for vision computation.
• Visual embeddings are learned representations computed by the Vision Encoder.

During the model's forward pass, SGLang inserts the computed visual embeddings at the corresponding positions in the language model's input embeddings.

How Does the KV Cache Know Which Image Is Which?

There is a potential correctness problem here. Two different images can produce the same layout of image placeholders, even when their contents differ.

If prefix matching used only generic placeholder IDs, two requests could appear identical in the image region and incorrectly reuse each other's image-specific KV.

SGLang addresses this by incorporating media identity into the padded IDs used for prefix matching. These identity-aware IDs remain cache-matching information, not the actual visual embeddings.

In a negative-control experiment, I alternated structurally similar images with different contents. Across 300 requests, the image-specific KV prefix was not incorrectly reused, and all requests passed the shape-and-color correctness check. This supports the mechanism for the tested case; it is not a proof for every possible multimodal input.

Relevant source:

• Qwen-VL processor
• Scheduler request types
• Multimodal embedding utilities

From Scheduler to the First Token

Once multimodal preprocessing is complete, the request is handed to SGLang's scheduler.

The scheduler manages request state, prefix KV reuse, and the work submitted to the model runner. Before executing the model, it checks how much of the prompt can reuse an existing attention-KV prefix.

Prefix Matching Before Model Execution

For different images with the same question, my experiment recorded only 4/284 cached tokens: the common prompt prefix before the image. Repeating the same image and question produced 283/284 cached tokens.

These numbers describe Radix KV-prefix reuse, not whether preprocessing or the VLM embedding cache hit.

This distinction matters because image preprocessing occurs before the scheduler's Radix match. Even a near-complete KV hit does not retroactively undo the work that has already happened upstream.

The Vision Encoder and Embedding Injection

For a request that needs new visual computation, the model runner processes the prepared image with Qwen3-VL's Vision Encoder.

The Vision Encoder applies patch embedding, Vision Transformer blocks, and spatial merging to produce visual embeddings. The multimodal embedding routine then places those features at the corresponding image positions in the language model's input embeddings.

The language model processes the combined text and visual embeddings during prefill, writes attention KV, and generates the next-token logits. From there, the serving system generates and streams the first output token.

A repeated request whose image positions are covered by reusable KV may not need to run the Vision Encoder again. Separately, the VLM embedding cache can reuse vision features even when Radix's prefix does not match. These two mechanisms skip different work.

Does Prompt Order Actually Matter?

I tested the same image with 100 changing questions, comparing image-first and question-first orders. The difference in Radix prefix matching was dramatic:

Moving the changing question before the image caused the requests to diverge earlier, so Radix could no longer reuse the image's KV prefix.

Yet with the embedding cache enabled, median TTFT did not show a corresponding slowdown in this experiment. Once that cache was disabled, the question-first scenario was about 19 ms slower. The comparison is consistent with the embedding cache avoiding repeated vision computation when Radix reuse is unavailable.

So prompt order does affect prefix reuse. Whether it affects end-to-end TTFT depends on other caches, hardware, prompt length, and workload conditions. This is why cached_tokens alone is not enough to tell us how much actual vision work occurred.

The First Token Reaches the Client

After the required forward computation, SGLang generates an output token, handles the output/detokenization path, and streams the content back to the client.

This is the endpoint of our main trace.

TTFT includes much more than GPU model execution. It also reflects request handling, multimodal preprocessing, inter-process communication, scheduling, and output delivery.

Relevant source:

• Scheduler
• Qwen3-VL model
• Multimodal embedding utilities

What the Runtime Measurements Revealed

Following the source code gave me a model of how the request should execute. The experiments helped establish which parts of that model mattered for actual latency.

Most of the Extra TTFT Occurred Before Scheduling

With SGLang's request-time statistics enabled, I compared repeated text-only and image requests and divided client-observed TTFT into two coarse intervals: before entering the scheduler queue and after that boundary.

Most of the measured extra TTFT appeared before the scheduler queue boundary. That interval includes request handling, image preparation, transport, and related overhead. It narrows down where to investigate, but it does not measure individual function timings.

The time-stats configuration also differs from some earlier experiments, so I use this comparison to locate the overhead rather than combining its absolute latencies with other runs.

Cache Reuse Is Not Just One Mechanism

I then compared preprocessing-cache and Radix-cache configurations. For repeated images, Radix reuse lowered median TTFT by roughly 13–16 ms across the tested configurations, while preprocessing caching saved approximately 4–10 ms, depending on the other cache settings.

The prompt-order experiment provided a second insight: Radix can lose almost its entire reusable prefix while the VLM embedding cache still avoids rerunning the Vision Encoder.

The broader lesson is that preprocessing artifacts, vision embeddings, and language-model KV are different reusable objects. A hit at one layer does not imply a hit at another. Their interactions deserve their own controlled benchmark, which I'll cover in the next article.

The Feature Transport Path Was a Major Contributor

The source code provided another clue: in the cpu transport path, GPU-prepared pixel data is copied back to CPU memory and transferred through shared memory before the scheduler uses it.

I compared that implementation with cuda_ipc, which uses a GPU-resident buffer and CUDA IPC handles for inter-process feature access. The A/B experiments disabled preprocessing caching and VLM embedding caching while keeping Radix available.

For repeated-image requests, the image-versus-text TTFT gap fell from 20.6 ms to 6.2 ms. The new-image workload also improved, while the text-only baseline was approximately unchanged.

These controlled comparisons support feature transport as a major contributor to latency under the tested conditions. However, they do not identify the exact time spent in each .cpu() copy or shared-memory operation. That would require more granular profiling.

There is a trade-off. The CUDA IPC configuration reserved a 1 GiB GPU buffer, and the observed scheduler memory allocation was about 2.2 GiB smaller, leaving less room for its static/KV allocation. These results are from 512 px synthetic images at concurrency 1; larger images and higher concurrency may change the balance.

A Lesson in Measurement

There was another important finding during this investigation, unrelated to any individual SGLang component.

My initial comparison suggested image requests were 70–80 ms slower than text-only requests. However, the comparison crossed different server processes and did not adequately control the conditions.

After revising the measurement script and running interleaved comparisons within the same server process, the estimated difference became smaller and more consistent with the request-time statistics.

I also corrected a measurement assumption: streaming content chunks are not necessarily equivalent to model tokens. The updated benchmark records server token usage and content-chunk timing separately, including the case where a stop-generated EOS token counts toward completion usage without producing a normal text chunk.

This experience changed how I approached the investigation: instead of accepting the first latency number, I made sure I understood what each metric actually represented.

Tracing code tells us what may happen. Controlled experiments tell us what changes performance under the tested conditions.

Takeaways and What's Next

Tracing a Qwen3-VL request through SGLang revealed several lessons about multimodal serving.

Image placeholders are not image embeddings. The chat template records where an image belongs. The multimodal processor prepares data and expands positions; the Vision Encoder produces the learned features that the language model consumes.

KV-cache reuse does not eliminate every image-related operation. Even when most of the prefix can be reused, media handling, preprocessing, and feature transport can remain on the critical path.

Performance bottlenecks are not always where the expensive model computations are. In this experiment, changing feature transport reduced median TTFT for repeated-image requests by about 15 ms, showing why measuring the full request path matters.

Reproducibility and Limitations

The investigation uses pinned model and SGLang revisions, a single NVIDIA A100, controlled synthetic images, and independent experimental runs. Its conclusions should be interpreted within those conditions:

• The main reported workloads used 512 × 512 images at concurrency 1.
• Synthetic shapes and text are useful controls, but are not representative of production photographs or documents.
• Some cache behavior is inferred from controlled comparisons and source code rather than directly exported hit/miss counters.
• The TTFT and request-time data do not constitute a complete per-function or GPU-kernel profile.
• Exact OCR accuracy on this workload was poor, so latency measurements and answer-correctness checks were kept separate. Zero request errors does not mean zero answer errors.
• The NVIDIA driver, CUDA runtime, Python, and PyTorch versions listed above come from the preserved environment manifest rather than being inferred from the container tag; raw nvidia-smi and check_env output was not retained.

The main source-code entry points for reproducing this trace are:

• serving_chat.py
• tokenizer_manager.py
• qwen_vl.py
• scheduler.py
• qwen3_vl.py

Next in the Series

• Understanding Multimodal Cache Reuse in SGLang: Preprocessing, Vision Embeddings, and Radix KV — isolate which work each cache actually saves.
• Benchmarking Qwen3-VL Serving: Image Resolution, Concurrency, and Tail Latency — understand how workload shape changes the bottlenecks.
• Investigating Multimodal Serving Bottlenecks: Feature Transport, Profiling, and Trade-offs — measure deeper stages and test whether improvements hold under load.

Understanding the request path is the first step. Measuring what happens along that path is what turns a code trace into a performance investigation.