Tracing a CPU Threading Bottleneck in SGLang's Multimodal Inference

From CPU vs. CUDA IPC to a 2.7× throughput improvement

Introduction

When I started investigating multimodal inference in SGLang, I expected image preprocessing and vision-model execution to account for most of the overhead. Instead, I found a workload where an NVIDIA A100 spent much of its time idle.

I benchmarked CPU-based feature transport against CUDA IPC using Qwen3-VL-4B-Instruct, with one image per request, across concurrency levels of 1, 4, and 16.

At concurrency 1, throughput was similar: 3.3 req/s with CPU transport versus 3.6 req/s with CUDA IPC. However, as concurrency increased to 16, the gap widened significantly. CPU transport reached only 8.7 req/s with 26% GPU utilization, while CUDA IPC achieved 25.7 req/s with 76% GPU utilization.

This suggested that the CPU-side request path was struggling to keep the GPU busy as concurrency increased, so I started profiling the tokenizer manager to understand where the time was being spent. Along the way, I also compared the two transport paths in more detail, increased image resolution, and tested whether PyTorch CPU parallelism was part of the problem. Finally, I benchmarked an existing fix in SGLang PR #48881 against my own thread-limiting workaround.

What surprised me was that the performance gap wasn't simply caused by moving tensors through CPU memory. A major part of the bottleneck came from PyTorch's shared-memory copy operation, which could trigger excessive CPU thread parallelism under the container's limited CPU quota.

1. CPU vs. CUDA IPC: Where the Image Tensor Goes

In my previous Qwen3-VL request trace, I followed requests through multimodal preprocessing, scheduling, and model execution. In this investigation, I took a closer look at how preprocessed image tensors are transferred from the TokenizerManager to the scheduler, and how that transfer affects inference performance under concurrency.

For Qwen3-VL, image decoding and preprocessing produce image input tensors such as pixel_values before the vision encoder runs. SGLang supports different ways of transferring these tensors between processes, including CPU-based shared memory and CUDA IPC.

With --mm-feature-transport cpu, SGLang moves the preprocessed tensor into CPU memory, allocates a shared-memory segment, copies the bytes into it, and sends a reference to the scheduler. The scheduler can subsequently consume those bytes and transfer them back to the GPU when vision execution needs them.

CPU transport
Image -> decode / GPU preprocessing -> .cpu()
      -> CPU tensor -> allocate /dev/shm -> copy_()
      -> send SHM reference -> scheduler -> GPU when needed

With --mm-feature-transport cuda_ipc, the preprocessed tensor stays GPU-resident. SGLang places it in a managed GPU memory pool and sends a CUDA IPC handle so the scheduler can access the same GPU allocation, avoiding the CPU shared-memory copy on this path.

CUDA IPC
Image -> decode / GPU preprocessing -> GPU memory pool
      -> send CUDA IPC handle -> scheduler -> GPU-resident data

CUDA IPC is not free: my setup reserved a 1 GiB GPU pool, which changes the memory available to the rest of the server. In my measurements, scheduler GPU memory was also about 2.2 GiB lower in the CUDA IPC configuration. So this is a latency/throughput versus memory-capacity trade-off, not simply a universally faster toggle.

My first experiments suggested this transport boundary still mattered even when most prompt tokens were already cached. For repeated requests with 283/284 Radix-matched input tokens, median time to first token (TTFT) fell from 50.1 ms (cpu) to 34.8 ms (cuda_ipc). Radix reuse saves model-side work, but it does not automatically skip the earlier multimodal processing and transport path.

I then increased image resolution. For repeated-image requests with almost complete Radix reuse, the CPU-minus-CUDA-IPC median TTFT gap grew from 12.9 ms at 512 px, to 32.9 ms at 1024 px, and 138.0 ms at 2048 px. Larger tensors meant more work for the CPU transport path. These resolution experiments used a different static-memory setting from the earlier 512 px test, so I compare within each experiment, not the absolute latencies across sessions.

Those measurements explained why transport could matter for a single request. I next wanted to see what would happen when many requests reached the tokenizer manager together.

2. Concurrency Turned Transport Overhead into a Bottleneck

I used SGLang's official serving benchmark with one fresh 512x512 image per request, 64 input tokens, 32 output tokens, preprocessing and VLM embedding caches disabled, and concurrency levels of 1, 4, and 16. Figure 1 in the introduction shows the throughput at each level.

At concurrency 16, the contrast was even clearer in the resource measurements:

Here, 1,043% CPU means roughly 10.4 logical cores' worth of CPU time spent by the main process. Meanwhile, the scheduler was less busy and the GPU was mostly waiting. Increasing concurrency from 4 to 16 hardly improved CPU transport throughput, even though CUDA IPC continued scaling.

I had initially suspected that large GPU-to-CPU transfers were the whole story. But a single 512 px image should not require ten CPU cores merely to make its bytes available to another process. I followed that discrepancy into the main process rather than trying to optimize GPU kernels.

3. Profiling the Main Process: copy_(), Threads, and CPU Quotas

I used py-spy to locate the expensive call sites. One hotspot was the CPU transport wrapper, ShmPointerMMData, where SGLang allocates shared memory and copies the tensor into it:

# Simplified from the original SHM transport path
posix_fallocate(...)
dst = torch.frombuffer(shm.buf, dtype=torch.uint8)
dst.copy_(tensor.view(torch.uint8).reshape(-1))

The profiler gave me a useful source-code location, but it did not account for all of the CPU consumption. During the concurrency-16 CPU benchmark, process monitoring showed approximately 10.4 CPU cores in use, whereas sampled Python execution accounted for roughly 0.56 core-equivalents. That mismatch pointed toward work below Python, including native PyTorch CPU operations. py-spy alone could not identify or time every native worker thread.

The container environment added another clue:

Visible logical CPUs:     128
torch.get_num_threads(): 128

$ cat /sys/fs/cgroup/cpu.max
1360000 100000

Effective sustained CPU budget = 1360000 / 100000 = 13.6 cores

A cgroup CPU quota is a time budget, not a dedicated set of 13.6 physical cores. Threads can run on many visible CPUs, but the group is permitted only 1,360,000 microseconds of CPU time per 100,000-microsecond period. Once that quota is exhausted, runnable work can be throttled until the next period.

PyTorch's configured intra-op parallelism was 128, while the container had 13.6 core-equivalents of sustained CPU time. Large CPU tensor operations can fan out into native worker threads. With a limited CPU budget, the overhead from dispatch, coordination, contention, and possible quota throttling may outweigh the benefit of parallel copying. This does not establish that every copy_() call launched exactly 128 active threads; it made thread oversubscription a testable hypothesis.

I also sampled /sys/fs/cgroup/cpu.stat. All four later 2x2 configurations showed a high share of throttled periods (roughly 79-86%), but the client generated random images inside the same cgroup as the server. Its CPU work was mixed into these counters, and the ratios were similar across configurations. So the counters cannot tell us how much server-side slowdown was directly caused by throttling. The evidence supports a CPU-threading bottleneck; it does not isolate the precise cost of cgroup throttling.

I therefore tested a simpler prediction first: if PyTorch CPU parallelism is involved, does reducing it recover throughput?

4. Validating the Fix: A 2x2 Experiment

As a first intervention, I launched the original server with:

OMP_NUM_THREADS=8

On the earlier P6b benchmark, throughput improved from 8.7 to 23.8 req/s, median TTFT dropped from 1,477 to 326 ms, and GPU utilization rose from 29% to 79%. This was a strong causal clue, but OMP_NUM_THREADS can influence operations throughout the server rather than one copy site.

While checking upstream work, I found SGLang PR #48881, "Avoid CPU thread-pool fanout in SHM copy". The PR avoids PyTorch intra-op dispatch for that particular shared-memory copy. In simplified form, it replaces:

dst.copy_(tensor.view(torch.uint8).reshape(-1))

with:

shm.buf[:] = tensor.detach().view(torch.uint8).reshape(-1).numpy()

The tensor is still copied into shared memory, and allocation and other transport costs remain. The change is that this copy no longer goes through PyTorch's copy_() CPU thread-pool path.

I wanted to know whether the PR alone was sufficient under my CPU quota, or whether a thread limit would still help. I compared the two copy implementations with default threading and OMP_NUM_THREADS=8:

For this controlled B1 experiment, I used the PR author's pinned upstream base (1442917b) with or without the exact PR diff. The GPU, model revision, container image, transport configuration, cache settings, and benchmark client were held constant. Each arm used a freshly started server; the four-arm order was shuffled; and matched seeds provided identical input workloads within each run. There were three runs per arm, each with 320 measured requests and concurrency 16.

PR #48881 alone raised throughput by approximately 2.7x and reduced median TTFT by 77%. Adding OMP_NUM_THREADS=8 on top of the patch did not improve throughput. The targeted PR also consumed less main-process CPU than the global thread cap alone (44% versus 135%) at comparable throughput.

All 3,840 measured requests completed without errors. Generated texts were not bitwise identical across every concurrent run; the mismatch frequency was similar in the thread-cap-only control, consistent with batch-dependent numerical variation, but this is not a formal correctness proof. Three independent runs also do not establish general-purpose confidence intervals or guarantee the same speedup on other hardware.

These results gave me enough evidence to stop looking for a separate B1 fix. I shared the CPU-quota benchmark on PR #48881 as an independent A100 validation case rather than proposing a duplicate implementation.

5. What the Results Mean (and What They Don't)

The transport comparison revealed two related but distinct costs.

First, CPU transport has real memory-movement and allocation overhead. The repeated-image measurements, particularly at 2048 px, showed that a larger tensor can add substantial latency even when the model-side prefix is cached. PR #48881 does not remove the .cpu() transfer, the shared-memory allocation, or the copy itself.

Second, in this particular 512 px, concurrency-16, CPU-quota-constrained workload, the original PyTorch copy_() implementation imposed an additional threading cost severe enough to leave the GPU underutilized. Capping OpenMP threads and changing the copy implementation independently recovered nearly the same throughput; combining them added little. That makes the SHM copy's thread-pool dispatch the best-supported explanation for the large high-concurrency regression.

The study has boundaries: one A100 server, one VLM, specific tensor shapes and concurrency settings, a shared client/server cgroup, and Python-level rather than full native-thread profiling. The throttling counters are useful context, not a measurement of the server-only cause. A different CPU quota, image size, model, or workload may shift the bottleneck.

For me, the most useful lesson was methodological. A transport A/B test identifies where to look, not necessarily why a path is slow. Process-level monitoring, source-code profiling, an isolated configuration workaround, and a controlled code-diff experiment were needed to separate unavoidable data movement from avoidable CPU-thread overhead.

6. Conclusion

I began with a simple question: why did CPU feature transport trail CUDA IPC so badly in multimodal serving? The answer changed as I moved from single-request measurements to concurrency, CPU profiling, and targeted interventions.

On an A100 with a 13.6-core-equivalent container CPU quota, the original CPU path reached only 8.6-8.7 req/s at concurrency 16. A focused change to its shared-memory copy, already proposed in SGLang PR #48881, recovered roughly 2.7x throughput without changing GPU kernels or tuning thread counts process-wide.

The broader lesson is that inference performance depends on the entire request pipeline. Sometimes the GPU is waiting not because the model is slow, but because a seemingly ordinary CPU tensor copy is doing much more work than expected.

Reproducibility note: P4/P5/P6b measurements used SGLang v0.5.23 (297e54f); the B1 2x2 used pinned upstream base 1442917b with/without PR #48881. All measurements used Qwen3-VL-4B-Instruct on A100-SXM4-80GB and a 13.6-core-equivalent cgroup quota. Reported values are measured medians under the stated workloads, not universal performance claims.