We Removed a CUDA Graph Fallback. It Got Slower.

The Problem

Removing a fallback can make a system look cleaner while moving cost somewhere less visible. In an irregular serving workload, the useful question is not merely whether a path is captured. It is whether the captured work still matches the distribution the runtime actually sees.

This is a working systems note. Hardware, commit, configuration, and benchmark values remain TODO until they can be attached to a reproducible run.

Execution trace · schematic

0255075100

GPU 0

GPU 1

GPU 2

GPU 3

ComputeCommunicationCPUMemory

Figure 1. Illustrative execution trace for layout validation only; replace with a verified profiler capture before publication.

Execution Path

The simplified path has four conceptual stages: routing, dispatch, expert computation, and combine. Their dependency chain is easy to draw and harder to schedule:

flowchart LR
  A[Route] --> B[Dispatch]
  B --> C[Grouped GEMM]
  C --> D[Combine]

The shape presented to expert computation varies with the routing decision. A static graph can reduce launch overhead, but it can also preserve work sized for a conservative case.

Hypothesis

The working hypothesis is that reducing fallback frequency changed the balance between launch savings and padded or poorly distributed work. This is not yet a conclusion; it is the question the profiler run must isolate.

For a batch with expert loads $n_1, \ldots, n_E$ , a useful first description of imbalance is:

I = \frac{\max_i n_i}{\frac{1}{E}\sum_{i=1}^{E}n_i}

This scalar is incomplete, but it makes the experimental condition explicit.

Implementation

Keep the change small enough to compare. Record the exact revision and invocation alongside every trace.

// Schematic only — not production code.
if (capture_eligible(shape, capacity)) {
  replay_graph(shape);
} else {
  run_eager(shape);
}

Field	Required record
Runtime revision	TODO
GPU and driver	TODO
Model / precision	TODO
Batch distribution	TODO
Capture policy	TODO

Performance Surprise

An end-to-end regression does not identify the losing stage. It only tells us that the new balance is worse under the measured condition. Separate graph replay, padding, synchronization, and communication effects before assigning a cause.

Profiling

Use a paired capture with identical inputs. Compare GPU-active time, host gaps, collective overlap, and the expert-load distribution. A timeline is evidence only when its run configuration is preserved with it.

Do not interpret the schematic figure above as measured output. The published version should embed the original profiler export and its configuration.

Root Cause

TODO: State a root cause only after the paired traces and counters agree. If the evidence remains mixed, retain multiple hypotheses rather than collapsing them into a cleaner story.

Lessons

A fallback can be part of the performance policy, not merely an error path.
Capture coverage is not equivalent to useful work coverage.
Irregularity should be described as a distribution, not a representative batch.
Every performance claim needs a reproducible configuration.

What’s Next

Attach the verified trace, publish the exact experiment matrix, and link the relevant implementation only after its status can be represented accurately.

问题

移除回退路径可能让系统看起来更简洁，却把成本转移到更难观察的位置。在不规则服务负载中，真正有价值的问题不只是某条路径是否被捕获，而是被捕获的工作是否仍与运行时实际遇到的分布相匹配。

这是一份仍在推进中的系统研究笔记。在能够关联到可复现实验之前，硬件、提交版本、配置与基准数值均保留为 TODO。

Execution trace · schematic

0255075100

GPU 0

GPU 1

GPU 2

GPU 3

ComputeCommunicationCPUMemory

Figure 1. Illustrative execution trace for layout validation only; replace with a verified profiler capture before publication.

执行路径

简化后的路径包含四个概念阶段：路由、分发、专家计算与合并。依赖关系容易画出，却更难调度：

flowchart LR
  A[路由] --> B[分发]
  B --> C[Grouped GEMM]
  C --> D[合并]

专家计算接收到的形状会随路由决策变化。静态图可以降低启动开销，但也可能保留为保守情形预留的计算工作。

假设

当前假设是：降低回退路径的出现频率，改变了启动收益与填充计算或分布不佳工作之间的平衡。这还不是结论，而是性能分析实验必须隔离验证的问题。

对于专家负载为 $n_1, \ldots, n_E$ 的批次，可以先用下面的指标描述不均衡程度：

I = \frac{\max_i n_i}{\frac{1}{E}\sum_{i=1}^{E}n_i}

这个标量并不完整，但它能把实验条件明确表达出来。

实现

将改动控制在足够小的范围内，以便进行可靠对比。每一条性能轨迹都应同时记录精确版本和调用方式。

// 仅为示意，并非生产代码。
if (capture_eligible(shape, capacity)) {
  replay_graph(shape);
} else {
  run_eager(shape);
}

字段	必须记录的内容
运行时版本	TODO
GPU 与驱动	TODO
模型 / 精度	TODO
批次分布	TODO
捕获策略	TODO

性能反常

端到端性能回退并不能指出损失发生在哪个阶段。它只能说明，在当前测量条件下，新的平衡更差。在判断根因之前，需要分别隔离图回放、填充、同步与通信带来的影响。

性能分析

使用完全相同的输入进行成对采集。比较 GPU 活跃时间、主机间隙、集合通信重叠情况和专家负载分布。只有当运行配置与时间线一同保存时，这条时间线才构成有效证据。

不要把上面的示意图解释为测量输出。正式版本应嵌入原始性能分析导出文件及其配置。

根因

TODO： 只有在成对性能轨迹与计数器相互印证后，才能陈述根因。如果证据仍然混杂，应保留多个假设，而不是强行把它们压缩成一个更整洁的故事。

经验

回退路径可能是性能策略的一部分，而不只是错误处理路径。
捕获覆盖率不等于有效工作覆盖率。
不规则性应以分布描述，而不是用一个“代表性批次”概括。
每一项性能结论都需要可复现的配置。

下一步

附上经验证的性能轨迹，发布完整实验矩阵，并且只在实现状态能够被准确表述后再链接对应代码。

vLLM · Next in this project接着读Communication Overlap Is Not Free通信重叠并非免费 →Working outline研究提纲