Plan a paired benchmark you can inspect later设计一次日后能够复查的配对测试

Define the boundary, validate outputs and retain the individual observations.定义边界、验证输出,并保留每次观测。

Define the comparison

Write down the workload, output contract, metric and boundary before running A and B. Record code revisions, hardware, dependency versions, inputs and invocation. A layer measurement cannot alone establish a serving improvement.

Check correctness first

Use the same inputs and a declared comparison rule. Exact equality and tolerance-based comparisons answer different questions. Preserve mismatches and explain any permitted tolerance.

Pair and alternate

Warm up both paths. Collect repeated pairs, alternating A/B and B/A order. Retain each observation and the run order, rather than only the final average. This helps expose drift; it does not eliminate every source of interference.

For asynchronous devices, define and apply the synchronization or device-timing boundary consistently. Keep setup and compilation separate when they are outside the intended metric.

Inspect and report

Plot or inspect the distribution and paired differences. Report the sample count, aggregation, uncertainty and controls. If the direction reverses between rounds, retain that observation and investigate the order or environment before claiming a stable improvement.

Save the artifact

Keep a manifest with the configuration, exact command, raw samples, correctness results and analysis script. The MoEFlow order-control report is a concrete repository reference. Follow the project page for its scope and related evidence.

定义对比

运行 A 与 B 之前,写清负载、输出约定、指标与测量边界。记录代码版本、硬件、依赖版本、输入与命令。单层测量不能单独证明服务性能提升。

先检查正确性

使用相同输入与明确的比较规则。精确相等和容差比较回答不同问题。保留不匹配结果,并解释允许的误差。

配对并交替顺序

两条路径都先预热。重复采集配对样本,交替使用 A/B 与 B/A 顺序。保留每次观测及运行顺序,而不只留下最终平均数。这有助于发现漂移,但不能消除全部干扰。

对于异步设备,要统一定义同步或设备计时边界。如果准备与编译不在目标指标中,就分别记录。

检查并报告

检查分布与配对差值,报告样本数、汇总方式、不确定性和控制条件。如果不同轮次的方向发生反转,应保留现象,先调查顺序或环境,再判断是否存在稳定收益。

保存材料

保留配置、完整命令、原始样本、正确性结果与分析脚本。MoEFlow 测量顺序控制报告提供一个具体仓库参考。项目页说明其研究范围与相关证据。