Trace Diff using Holistic Trace Analysis

Author: Anupam Bhatnagar

Occasionally, users need to identify the changes in PyTorch operators and CUDA kernels resulting from a code change. To support this requirement, HTA provides a trace comparison feature. This feature allows the user to input two sets of trace files where the first can be thought of as the control group and the second as the test group, similar to an A/B test. The TraceDiff class provides functions to compare the differences between traces and functionality to visualize these differences. In particular, users can find operators and kernels that were added and removed from each group, along with the frequency of each operator/kernel and the cumulative time taken by the operator/kernel.

The TraceDiff class has the following methods:

• compare_traces: Compare the frequency and total duration of CPU operators and GPU kernels from two sets of traces.

• ops_diff: Get the operators and kernels which have been:

  1. added to the test trace and are absent in the control trace

  2. deleted from the test trace and are present in the control trace

  3. increased in frequency in the test trace and exist in the control trace

  4. decreased in frequency in the test trace and exist in the control trace

  5. unchanged between the two sets of traces

• visualize_counts_diff

• visualize_duration_diff

The last two methods can be used to visualize various changes in frequency and duration of CPU operators and GPU kernels, using the output of the compare_traces method.

For example, the top ten operators with increase in frequency can be computed as follows:

df = compare_traces_output.sort_values(by="diff_counts", ascending=False).head(10)
TraceDiff.visualize_counts_diff(df)

Similarly, the top ten operators with the largest change in duration can be computed as follows:

df = compare_traces_output.sort_values(by="diff_duration", ascending=False)
# The duration differerence can be overshadowed by the "ProfilerStep",
# so we can filter it out to show the trend of other operators.
df = df.loc[~df.index.str.startswith("ProfilerStep")].head(10)
TraceDiff.visualize_duration_diff(df)

For a detailed example of this feature see the trace_diff_demo notebook in the examples folder of the repository.