Trace Diff using Holistic Trace Analysis

Author: Anupam Bhatnagar

Occasionally, users need to identify the changes in PyTorch operators and CUDA kernels resulting from a code change. To support this requirement, HTA provides a trace comparison feature. This feature allows the user to input two sets of trace files where the first can be thought of as the control group and the second as the test group, similar to an A/B test. The TraceDiff class provides functions to compare the differences between traces and functionality to visualize these differences. In particular, users can find operators and kernels that were added and removed from each group, along with the frequency of each operator/kernel and the cumulative time taken by the operator/kernel.

The TraceDiff class has the following methods:

  • compare_traces: Compare the frequency and total duration of CPU operators and GPU kernels from two sets of traces.

  • ops_diff: Get the operators and kernels which have been:

    1. added to the test trace and are absent in the control trace

    2. deleted from the test trace and are present in the control trace

    3. increased in frequency in the test trace and exist in the control trace

    4. decreased in frequency in the test trace and exist in the control trace

    5. unchanged between the two sets of traces

  • visualize_counts_diff

  • visualize_duration_diff

The last two methods can be used to visualize various changes in frequency and duration of CPU operators and GPU kernels, using the output of the compare_traces method.

For example, the top ten operators with increase in frequency can be computed as follows:

df = compare_traces_output.sort_values(by="diff_counts", ascending=False).head(10)

Similarly, the top ten operators with the largest change in duration can be computed as follows:

df = compare_traces_output.sort_values(by="diff_duration", ascending=False)
# The duration differerence can be overshadowed by the "ProfilerStep",
# so we can filter it out to show the trend of other operators.
df = df.loc[~df.index.str.startswith("ProfilerStep")].head(10)

For a detailed example of this feature see the trace_diff_demo notebook in the examples folder of the repository.

더 궁금하시거나 개선할 내용이 있으신가요? 커뮤니티에 참여해보세요!

이 튜토리얼이 어떠셨나요? 평가해주시면 이후 개선에 참고하겠습니다! :)

© Copyright 2018-2024, PyTorch & 파이토치 한국 사용자 모임(PyTorch Korea User Group).

Built with Sphinx using a theme provided by Read the Docs.

PyTorchKorea @ GitHub

파이토치 한국 사용자 모임을 GitHub에서 만나보세요.

GitHub로 이동

한국어 튜토리얼

한국어로 번역 중인 PyTorch 튜토리얼입니다.

튜토리얼로 이동


다른 사용자들과 의견을 나누고, 도와주세요!

커뮤니티로 이동