(beta) Accelerating BERT with semi-structured (2:4) sparsity

Author: Jesse Cai

Overview

Like other forms of sparsity, semi-structured sparsity is a model optimization technique that seeks to reduce the memory overhead and latency of a neural network at the expense of some model accuracy. It is also known as fine-grained structured sparsity or 2:4 structured sparsity.

Semi-structured sparsity derives its name from its unique sparsity pattern, where n out of every 2n elements are pruned. We most often see n=2, hence 2:4 sparsity Semi-structured sparsity is particularly interesting because it can be efficiently accelerated on GPUs and doesn’t degrade model accuracy as much as other sparsity patterns.

With the introduction of semi-structured sparsity support, it is possible to prune and accelerate a semi-structured sparse model without leaving PyTorch. We will explain this process in this tutorial.

By the end of this tutorial, we will have sparsified a BERT question-answering model to be 2:4 sparse, fine-tuning it to recover nearly all F1 loss (86.92 dense vs 86.48 sparse). Finally, we will accelerate this 2:4 sparse model for inference, yielding a 1.3x speedup.

Requirements

• PyTorch >= 2.1.

• A NVIDIA GPU with semi-structured sparsity support (Compute Capability 8.0+).

This tutorial is designed for beginners to semi-structured sparsity and sparsity in general. For users with existing 2:4 sparse models, accelerating nn.Linear layers for inference with to_sparse_semi_structured is quite straightforward. Here is an example:

import torch
from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor
from torch.utils.benchmark import Timer
SparseSemiStructuredTensor._FORCE_CUTLASS = True

# mask Linear weight to be 2:4 sparse
mask = torch.Tensor([0, 0, 1, 1]).tile((3072, 2560)).cuda().bool()
linear = torch.nn.Linear(10240, 3072).half().cuda().eval()
linear.weight = torch.nn.Parameter(mask * linear.weight)

x = torch.rand(3072, 10240).half().cuda()

with torch.inference_mode():
    dense_output = linear(x)
    dense_t = Timer(stmt="linear(x)",
                    globals={"linear": linear,
                             "x": x}).blocked_autorange().median * 1e3

    # accelerate via SparseSemiStructuredTensor
    linear.weight = torch.nn.Parameter(to_sparse_semi_structured(linear.weight))

    sparse_output = linear(x)
    sparse_t = Timer(stmt="linear(x)",
                    globals={"linear": linear,
                             "x": x}).blocked_autorange().median * 1e3

    # sparse and dense matmul are numerically equivalent
    # On an A100 80GB, we see: `Dense: 0.870ms Sparse: 0.630ms | Speedup: 1.382x`
    assert torch.allclose(sparse_output, dense_output, atol=1e-3)
    print(f"Dense: {dense_t:.3f}ms Sparse: {sparse_t:.3f}ms | Speedup: {(dense_t / sparse_t):.3f}x")

Dense: 1.579ms Sparse: 1.335ms | Speedup: 1.183x

What problem does semi-structured sparsity solve?

The general motivation behind sparsity is simple: if there are zeros in your network, you can optimize efficiency by not storing or computing those parameters. However, the specifics of sparsity are tricky. Zeroing out parameters doesn’t affect the latency / memory overhead of our model out of the box.

This is because the dense tensor still contains the pruned (zero) elements, which the dense matrix multiplication kernel will still operate on this elements. In order to realize performance gains, we need to swap out dense kernels for sparse kernels, which skip calculation involving pruned elements.

To do this, these kernels work on sparse matrices, which do not store the pruned elements and store the specified elements in a compressed format.

For semi-structured sparsity, we store exactly half of the original parameters along with some compressed metadata about how the elements were arranged.

There are many different sparse layouts, each with their own benefits and drawbacks. The 2:4 semi-structured sparse layout is particularly interesting for two reasons:

• Unlike previous sparse formats, semi-structured sparsity was designed to be efficiently accelerated on GPUs. In 2020, NVIDIA introduced hardware support for semi-structured sparsity with their Ampere architecture, and have also released fast sparse kernels via CUTLASS cuSPARSELt.

• At the same time, semi-structured sparsity tends to have a milder impact on model accuracy compared to other sparse formats, especially when accounting for more advanced pruning / fine-tuning methods. NVIDIA has shown in their white paper that a simple paradigm of magnitude pruning once to be 2:4 sparse and then retraining the model yields nearly identical model accuracies.

Semi-structured exists in a sweet spot, providing a 2x (theoretical) speedup at a much lower sparsity level (50%), while still being granular enough to preserve model accuracy.

Network	Data Set	Metric	Dense FP16	Sparse FP16
ResNet-50	ImageNet	Top-1	76.1	76.2
ResNeXt-101_32x8d	ImageNet	Top-1	79.3	79.3
Xception	ImageNet	Top-1	79.2	79.2
SSD-RN50	COCO2017	bbAP	24.8	24.8
MaskRCNN-RN50	COCO2017	bbAP	37.9	37.9
FairSeq Transformer	EN-DE WMT14	BLEU	28.2	28.5
BERT-Large	SQuAD v1.1	F1	91.9	91.9

Semi-structured sparsity has an additional advantage from a workflow perspective. Because the sparsity level is fixed at 50%, it is easier to decompose the problem of sparsifying a model into two distinct subproblems:

• Accuracy - How can we find a set of 2:4 sparse weights that minimize the accuracy degradation of our model?

• Performance - How can we accelerate our 2:4 sparse weights for inference and reduced memory overhead?

\[\begin{bmatrix} 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 1 \\ \end{bmatrix}\]

The natural handoff point between these two problems are zeroed-out dense tensors. Our inference solution is designed to compress and accelerate tensors in this format. We anticipate many users coming up with custom masking solution, as this is an active area of research.

Now that we’ve learned a little more about semi-structured sparsity, let’s apply it to a BERT model trained on a question answering task, SQuAD.

Intro & Setup

Let’s start by importing all the packages we need.

# If you are running this in Google Colab, run:
# .. code-block: python
#
#    !pip install datasets transformers evaluate accelerate pandas
#
import os
os.environ["WANDB_DISABLED"] = "true"

import collections
import datasets
import evaluate
import numpy as np
import torch
import torch.utils.benchmark as benchmark
from torch import nn
from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor
from torch.ao.pruning import WeightNormSparsifier
import transformers

# force CUTLASS use if ``cuSPARSELt`` is not available
SparseSemiStructuredTensor._FORCE_CUTLASS = True
torch.manual_seed(100)

<torch._C.Generator object at 0x752518a55130>

We’ll also need to define some helper functions that are specific to the dataset / task at hand. These were adapted from this Hugging Face course as a reference.

def preprocess_validation_function(examples, tokenizer):
    inputs = tokenizer(
        [q.strip() for q in examples["question"]],
        examples["context"],
        max_length=384,
        truncation="only_second",
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )
    sample_map = inputs.pop("overflow_to_sample_mapping")
    example_ids = []

    for i in range(len(inputs["input_ids"])):
        sample_idx = sample_map[i]
        example_ids.append(examples["id"][sample_idx])
        sequence_ids = inputs.sequence_ids(i)
        offset = inputs["offset_mapping"][i]
        inputs["offset_mapping"][i] = [
            o if sequence_ids[k] == 1 else None for k, o in enumerate(offset)
        ]

    inputs["example_id"] = example_ids
    return inputs


def preprocess_train_function(examples, tokenizer):
    inputs = tokenizer(
        [q.strip() for q in examples["question"]],
        examples["context"],
        max_length=384,
        truncation="only_second",
        return_offsets_mapping=True,
        padding="max_length",
    )

    offset_mapping = inputs["offset_mapping"]
    answers = examples["answers"]
    start_positions = []
    end_positions = []

    for i, (offset, answer) in enumerate(zip(offset_mapping, answers)):
        start_char = answer["answer_start"][0]
        end_char = start_char + len(answer["text"][0])
        sequence_ids = inputs.sequence_ids(i)

        # Find the start and end of the context
        idx = 0
        while sequence_ids[idx] != 1:
            idx += 1
        context_start = idx
        while sequence_ids[idx] == 1:
            idx += 1
        context_end = idx - 1

        # If the answer is not fully inside the context, label it (0, 0)
        if offset[context_start][0] > end_char or offset[context_end][1] < start_char:
            start_positions.append(0)
            end_positions.append(0)
        else:
            # Otherwise it's the start and end token positions
            idx = context_start
            while idx <= context_end and offset[idx][0] <= start_char:
                idx += 1
            start_positions.append(idx - 1)

            idx = context_end
            while idx >= context_start and offset[idx][1] >= end_char:
                idx -= 1
            end_positions.append(idx + 1)

    inputs["start_positions"] = start_positions
    inputs["end_positions"] = end_positions
    return inputs


def compute_metrics(start_logits, end_logits, features, examples):
    n_best = 20
    max_answer_length = 30
    metric = evaluate.load("squad")

    example_to_features = collections.defaultdict(list)
    for idx, feature in enumerate(features):
        example_to_features[feature["example_id"]].append(idx)

    predicted_answers = []
    # for example in ``tqdm`` (examples):
    for example in examples:
        example_id = example["id"]
        context = example["context"]
        answers = []

        # Loop through all features associated with that example
        for feature_index in example_to_features[example_id]:
            start_logit = start_logits[feature_index]
            end_logit = end_logits[feature_index]
            offsets = features[feature_index]["offset_mapping"]

            start_indexes = np.argsort(start_logit)[-1 : -n_best - 1 : -1].tolist()
            end_indexes = np.argsort(end_logit)[-1 : -n_best - 1 : -1].tolist()
            for start_index in start_indexes:
                for end_index in end_indexes:
                    # Skip answers that are not fully in the context
                    if offsets[start_index] is None or offsets[end_index] is None:
                        continue
                    # Skip answers with a length that is either < 0
                    # or > max_answer_length
                    if (
                        end_index < start_index
                        or end_index - start_index + 1 > max_answer_length
                    ):
                        continue

                    answer = {
                        "text": context[
                            offsets[start_index][0] : offsets[end_index][1]
                        ],
                        "logit_score": start_logit[start_index] + end_logit[end_index],
                    }
                    answers.append(answer)

        # Select the answer with the best score
        if len(answers) > 0:
            best_answer = max(answers, key=lambda x: x["logit_score"])
            predicted_answers.append(
                {"id": example_id, "prediction_text": best_answer["text"]}
            )
        else:
            predicted_answers.append({"id": example_id, "prediction_text": ""})

    theoretical_answers = [
        {"id": ex["id"], "answers": ex["answers"]} for ex in examples
    ]
    return metric.compute(predictions=predicted_answers, references=theoretical_answers)

Now that those are defined, we just need one additional helper function, which will help us benchmark our model.

def measure_execution_time(model, batch_sizes, dataset):
    dataset_for_model = dataset.remove_columns(["example_id", "offset_mapping"])
    dataset_for_model.set_format("torch")
    batch_size_to_time_sec = {}
    for batch_size in batch_sizes:
        batch = {
            k: dataset_for_model[k][:batch_size].cuda()
            for k in dataset_for_model.column_names
        }

        with torch.no_grad():
            baseline_predictions = model(**batch)
            timer = benchmark.Timer(
                stmt="model(**batch)", globals={"model": model, "batch": batch}
            )
            p50 = timer.blocked_autorange().median * 1000
            batch_size_to_time_sec[batch_size] = p50

            model_c = torch.compile(model, fullgraph=True)
            timer = benchmark.Timer(
                stmt="model(**batch)", globals={"model": model_c, "batch": batch}
            )
            p50 = timer.blocked_autorange().median * 1000
            batch_size_to_time_sec[f"{batch_size}_compile"] = p50
            new_predictions = model_c(**batch)

    return batch_size_to_time_sec

We will get started by loading our model and tokenizer, and then setting up our dataset.

# load model
model_name = "bert-base-cased"
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForQuestionAnswering.from_pretrained(model_name)
print(f"Loading tokenizer: {model_name}")
print(f"Loading model: {model_name}")

# set up train and val dataset
squad_dataset = datasets.load_dataset("squad")
tokenized_squad_dataset = {}
tokenized_squad_dataset["train"] = squad_dataset["train"].map(
    lambda x: preprocess_train_function(x, tokenizer), batched=True
)
tokenized_squad_dataset["validation"] = squad_dataset["validation"].map(
    lambda x: preprocess_validation_function(x, tokenizer),
    batched=True,
    remove_columns=squad_dataset["train"].column_names,
)
data_collator = transformers.DataCollatorWithPadding(tokenizer=tokenizer)

Loading tokenizer: bert-base-cased
Loading model: bert-base-cased


Downloading readme:   0%|          | 0.00/7.62k [00:00<?, ?B/s]
Downloading readme: 100%|##########| 7.62k/7.62k [00:00<00:00, 29.6MB/s]


Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]


Downloading data:   0%|          | 0.00/14.5M [00:00<?, ?B/s]


Downloading data:  29%|##9       | 4.19M/14.5M [00:00<00:01, 7.47MB/s]


Downloading data:  87%|########7 | 12.6M/14.5M [00:01<00:00, 12.1MB/s]
Downloading data: 100%|##########| 14.5M/14.5M [00:01<00:00, 12.3MB/s]


Downloading data files:  50%|#####     | 1/2 [00:01<00:01,  1.17s/it]


Downloading data:   0%|          | 0.00/1.82M [00:00<?, ?B/s]


Downloading data: 100%|##########| 1.82M/1.82M [00:00<00:00, 4.43MB/s]
Downloading data: 100%|##########| 1.82M/1.82M [00:00<00:00, 4.41MB/s]


Downloading data files: 100%|##########| 2/2 [00:01<00:00,  1.37it/s]
Downloading data files: 100%|##########| 2/2 [00:01<00:00,  1.26it/s]


Extracting data files:   0%|          | 0/2 [00:00<?, ?it/s]
Extracting data files: 100%|##########| 2/2 [00:00<00:00, 1360.68it/s]


Generating train split:   0%|          | 0/87599 [00:00<?, ? examples/s]

Generating train split:  80%|#######9  | 70000/87599 [00:00<00:00, 525364.06 examples/s]
Generating train split: 100%|##########| 87599/87599 [00:00<00:00, 519395.63 examples/s]


Generating validation split:   0%|          | 0/10570 [00:00<?, ? examples/s]
Generating validation split: 100%|##########| 10570/10570 [00:00<00:00, 461829.59 examples/s]


Map:   0%|          | 0/87599 [00:00<?, ? examples/s]

Map:   1%|1         | 1000/87599 [00:00<00:24, 3590.65 examples/s]

Map:   2%|2         | 2000/87599 [00:00<00:22, 3727.91 examples/s]

Map:   3%|3         | 3000/87599 [00:00<00:22, 3794.16 examples/s]

Map:   5%|4         | 4000/87599 [00:01<00:21, 3836.65 examples/s]

Map:   6%|5         | 5000/87599 [00:01<00:21, 3837.95 examples/s]

Map:   7%|6         | 6000/87599 [00:01<00:21, 3849.21 examples/s]

Map:   8%|7         | 7000/87599 [00:01<00:21, 3824.32 examples/s]

Map:   9%|9         | 8000/87599 [00:02<00:20, 3857.91 examples/s]

Map:  10%|#         | 9000/87599 [00:02<00:20, 3812.49 examples/s]

Map:  11%|#1        | 10000/87599 [00:02<00:20, 3818.68 examples/s]

Map:  13%|#2        | 11000/87599 [00:02<00:19, 3851.63 examples/s]

Map:  14%|#3        | 12000/87599 [00:03<00:19, 3843.48 examples/s]

Map:  15%|#4        | 13000/87599 [00:03<00:19, 3852.96 examples/s]

Map:  16%|#5        | 14000/87599 [00:03<00:19, 3858.53 examples/s]

Map:  17%|#7        | 15000/87599 [00:03<00:18, 3870.46 examples/s]

Map:  18%|#8        | 16000/87599 [00:04<00:18, 3895.14 examples/s]

Map:  19%|#9        | 17000/87599 [00:04<00:18, 3876.38 examples/s]

Map:  21%|##        | 18000/87599 [00:04<00:17, 3907.30 examples/s]

Map:  22%|##1       | 19000/87599 [00:04<00:17, 3903.84 examples/s]

Map:  23%|##2       | 20000/87599 [00:05<00:17, 3885.01 examples/s]

Map:  24%|##3       | 21000/87599 [00:05<00:17, 3882.53 examples/s]

Map:  25%|##5       | 22000/87599 [00:05<00:16, 3874.77 examples/s]

Map:  26%|##6       | 23000/87599 [00:05<00:16, 3891.04 examples/s]

Map:  27%|##7       | 24000/87599 [00:06<00:23, 2657.66 examples/s]

Map:  29%|##8       | 25000/87599 [00:06<00:21, 2931.69 examples/s]

Map:  30%|##9       | 26000/87599 [00:07<00:19, 3147.10 examples/s]

Map:  31%|###       | 27000/87599 [00:07<00:18, 3287.04 examples/s]

Map:  32%|###1      | 28000/87599 [00:07<00:17, 3412.81 examples/s]

Map:  33%|###3      | 29000/87599 [00:07<00:16, 3497.09 examples/s]

Map:  34%|###4      | 30000/87599 [00:08<00:16, 3572.27 examples/s]

Map:  35%|###5      | 31000/87599 [00:08<00:15, 3621.61 examples/s]

Map:  37%|###6      | 32000/87599 [00:08<00:15, 3636.44 examples/s]

Map:  38%|###7      | 33000/87599 [00:09<00:14, 3666.31 examples/s]

Map:  39%|###8      | 34000/87599 [00:09<00:14, 3678.30 examples/s]

Map:  40%|###9      | 35000/87599 [00:09<00:14, 3661.75 examples/s]

Map:  41%|####1     | 36000/87599 [00:09<00:14, 3674.69 examples/s]

Map:  42%|####2     | 37000/87599 [00:10<00:13, 3682.14 examples/s]

Map:  43%|####3     | 38000/87599 [00:10<00:13, 3687.98 examples/s]

Map:  45%|####4     | 39000/87599 [00:10<00:13, 3674.53 examples/s]

Map:  46%|####5     | 40000/87599 [00:10<00:12, 3680.78 examples/s]

Map:  47%|####6     | 41000/87599 [00:11<00:12, 3680.18 examples/s]

Map:  48%|####7     | 42000/87599 [00:11<00:12, 3682.52 examples/s]

Map:  49%|####9     | 43000/87599 [00:11<00:12, 3716.24 examples/s]

Map:  50%|#####     | 44000/87599 [00:11<00:11, 3729.28 examples/s]

Map:  51%|#####1    | 45000/87599 [00:12<00:11, 3706.22 examples/s]

Map:  53%|#####2    | 46000/87599 [00:12<00:11, 3702.44 examples/s]

Map:  54%|#####3    | 47000/87599 [00:12<00:10, 3692.61 examples/s]

Map:  55%|#####4    | 48000/87599 [00:13<00:10, 3697.94 examples/s]

Map:  56%|#####5    | 49000/87599 [00:13<00:10, 3689.44 examples/s]

Map:  57%|#####7    | 50000/87599 [00:14<00:14, 2570.26 examples/s]

Map:  58%|#####8    | 51000/87599 [00:14<00:12, 2824.70 examples/s]

Map:  59%|#####9    | 52000/87599 [00:14<00:11, 3029.65 examples/s]

Map:  61%|######    | 53000/87599 [00:14<00:10, 3204.52 examples/s]

Map:  62%|######1   | 54000/87599 [00:15<00:10, 3340.91 examples/s]

Map:  63%|######2   | 55000/87599 [00:15<00:09, 3445.44 examples/s]

Map:  64%|######3   | 56000/87599 [00:15<00:09, 3506.73 examples/s]

Map:  65%|######5   | 57000/87599 [00:15<00:08, 3552.01 examples/s]

Map:  66%|######6   | 58000/87599 [00:16<00:08, 3588.71 examples/s]

Map:  67%|######7   | 59000/87599 [00:16<00:07, 3608.72 examples/s]

Map:  68%|######8   | 60000/87599 [00:16<00:07, 3639.52 examples/s]

Map:  70%|######9   | 61000/87599 [00:17<00:07, 3649.30 examples/s]

Map:  71%|#######   | 62000/87599 [00:17<00:07, 3648.32 examples/s]

Map:  72%|#######1  | 63000/87599 [00:17<00:06, 3664.40 examples/s]

Map:  73%|#######3  | 64000/87599 [00:17<00:06, 3637.43 examples/s]

Map:  74%|#######4  | 65000/87599 [00:18<00:06, 3643.25 examples/s]

Map:  75%|#######5  | 66000/87599 [00:18<00:05, 3631.49 examples/s]

Map:  76%|#######6  | 67000/87599 [00:18<00:05, 3637.27 examples/s]

Map:  78%|#######7  | 68000/87599 [00:18<00:05, 3658.15 examples/s]

Map:  79%|#######8  | 69000/87599 [00:19<00:05, 3645.90 examples/s]

Map:  80%|#######9  | 70000/87599 [00:19<00:04, 3618.59 examples/s]

Map:  81%|########1 | 71000/87599 [00:19<00:04, 3612.92 examples/s]

Map:  82%|########2 | 72000/87599 [00:20<00:04, 3612.02 examples/s]

Map:  83%|########3 | 73000/87599 [00:20<00:04, 3620.30 examples/s]

Map:  84%|########4 | 74000/87599 [00:20<00:03, 3618.82 examples/s]

Map:  86%|########5 | 75000/87599 [00:20<00:03, 3622.39 examples/s]

Map:  87%|########6 | 76000/87599 [00:21<00:04, 2549.34 examples/s]

Map:  88%|########7 | 77000/87599 [00:21<00:03, 2800.48 examples/s]

Map:  89%|########9 | 78000/87599 [00:22<00:03, 3017.44 examples/s]

Map:  90%|######### | 79000/87599 [00:22<00:02, 3190.44 examples/s]

Map:  91%|#########1| 80000/87599 [00:22<00:02, 3317.50 examples/s]

Map:  92%|#########2| 81000/87599 [00:22<00:01, 3406.95 examples/s]

Map:  94%|#########3| 82000/87599 [00:23<00:01, 3464.97 examples/s]

Map:  95%|#########4| 83000/87599 [00:23<00:01, 3486.66 examples/s]

Map:  96%|#########5| 84000/87599 [00:23<00:01, 3533.03 examples/s]

Map:  97%|#########7| 85000/87599 [00:24<00:00, 3560.50 examples/s]

Map:  98%|#########8| 86000/87599 [00:24<00:00, 3597.68 examples/s]

Map:  99%|#########9| 87000/87599 [00:24<00:00, 3611.61 examples/s]

Map: 100%|##########| 87599/87599 [00:24<00:00, 3602.26 examples/s]
Map: 100%|##########| 87599/87599 [00:30<00:00, 2827.22 examples/s]


Map:   0%|          | 0/10570 [00:00<?, ? examples/s]

Map:   9%|9         | 1000/10570 [00:00<00:02, 4300.94 examples/s]

Map:  19%|#8        | 2000/10570 [00:00<00:01, 4638.20 examples/s]

Map:  28%|##8       | 3000/10570 [00:00<00:01, 4709.76 examples/s]

Map:  38%|###7      | 4000/10570 [00:00<00:01, 4722.79 examples/s]

Map:  47%|####7     | 5000/10570 [00:01<00:01, 4361.73 examples/s]

Map:  57%|#####6    | 6000/10570 [00:01<00:01, 4393.55 examples/s]

Map:  66%|######6   | 7000/10570 [00:01<00:00, 4449.92 examples/s]

Map:  76%|#######5  | 8000/10570 [00:01<00:00, 4448.65 examples/s]

Map:  85%|########5 | 9000/10570 [00:02<00:00, 4446.55 examples/s]

Map:  95%|#########4| 10000/10570 [00:02<00:00, 4463.21 examples/s]

Map: 100%|##########| 10570/10570 [00:02<00:00, 4465.92 examples/s]
Map: 100%|##########| 10570/10570 [00:02<00:00, 3848.95 examples/s]

Establishing a baseline

Next, we’ll train a quick baseline of our model on SQuAD. This task asks our model to identify spans, or segments of text, in a given context (Wikipedia articles) that answer a given question. Running the following code gives me an F1 score of 86.9. This is quite close to the reported NVIDIA score and the difference is likely due to BERT-base vs. BERT-large or fine-tuning hyperparameters.

training_args = transformers.TrainingArguments(
    "trainer",
    num_train_epochs=1,
    lr_scheduler_type="constant",
    per_device_train_batch_size=32,
    per_device_eval_batch_size=256,
    logging_steps=50,
    # Limit max steps for tutorial runners. Delete the below line to see the reported accuracy numbers.
    max_steps=500,
    report_to=None,
)

trainer = transformers.Trainer(
    model,
    training_args,
    train_dataset=tokenized_squad_dataset["train"],
    eval_dataset=tokenized_squad_dataset["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
)

trainer.train()

# batch sizes to compare for eval
batch_sizes = [4, 16, 64, 256]
# 2:4 sparsity require fp16, so we cast here for a fair comparison
with torch.autocast("cuda"):
    with torch.no_grad():
        predictions = trainer.predict(tokenized_squad_dataset["validation"])
        start_logits, end_logits = predictions.predictions
        fp16_baseline = compute_metrics(
            start_logits,
            end_logits,
            tokenized_squad_dataset["validation"],
            squad_dataset["validation"],
        )
        fp16_time = measure_execution_time(
            model,
            batch_sizes,
            tokenized_squad_dataset["validation"],
        )

print("fp16", fp16_baseline)
print("cuda_fp16 time", fp16_time)

import pandas as pd
df = pd.DataFrame(trainer.state.log_history)
df.plot.line(x='step', y='loss', title="Loss vs. # steps", ylabel="loss")

Traceback (most recent call last):
  File "/workspace/tutorials-kr/advanced_source/semi_structured_sparse.py", line 466, in <module>
    fp16_time = measure_execution_time(
  File "/workspace/tutorials-kr/advanced_source/semi_structured_sparse.py", line 384, in measure_execution_time
    p50 = timer.blocked_autorange().median * 1000
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/timer.py", line 372, in blocked_autorange
    number = self._estimate_block_size(min_run_time)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/timer.py", line 319, in _estimate_block_size
    time_taken = self._timeit(number)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/timer.py", line 264, in _timeit
    return max(self._timer.timeit(number), 1e-9)
  File "/usr/lib/python3.10/timeit.py", line 178, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 921, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert
    return _compile(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 676, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 535, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/bytecode_transformation.py", line 1036, in transform_code_object
    transformations(instructions, code_options)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 165, in _fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/convert_frame.py", line 500, in transform
    tracer.run()
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 2149, in run
    super().run()
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 810, in run
    and self.step()
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 773, in step
    getattr(self, inst.opname)(inst)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/symbolic_convert.py", line 2268, in RETURN_VALUE
    self.output.compile_subgraph(
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/output_graph.py", line 1001, in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/output_graph.py", line 1178, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/output_graph.py", line 1251, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/output_graph.py", line 1232, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 1731, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py", line 1330, in compile_fx
    return aot_autograd(
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/backends/common.py", line 58, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py", line 903, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py", line 628, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
  File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 443, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
  File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 648, in aot_wrapper_synthetic_base
    return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
  File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 119, in aot_dispatch_base
    compiled_fw = compiler(fw_module, updated_flat_args)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py", line 1257, in fw_compiler_base
    return inner_compile(
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/debug.py", line 304, in inner
    return fn(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py", line 438, in compile_fx_inner
    compiled_graph = fx_codegen_and_compile(
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py", line 714, in fx_codegen_and_compile
    compiled_fn = graph.compile_to_fn()
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/graph.py", line 1307, in compile_to_fn
    return self.compile_to_module().call
  File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
    r = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/graph.py", line 1254, in compile_to_module
    mod = PyCodeCache.load_by_key_path(
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/codecache.py", line 2160, in load_by_key_path
    exec(code, mod.__dict__, mod.__dict__)
  File "/tmp/torchinductor_root/qx/cqxpwpmbylgebn7m6itljuad5qjyvmgcmjlnag65ak2eiyu32npq.py", line 800, in <module>
    async_compile.wait(globals())
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/codecache.py", line 2715, in wait
    scope[key] = result.result()
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/codecache.py", line 2522, in result
    self.future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpkfpm1erd/main.c', '-O3', '-I/usr/local/lib/python3.10/dist-packages/triton/common/../third_party/cuda/include', '-I/usr/include/python3.10', '-I/tmp/tmpkfpm1erd', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmpkfpm1erd/triton_.cpython-310-x86_64-linux-gnu.so', '-L/lib/x86_64-linux-gnu', '-L/lib/x86_64-linux-gnu']' returned non-zero exit status 1.

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

Pruning BERT to be 2:4 sparse

Now that we have our baseline, it’s time we prune BERT. There are many different pruning strategies, but one of the most common is magnitude pruning, which seeks to remove the weights with the lowest L1 norm. Magnitude pruning was used by NVIDIA in all their results and is a common baseline.

To do this, we will use the torch.ao.pruning package, which contains a weight-norm (magnitude) sparsifier. These sparsifiers work by applying mask parametrizations to the weight tensors in a model. This lets them simulate sparsity by masking out the pruned weights.

We’ll also have to decide what layers of the model to apply sparsity to, which in this case is all of the nn.Linear layers, except for the task-specific head outputs. That’s because semi-structured sparsity has shape constraints, and the task-specific nn.Linear layers do not satisfy them.

sparsifier = WeightNormSparsifier(
    # apply sparsity to all blocks
    sparsity_level=1.0,
    # shape of 4 elements is a block
    sparse_block_shape=(1, 4),
    # two zeros for every block of 4
    zeros_per_block=2
)

# add to config if ``nn.Linear`` and in the BERT model.
sparse_config = [
    {"tensor_fqn": f"{fqn}.weight"}
    for fqn, module in model.named_modules()
    if isinstance(module, nn.Linear) and "layer" in fqn
]

The first step for pruning the model is to insert parametrizations for masking the weights of the model. This is done by the prepare step. Anytime we try to access the .weight we will get mask * weight instead.

# Prepare the model, insert fake-sparsity parametrizations for training
sparsifier.prepare(model, sparse_config)
print(model.bert.encoder.layer[0].output)

Then, we’ll take a single pruning step. All pruners implement a update_mask() method that updates the mask with the logic being determined by the pruner implementation. The step method calls this update_mask functions for the weights specified in the sparse config.

We will also evaluate the model to show the accuracy degradation of zero-shot pruning, or pruning without fine-tuning / retraining.

sparsifier.step()
with torch.autocast("cuda"):
    with torch.no_grad():
        predictions = trainer.predict(tokenized_squad_dataset["validation"])
    pruned = compute_metrics(
        *predictions.predictions,
        tokenized_squad_dataset["validation"],
        squad_dataset["validation"],
    )
print("pruned eval metrics:", pruned)

In this state, we can start fine-tuning the model, updating the elements that wouldn’t be pruned to better account for the accuracy loss. Once we’ve reached a satisfied state, we can call squash_mask to fuse the mask and the weight together. This will remove the parametrizations and we are left with a zeroed-out 2:4 dense model.

trainer.train()
sparsifier.squash_mask()
torch.set_printoptions(edgeitems=4)
print(model.bert.encoder.layer[0].intermediate.dense.weight[:8, :8])

df["sparse_loss"] = pd.DataFrame(trainer.state.log_history)["loss"]
df.plot.line(x='step', y=["loss", "sparse_loss"], title="Loss vs. # steps", ylabel="loss")

Accelerating 2:4 sparse models for inference

Now that we have a model in this format, we can accelerate it for inference just like in the QuickStart Guide.

model = model.cuda().half()
# accelerate for sparsity
for fqn, module in model.named_modules():
    if isinstance(module, nn.Linear) and "layer" in fqn:
        module.weight = nn.Parameter(to_sparse_semi_structured(module.weight))

with torch.no_grad():
    predictions = trainer.predict(tokenized_squad_dataset["validation"])
start_logits, end_logits = predictions.predictions
metrics_sparse = compute_metrics(
    start_logits,
    end_logits,
    tokenized_squad_dataset["validation"],
    squad_dataset["validation"],
)
print("sparse eval metrics: ", metrics_sparse)
sparse_perf = measure_execution_time(
    model,
    batch_sizes,
    tokenized_squad_dataset["validation"],
)
print("sparse perf metrics: ", sparse_perf)

Retraining our model after magnitude pruning has recovered nearly all of the F1 that has been lost when the model was pruned. At the same time we have achieved a 1.28x speedup for bs=16. Note that not all shapes are amenable to performance improvements. When batch sizes are small and limited time is spent in compute sparse kernels may be slower than their dense counterparts.

Because semi-structured sparsity is implemented as a tensor subclass, it is compatible with torch.compile. When composed with to_sparse_semi_structured, we are able to achieve a total 2x speedup on BERT.

Metrics	fp16	2:4 sparse	delta / speedup	compiled
Exact Match (%)	78.53	78.44	-0.09
F1 (%)	86.93	86.49	-0.44
Time (bs=4)	11.10	15.54	0.71x	no
Time (bs=16)	19.35	15.74	1.23x	no
Time (bs=64)	72.71	59.41	1.22x	no
Time (bs=256)	286.65	247.63	1.14x	no
Time (bs=4)	7.59	7.46	1.02x	yes
Time (bs=16)	11.47	9.68	1.18x	yes
Time (bs=64)	41.57	36.92	1.13x	yes
Time (bs=256)	159.22	142.23	1.12x	yes

Conclusion

In this tutorial, we have shown how to prune BERT to be 2:4 sparse and how to accelerate a 2:4 sparse model for inference. By taking advantage of our SparseSemiStructuredTensor subclass, we were able to achieve a 1.3x speedup over the fp16 baseline, and up to 2x with torch.compile. We also demonstrated the benefits of 2:4 sparsity by fine-tuning BERT to recover any lost F1 (86.92 dense vs 86.48 sparse).