(Prototype) Use iOS GPU in PyTorch

Author: Tao Xu


This tutorial introduces the steps to run your models on iOS GPU. We’ll be using the mobilenetv2 model as an example. Since the mobile GPU features are currently in the prototype stage, you’ll need to build a custom pytorch binary from source. For the time being, only a limited number of operators are supported, and certain client side APIs are subject to change in the future versions.

Model Preparation

Since GPUs consume weights in a different order, the first step we need to do is to convert our TorchScript model to a GPU compatible model. This step is also known as “prepacking”. To do that, we’ll build a custom pytorch binary from source that includes the Metal backend. Go ahead checkout the pytorch source code from github and run the command below

USE_PYTORCH_METAL=ON python install --cmake

The command above will build a custom pytorch binary from master. The install argument simply tells to override the existing PyTorch on your desktop. Once the build finished, open another terminal to check the PyTorch version to see if the installation was successful. As the time of writing of this recipe, the version is 1.8.0a0+41237a4. You might be seeing different numbers depending on when you check out the code from master, but it should be greater than 1.7.0.

import torch
torch.__version__ #1.8.0a0+41237a4

The next step is going to be converting the mobilenetv2 torchscript model to a Metal compatible model. We’ll be leveraging the optimize_for_mobile API from the torch.utils module. As shown below

import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

model = torchvision.models.mobilenet_v2(pretrained=True)
scripted_model = torch.jit.script(model)
optimized_model = optimize_for_mobile(scripted_model, backend='metal')
print(torch.jit.export_opnames(optimized_model)), './')

Note that the torch.jit.export_opnames(optimized_model) is going to dump all the optimized operators from the optimized_mobile. If everything works well, you should be able to see the following ops being printed out from the console


Those are all the ops we need to run the mobilenetv2 model on iOS GPU. Cool! Now that you have the saved on your disk, let’s move on to the iOS part.

Use C++ APIs

In this section, we’ll be using the HelloWorld example to demonstrate how to use the C++ APIs. The first thing we need to do is to build a custom LibTorch from Source. Make sure you have deleted the build folder from the previous step in PyTorch root directory. Then run the command below

IOS_ARCH=arm64 USE_PYTORCH_METAL=1 ./scripts/

Note IOS_ARCH tells the script to build a arm64 version of Libtorch. This is because in PyTorch, Metal is only available for the iOS devices that support the Apple A9 chip or above. Once the build finished, follow the Build PyTorch iOS libraries from source section from the iOS tutorial to setup the XCode settings properly. Don’t forget to copy the ./ to your XCode project.

Next we need to make some changes in

// #import <Libtorch-Lite.h>
// If it's built from source with xcode, comment out the line above
// and use following headers
#include <torch/csrc/jit/mobile/import.h>
#include <torch/csrc/jit/mobile/module.h>
#include <torch/script.h>

- (NSArray<NSNumber*>*)predictImage:(void*)imageBuffer {
  torch::jit::GraphOptimizerEnabledGuard opguard(false);
  at::Tensor tensor = torch::from_blob(imageBuffer, {1, 3, 224, 224}, at::kFloat).metal();
  auto outputTensor = _impl.forward({tensor}).toTensor().cpu();

As you can see, we simply just call .metal() to move our input tensor from CPU to GPU, and then call .cpu() to move the result back. Internally, .metal() will copy the input data from the CPU buffer to a GPU buffer with a GPU compatible memory format. When .cpu() is invoked, the GPU command buffer will be flushed and synced. After forward finished, the final result will then be copied back from the GPU buffer back to a CPU buffer.

The last step we have to do is to add the Accelerate.framework and the MetalShaderPerformance.framework to your xcode project.

If everything works fine, you should be able to see the inference results on your phone. The result below was captured from an iPhone11 device

- timber wolf, grey wolf, gray wolf, Canis lupus
- malamute, malemute, Alaskan malamute
- Eskimo dog, husky

You may notice that the results are slighly different from the results we got from the CPU model as shown in the iOS tutorial. This is because by default Metal uses fp16 rather than fp32 to compute. The precision loss is expected.


In this tutorial, we demonstrated how to convert a mobilenetv2 model to a GPU compatible model. We walked through a HelloWorld example to show how to use the C++ APIs to run models on iOS GPU. Please be aware of that GPU feature is still under development, new operators will continue to be added. APIs are subject to change in the future versions.

Thanks for reading! As always, we welcome any feedback, so please create an issue here if you have any.

Learn More

Rate this Tutorial

© Copyright 2021, PyTorch & PyTorch Korea Community.

Built with Sphinx using a theme provided by Read the Docs.

공식 문서 (영어)

PyTorch 공식 문서입니다.

공식 문서로 이동

한국어 튜토리얼

한국어로 번역 중인 PyTorch 튜토리얼입니다.

튜토리얼로 이동


다른 사용자들과 의견을 나눠보세요!

커뮤니티로 이동