I’m a Linux Gamer and for the most part I ignore games that are not released on Linux or at least run well in Wine, however I do have a back catalog of Windows only games from before Steam arrived on Linux.

Dual booting works, but it’s not all that convenient. Having read about the progress in GPU pass-through and project looking glass, I decided to have a go at setting up a Windows VM with GPU pass-through.

I had a strict criteria for this setup. I wanted to still use the dedicated GPU (dGPU) on the Linux Host for games and programs or in the Windows VM without having to reboot, run any scripts, log out or otherwise restart the X server.

Here’s how I accomplished that.

Overview

Once setup, usage is virtually* transparent allowing you to use your X Session as normal. Programs that need the dGPU will automatically use it and anytime you start a VM setup for pass-through the dGPU will be automatically assigned to the VM. With the proviso that the Linux Host and VM Guest cannot both use the dGPU at the same time.

As with most setups you need two graphics cards although one can be (and in my case is) the on-board intel iGPU.

Setup

The setup I’ve gone with makes use of PRIME GPU Offloading and whilst this post is quite verbose, it really requires only two steps to configure the Host and adding the dGPU / dGPU’s audio to the VM via virt-manager.

As there are many guides available that cover selecting suitable hardware to use gpu pass-through I will not recover that. Suffice it to say your hardware must support VT-D/iommu or the AMD equivalent.

I use an i5-3570 CPU, GA-B75-D3V motherboard and a Sapphire 8GB AMD RX580 Nitro+ and the Linux Host runs Debian Stretch with backports for kernel 4.17 and mesa.

I’m not sure if this PRIME setup will work with NVidia cards as I’ve read the drivers may not support rebinding. Also don’t expect NVidia to be receptive to any bug reports as they actively block the usage of their cards in VMs (Search Error 43). This and a few other reasons are why my upgrade from a GTX 660 was to a AMD RX580 and not a GTX 1070.

1. Base Setup

Ensure VT-D is enabled in bios and the integrated GPU enabled as the primary card. Also IOMMU should be enabled via a kernel option in /etc/defaults/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

and run update-grub. AMD CPU users should instead use amd_iommu=on.

Many guides will also cover loading vfio modules, adding them to initramfs to load early and setting options for vfio ids to have it bind to your dGPU rather than the normal amdgpu. These steps are not required for this setup and should not be done.

2. XOrg Configuration

X should be configured such that the server runs on the integrated graphics card to which your monitor should be connected. In addition Xorg must be told not to claim the dGPU.

Create /etc/X11/xorg.conf.d/10-passthrough.conf with the following content and restart.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
Section "Device"
    Identifier "Intel Graphics"
    Driver "intel"
    Option "DRI" "3"
EndSection

Section "ServerFlags" 
	Option "AutoAddGPU" "off"
EndSection

Section "Device"
    Identifier "AMDGPU"
    Driver "amdgpu"
    Option "DRI3" "1"
    Option "Ignore" "1"
EndSection

This will allow Xorg to use the intel iGPU device but (line 8 & 15) to not detect or auto configure the dGPU. Note the usage of DRI3 also for PRIME support. Running “xrandr –listproviders” should show only your iGPU. It should not list the dGPU.

gary@icarus:$ xrandr --listproviders 
Providers: number : 1
Provider 0: id: 0x47 cap: 0xb, Source Output, Sink Output, Sink Offload crtcs: 4 outputs: 4 associated providers: 0 name:Intel

With those changes made you will now be using the on-board integrated intel graphics, which you can verify with glxgears.

gary@icarus:$ glxgears -info | grep GL_REN
GL_RENDERER   = Mesa DRI Intel(R) Ivybridge Desktop 

To force a program to run on the dGPU simply prefix with DRI_PRIME=1.

gary@icarus:$ DRI_PRIME=1 glxgears -info | grep GL_REN
GL_RENDERER   = Radeon RX 580 Series (POLARIS10 / DRM 3.25.0 / 4.17.0-0.bpo.1-amd64, LLVM 5.0.1)

This is much the same as you’d use a Laptop with a Hybrid graphics card setup.

PRIME Auto GPU Switching

PRIME on a couple of games appears to automatically offload to the dGPU. I’m not sure what heuristic it uses for this or if the game’s launch script is perhaps setting DRI_PRIME=1. For example running F1 2017 from steam and checking lsof shows F1 is using the dGPU (renderD128 is the intel iGPU whilst renderD129 is the RX580 dGPU)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
gary@icarus:$ lsof /dev/dri/*
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF  NODE NAME
xfwm4     1964 gary    8u   CHR   226,0      0t0 16131 /dev/dri/card0
steam     6104 gary   21u   CHR 226,128      0t0 16130 /dev/dri/renderD128
steam     6104 gary  114u   CHR 226,128      0t0 16130 /dev/dri/renderD128
steamwebh 6113 gary   15u   CHR 226,128      0t0 16130 /dev/dri/renderD128
F12017    6314 gary  mem    CHR 226,129          16201 /dev/dri/renderD129
F12017    6314 gary   21u   CHR 226,128      0t0 16130 /dev/dri/renderD128
F12017    6314 gary   39u   CHR 226,129      0t0 16201 /dev/dri/renderD129
F12017    6314 gary   40u   CHR 226,129      0t0 16201 /dev/dri/renderD129
F12017    6314 gary   41u   CHR 226,128      0t0 16130 /dev/dri/renderD128
F12017    6314 gary   42u   CHR 226,129      0t0 16201 /dev/dri/renderD129
F12017    6314 gary   45u   CHR 226,128      0t0 16130 /dev/dri/renderD128

However, some games like geometry wars will still run on the iGPU.

gary@icarus:$ lsof /dev/dri/*
COMMAND     PID USER   FD   TYPE  DEVICE SIZE/OFF  NODE NAME
xfwm4      1964 gary    8u   CHR   226,0      0t0 16131 /dev/dri/card0
steam      6104 gary   21u   CHR 226,128      0t0 16130 /dev/dri/renderD128
steam      6104 gary  114u   CHR 226,128      0t0 16130 /dev/dri/renderD128
steamwebh  6113 gary   15u   CHR 226,128      0t0 16130 /dev/dri/renderD128
GeometryW 10553 gary   31u   CHR 226,128      0t0 16130 /dev/dri/renderD128

In order to force any game or program to run on the dGPU, set the env variable “DRI_PRIME=1”. For steam games, right click on the game, select “properties” and “set launch options” and enter

DRI_PRIME=1 %command%

and the game will now launch on the dGPU, alternatively

export DRI_PRIME=1
steam &

Will ensure all games launched from steam will do so on the dGPU. Just remember to close steam before starting a VM otherwise the dGPU will not be available for pass-through.

VM Configuration

Others have covered VM setup in detail so I’ll just refer you to part 4 of Alex’s GPU pass-through series. The only real difference from his post is I used the Q35 chipset, a raw partition for the disk and didn’t setup huge pages.

The rest of his series is also worth reading, although keep in mind the setup he uses is slightly different to the one I have covered in this blog which is why there are additional steps to load vfio-pci modules and set options to allow it to claim the dGPU.

Benchmarks

With the above done you should now be able to make use of your dedicated GPU on the Linux Host or from within a VM Guest.

So what’s the performance like with this setup?

F1 2017 Benchmark, Windows VM

F1 2017 Benchmark, Windows VM

For comparison, below are fps stats from running F1 2017’s benchmark for 1 lap of the Australian circuit on ultra-high settings, 16xAA and TAA.

OS Min FPS Avg FPS Max FPS
Linux (Native) 39 57 84
Linux (Prime) 39 53 66
Linux (VM) Not Tested
Windows (Native) 71 86 109
Windows (VM) 60 81 105

As far as Windows Native vs VM goes, there’s not that much between the two. As for the Linux results, I believe the different fps result when running via PRIME may be impacted by VSync so the two Linux results are not really comparable. Linux does appear to be running at a loss of about 20fps compared to Windows however :(

Caveats

This setup does come with a few caveats to keep in mind.

Monitors

Once configured your monitor will be connected to the on-board GPU output and the Linux Host will output using the on-board graphics regardless of whether it’s offloading to the dGPU or not.

When passed through to a VM Guest such as Windows, the dGPU’s outputs will instead be used. Thus you need either two monitors, a monitor with multiple inputs or Project Looking Glass.

I went with a monitor that has multiple inputs although I’m considering looking into Project Looking Glass in the future.

Mouse and Keyboard

I pass my keyboard and mouse through to the VM to avoid any input latency. This prevents their usage in the Linux Host whilst the VM is running. This is not normally an issue, but should the VM crash or you want to use the Linux Host temporarily, it can be useful to have a spare mouse and keyboard, input switch or a program like Synergy.

Troubleshooting

In theory enabling the iommu kernel flag, creating the Xorg config and then adding the PCI GPU and GPU audio devices to the VM should be all you need to do. In practice…

IOMMU Groups

Check which iommu group your dGPU is part of. In my case it’s group 1 which contains the dGPU (01:00.0), the dGPU’s audio (01:00.1) and a controller (00:01.0).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:1a.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/8/devices/0000:00:1c.2
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/12/devices/0000:03:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1e.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1d.0

If you have the PCIe controller in the same IOMMU group as your dGPU, DO NOT pass the controller through. If you have any further devices in the same group, you WILL need to pass all those devices through too. If this cannot be done, you may need to read up on the ACS kernel patch.

Host/Guest Hangs

When shutting down the VM Guest, the Guest may freeze or in other instances, the Guest shuts down but is unable to be restarted and the dGPU is no longer usable by the Host.

00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family High Definition Audio Controller
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev e7)
	Subsystem: Device 1da2:e366
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0
	Subsystem: Device 1da2:aaf0
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

Note the dGPU audio and motherboard’s on-board audio both use snd_hda_intel. Whilst they are in different IOMMU groups and should both need to be passed through, I had to also pass through the on-board audio. If you experience any lockups, check if any other device shares the same kernel modules as your GPU/GPU Audio.

KMODE Exception BSOD

When running F1 2017 the VM Guest will BSOD with the error KMODE_EXCEPTION_NOT_HANDLED.

As noted on the kernel mailing list kvm is causing this crash when it sees an unknown msrs. You can configure KVM to not do this by creating /etc/modprobe.d/kvm.conf with:-

options kvm ignore_msrs=Y
options kvm report_ignored_msrs=N

Host Kernel Oops

This is the only issue I’ve run into so far and not solved.

Using the VM causes the dGPU to be rebound from the amdgpu kernel module to the vfio-pci module and vice-versa when the VM is shutdown. Any time the host is shut down or restarted after such a rebind has taken place, a kernel oops occurs

Shutdown Kernel Oops post GPU rebind

Shutdown Kernel Oops post GPU rebind

If anyone knows how to resolve this, please let me know.