I’m a Linux Gamer and for the most part I ignore games that are not released on Linux or at least run well in Wine, however I do have a back catalog of Windows only games from before Steam arrived on Linux.
Dual booting works, but it’s not all that convenient. Having read about the progress in GPU pass-through and project looking glass, I decided to have a go at setting up a Windows VM with GPU pass-through.
I had a strict criteria for this setup. I wanted to still use the dedicated GPU (dGPU) on the Linux Host for games and programs or in the Windows VM without having to reboot, run any scripts, log out or otherwise restart the X server.
Here’s how I accomplished that.
Once setup, usage is virtually* transparent allowing you to use your X Session as normal. Programs that need the dGPU will automatically use it and anytime you start a VM setup for pass-through the dGPU will be automatically assigned to the VM. With the proviso that the Linux Host and VM Guest cannot both use the dGPU at the same time.
As with most setups you need two graphics cards although one can be (and in my case is) the on-board intel iGPU.
The setup I’ve gone with makes use of PRIME GPU Offloading and whilst this post is quite verbose, it really requires only two steps to configure the Host and adding the dGPU / dGPU’s audio to the VM via virt-manager.
As there are many guides available that cover selecting suitable hardware to use gpu pass-through I will not recover that. Suffice it to say your hardware must support VT-D/iommu or the AMD equivalent.
I use an i5-3570 CPU, GA-B75-D3V motherboard and a Sapphire 8GB AMD RX580 Nitro+ and the Linux Host runs Debian Stretch with backports for kernel 4.17 and mesa.
I’m not sure if this PRIME setup will work with NVidia cards as I’ve read the drivers may not support rebinding. Also don’t expect NVidia to be receptive to any bug reports as they actively block the usage of their cards in VMs (Search Error 43). This and a few other reasons are why my upgrade from a GTX 660 was to a AMD RX580 and not a GTX 1070.
1. Base Setup
Ensure VT-D is enabled in bios and the integrated GPU enabled as the primary card. Also IOMMU should be enabled via a kernel option in /etc/defaults/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
and run update-grub. AMD CPU users should instead use amd_iommu=on.
Many guides will also cover loading vfio modules, adding them to initramfs to load early and setting options for vfio ids to have it bind to your dGPU rather than the normal amdgpu. These steps are not required for this setup and should not be done.
2. XOrg Configuration
X should be configured such that the server runs on the integrated graphics card to which your monitor should be connected. In addition Xorg must be told not to claim the dGPU.
Create /etc/X11/xorg.conf.d/10-passthrough.conf with the following content and restart.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Section "Device" Identifier "Intel Graphics" Driver "intel" Option "DRI" "3" EndSection Section "ServerFlags" Option "AutoAddGPU" "off" EndSection Section "Device" Identifier "AMDGPU" Driver "amdgpu" Option "DRI3" "1" Option "Ignore" "1" EndSection
This will allow Xorg to use the intel iGPU device but (line 8 & 15) to not detect or auto configure the dGPU. Note the usage of DRI3 also for PRIME support. Running “xrandr –listproviders” should show only your iGPU. It should not list the dGPU.
gary@icarus:$ xrandr --listproviders Providers: number : 1 Provider 0: id: 0x47 cap: 0xb, Source Output, Sink Output, Sink Offload crtcs: 4 outputs: 4 associated providers: 0 name:Intel
With those changes made you will now be using the on-board integrated intel graphics, which you can verify with glxgears.
gary@icarus:$ glxgears -info | grep GL_REN GL_RENDERER = Mesa DRI Intel(R) Ivybridge Desktop
To force a program to run on the dGPU simply prefix with DRI_PRIME=1.
gary@icarus:$ DRI_PRIME=1 glxgears -info | grep GL_REN GL_RENDERER = Radeon RX 580 Series (POLARIS10 / DRM 3.25.0 / 4.17.0-0.bpo.1-amd64, LLVM 5.0.1)
This is much the same as you’d use a Laptop with a Hybrid graphics card setup.
PRIME Auto GPU Switching
PRIME on a couple of games appears to automatically offload to the dGPU. I’m not sure what heuristic it uses for this or if the game’s launch script is perhaps setting DRI_PRIME=1. For example running F1 2017 from steam and checking lsof shows F1 is using the dGPU (renderD128 is the intel iGPU whilst renderD129 is the RX580 dGPU)
1 2 3 4 5 6 7 8 9 10 11 12 13
gary@icarus:$ lsof /dev/dri/* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME xfwm4 1964 gary 8u CHR 226,0 0t0 16131 /dev/dri/card0 steam 6104 gary 21u CHR 226,128 0t0 16130 /dev/dri/renderD128 steam 6104 gary 114u CHR 226,128 0t0 16130 /dev/dri/renderD128 steamwebh 6113 gary 15u CHR 226,128 0t0 16130 /dev/dri/renderD128 F12017 6314 gary mem CHR 226,129 16201 /dev/dri/renderD129 F12017 6314 gary 21u CHR 226,128 0t0 16130 /dev/dri/renderD128 F12017 6314 gary 39u CHR 226,129 0t0 16201 /dev/dri/renderD129 F12017 6314 gary 40u CHR 226,129 0t0 16201 /dev/dri/renderD129 F12017 6314 gary 41u CHR 226,128 0t0 16130 /dev/dri/renderD128 F12017 6314 gary 42u CHR 226,129 0t0 16201 /dev/dri/renderD129 F12017 6314 gary 45u CHR 226,128 0t0 16130 /dev/dri/renderD128
However, some games like geometry wars will still run on the iGPU.
gary@icarus:$ lsof /dev/dri/* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME xfwm4 1964 gary 8u CHR 226,0 0t0 16131 /dev/dri/card0 steam 6104 gary 21u CHR 226,128 0t0 16130 /dev/dri/renderD128 steam 6104 gary 114u CHR 226,128 0t0 16130 /dev/dri/renderD128 steamwebh 6113 gary 15u CHR 226,128 0t0 16130 /dev/dri/renderD128 GeometryW 10553 gary 31u CHR 226,128 0t0 16130 /dev/dri/renderD128
In order to force any game or program to run on the dGPU, set the env variable “DRI_PRIME=1”. For steam games, right click on the game, select “properties” and “set launch options” and enter
and the game will now launch on the dGPU, alternatively
export DRI_PRIME=1 steam &
Will ensure all games launched from steam will do so on the dGPU. Just remember to close steam before starting a VM otherwise the dGPU will not be available for pass-through.
Others have covered VM setup in detail so I’ll just refer you to part 4 of Alex’s GPU pass-through series. The only real difference from his post is I used the Q35 chipset, a raw partition for the disk and didn’t setup huge pages.
The rest of his series is also worth reading, although keep in mind the setup he uses is slightly different to the one I have covered in this blog which is why there are additional steps to load vfio-pci modules and set options to allow it to claim the dGPU.
With the above done you should now be able to make use of your dedicated GPU on the Linux Host or from within a VM Guest.
So what’s the performance like with this setup?
For comparison, below are fps stats from running F1 2017’s benchmark for 1 lap of the Australian circuit on ultra-high settings, 16xAA and TAA.
|OS||Min FPS||Avg FPS||Max FPS|
|Linux (VM)||Not Tested|
As far as Windows Native vs VM goes, there’s not that much between the two. As for the Linux results, I believe the different fps result when running via PRIME may be impacted by VSync so the two Linux results are not really comparable. Linux does appear to be running at a loss of about 20fps compared to Windows however :(
This setup does come with a few caveats to keep in mind.
Once configured your monitor will be connected to the on-board GPU output and the Linux Host will output using the on-board graphics regardless of whether it’s offloading to the dGPU or not.
When passed through to a VM Guest such as Windows, the dGPU’s outputs will instead be used. Thus you need either two monitors, a monitor with multiple inputs or Project Looking Glass.
I went with a monitor that has multiple inputs although I’m considering looking into Project Looking Glass in the future.
Mouse and Keyboard
I pass my keyboard and mouse through to the VM to avoid any input latency. This prevents their usage in the Linux Host whilst the VM is running. This is not normally an issue, but should the VM crash or you want to use the Linux Host temporarily, it can be useful to have a spare mouse and keyboard, input switch or a program like Synergy.
In theory enabling the iommu kernel flag, creating the Xorg config and then adding the PCI GPU and GPU audio devices to the VM should be all you need to do. In practice…
Check which iommu group your dGPU is part of. In my case it’s group 1 which contains the dGPU (01:00.0), the dGPU’s audio (01:00.1) and a controller (00:01.0).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/7/devices/0000:00:1c.0 /sys/kernel/iommu_groups/5/devices/0000:00:1a.0 /sys/kernel/iommu_groups/3/devices/0000:00:14.0 /sys/kernel/iommu_groups/11/devices/0000:00:1f.2 /sys/kernel/iommu_groups/11/devices/0000:00:1f.0 /sys/kernel/iommu_groups/11/devices/0000:00:1f.3 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/8/devices/0000:00:1c.2 /sys/kernel/iommu_groups/6/devices/0000:00:1b.0 /sys/kernel/iommu_groups/4/devices/0000:00:16.0 /sys/kernel/iommu_groups/12/devices/0000:03:00.0 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/10/devices/0000:00:1e.0 /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0
If you have the PCIe controller in the same IOMMU group as your dGPU, DO NOT pass the controller through. If you have any further devices in the same group, you WILL need to pass all those devices through too. If this cannot be done, you may need to read up on the ACS kernel patch.
When shutting down the VM Guest, the Guest may freeze or in other instances, the Guest shuts down but is unable to be restarted and the dGPU is no longer usable by the Host.
00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04) Subsystem: Gigabyte Technology Co., Ltd 7 Series/C216 Chipset Family High Definition Audio Controller Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev e7) Subsystem: Device 1da2:e366 Kernel driver in use: amdgpu Kernel modules: amdgpu 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aaf0 Subsystem: Device 1da2:aaf0 Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel
Note the dGPU audio and motherboard’s on-board audio both use snd_hda_intel. Whilst they are in different IOMMU groups and should both need to be passed through, I had to also pass through the on-board audio. If you experience any lockups, check if any other device shares the same kernel modules as your GPU/GPU Audio.
KMODE Exception BSOD
When running F1 2017 the VM Guest will BSOD with the error KMODE_EXCEPTION_NOT_HANDLED.
As noted on the kernel mailing list kvm is causing this crash when it sees an unknown msrs. You can configure KVM to not do this by creating /etc/modprobe.d/kvm.conf with:-
options kvm ignore_msrs=Y options kvm report_ignored_msrs=N
Host Kernel Oops
This is the only issue I’ve run into so far and not solved.
Using the VM causes the dGPU to be rebound from the amdgpu kernel module to the vfio-pci module and vice-versa when the VM is shutdown. Any time the host is shut down or restarted after such a rebind has taken place, a kernel oops occurs
If anyone knows how to resolve this, please let me know.