kvm 透传显卡至win10虚拟机的方法

环境

已安装nvidia 显卡驱动
操作系统:CentOS Linux release 7.9.2009 (Core)
内核版本:Linux 5.4.135-1.el7.elrepo.x86_64
显卡型号:rtx 6000

kvm版本:
virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API : QEMU 4.5.0
Running hypervisor: QEMU 1.5.3

解绑GPU 设备

1.配置GRUB_CMDLINE_LINUX(实测，有效)，查询要解绑的设备，记录设备id号

lspci -nn|grep NVIDIA |grep VGA
或者 lspci -nn|grep NVIDIA

2.修改 /etc/default/grub文件的GRUB_CMDLINE_LINUX

vi /etc/default/grub
以下是内容，新增加 GRUB_CMDLINE_LINUX行: intel_iommu=on pci-stub.ids=10de:1e30\”
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=\”$(sed \’s, release .*$,,g\’ /etc/system-release)\”
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT=\”console\”
GRUB_CMDLINE_LINUX=\”intel_iommu=on crashkernel=auto spectre_v2=retpoline rhgb quiet rdblacklist=nouveau pci-stub.ids=10de:1e30\”
GRUB_DISABLE_RECOVERY=\”true\”

3.更新grub2 conf，执行如下命令

grub2-mkconfig -o /boot/grub2/grub.cfg

4.重启系统

reboot

验证并查看nvidia 显卡

1.执行nvidia-smi ，看不到显卡，证明显卡已经被解绑

# nvidia-smi
NVIDIA-SMI has failed because it couldn\’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

2.验证内核支持iommu

cat /proc/cmdline | grep iommu
结果输出: intel_iommu=on

3.验证VT-D是否启用

dmesg |grep -e DMAR -e IOMMU

安装win10虚拟机

qemu-img create -f qcow2 /home/kvm-machines/shaoyan.qcow2 200G
virt-install –name shaoyan –ram 4096 –vcpus=8 –os-type=windows –accelerate –cdrom=/home/kvm_images/win10.iso –disk path=/home/kvm-machines/shaoyan.qcow2,format=qcow2,bus=ide –bridge=br0 –autostart –vnc –vncport=5910 –vnclisten=0.0.0.0

宿主机安装软件包

yum -y groups install \’GNOME Desktop\’ virt-manager virt-viewer dejavu-lgc-sans-fonts

显卡加载vfio驱动1.显卡加载vfio驱动

modprobe vfio
modprobe vfio-pci

2.从服务器卸载1张显卡的4个设备

virsh nodedev-detach pci_0000_b1_00_0
virsh nodedev-detach pci_0000_b1_00_1
virsh nodedev-detach pci_0000_b1_00_2
virsh nodedev-detach pci_0000_b1_00_3

3.验证显卡设备是否加载vfio-pci驱动,加载成功

# lspci -vv -s b1:00.0 | grep driver
Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.1 | grep driver
Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.2 | grep driver
Kernel driver in use: vfio-pci
[root@rtx_6000 10.147.100.107 ~ ]
# lspci -vv -s b1:00.3 | grep driver
Kernel driver in use: vfio-pci

win10虚拟机添加NVDIA显卡1.通过远控卡登录服务器透传显卡，执行startx 调出可视化页面

startx

2.添加显卡硬件

3.查看显卡

显卡驱动安装1.要安装显卡驱动，需要修改虚拟机win10的配置文件否则显卡驱动安装过程中自检会报错ERROR CODE 43，驱动不允许安装在虚拟机上。

#编辑VM XML
virsh edit shaoyan

#修改第一行
<domain type=\’kvm\’>
#为
<domain type=\’kvm\’ xmlns:qemu=\’http://libvirt.org/schemas/domain/qemu/1.0\’>

#找到<features>节点，添加内容
<features>
…
<kvm><hidden state=\’on\’/></kvm>
…
</features>

#找到</device>节点，在device后添加内容
<qemu:commandline><qemu:arg value=\’-cpu\’/><qemu:arg value=\’host,hv_time,kvm=off,hv_vendor_id=null\’/>
</qemu:commandline>

#重启VM
virsh destroy shaoyan
virsh start shaoyan

2.附录完整xml 文件如下，如果是拷贝至其他宿主机配置文件的话，只需要保留xml 文件中的第一组hostdev即可，否则会报错

# cat /etc/libvirt/qemu/shaoyan.xml

<!–
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit shaoyan
or other application using the libvirt API.
–>

<domain type=\’kvm\’>
<name>shaoyan</name>
<uuid>e06c6c3b-0a57-43e8-9d6f-02fd1e2aad83</uuid>
<title>gpu</title>
<description>gpu</description>
<memory unit=\’KiB\’>16777216</memory>
<currentMemory unit=\’KiB\’>16777216</currentMemory>
<vcpu placement=\’static\’>16</vcpu>
<os>
<type arch=\’x86_64\’ machine=\’pc-i440fx-rhel7.0.0\’>hvm</type>
<boot dev=\’hd\’/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state=\’on\’/>
<vapic state=\’on\’/>
<spinlocks state=\’on\’ retries=\’8191\’/>
</hyperv>
</features>
<cpu mode=\’host-passthrough\’ check=\’none\’/>
<clock offset=\’localtime\’>
<timer name=\’rtc\’ tickpolicy=\’catchup\’/>
<timer name=\’pit\’ tickpolicy=\’delay\’/>
<timer name=\’hpet\’ present=\’no\’/>
<timer name=\’hypervclock\’ present=\’yes\’/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled=\’no\’/>
<suspend-to-disk enabled=\’no\’/>
</pm>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type=\’file\’ device=\’disk\’>
<driver name=\’qemu\’ type=\’qcow2\’/>
<source file=\’/data/kvm/shaoyan.qcow2\’/>
<target dev=\’hda\’ bus=\’ide\’/>
<address type=\’drive\’ controller=\’0\’ bus=\’0\’ target=\’0\’ unit=\’0\’/>
</disk>
<controller type=\’usb\’ index=\’0\’ model=\’ich9-ehci1\’>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x04\’ function=\’0x7\’/>
</controller>
<controller type=\’usb\’ index=\’0\’ model=\’ich9-uhci1\’>
<master startport=\’0\’/>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x04\’ function=\’0x0\’ multifunction=\’on\’/>
</controller>
<controller type=\’usb\’ index=\’0\’ model=\’ich9-uhci2\’>
<master startport=\’2\’/>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x04\’ function=\’0x1\’/>
</controller>
<controller type=\’usb\’ index=\’0\’ model=\’ich9-uhci3\’>
<master startport=\’4\’/>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x04\’ function=\’0x2\’/>
</controller>
<controller type=\’pci\’ index=\’0\’ model=\’pci-root\’/>
<controller type=\’ide\’ index=\’0\’>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x01\’ function=\’0x1\’/>
</controller>
<interface type=\’bridge\’>
<mac address=\’52:54:00:48:b2:5a\’/>
<source bridge=\’br0\’/>
<model type=\’e1000\’/>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x03\’ function=\’0x0\’/>
</interface>
<serial type=\’pty\’>
<target type=\’isa-serial\’ port=\’0\’>
<model name=\’isa-serial\’/>
</target>
</serial>
<console type=\’pty\’>
<target type=\’serial\’ port=\’0\’/>
</console>
<input type=\’tablet\’ bus=\’usb\’>
<address type=\’usb\’ bus=\’0\’ port=\’1\’/>
</input>
<input type=\’mouse\’ bus=\’ps2\’/>
<input type=\’keyboard\’ bus=\’ps2\’/>
<graphics type=\’vnc\’ port=\’5910\’ autoport=\’no\’ listen=\’0.0.0.0\’>
<listen type=\’address\’ address=\’0.0.0.0\’/>
</graphics>
<video>
<model type=\’vga\’ vram=\’16384\’ heads=\’1\’ primary=\’yes\’/>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x02\’ function=\’0x0\’/>
</video>
<hostdev mode=\’subsystem\’ type=\’pci\’ managed=\’yes\’>
<source>
<address domain=\’0x0000\’ bus=\’0xb1\’ slot=\’0x00\’ function=\’0x0\’/>
</source>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x06\’ function=\’0x0\’/>
</hostdev>
<hostdev mode=\’subsystem\’ type=\’pci\’ managed=\’yes\’>
<source>
<address domain=\’0x0000\’ bus=\’0xb1\’ slot=\’0x00\’ function=\’0x1\’/>
</source>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x07\’ function=\’0x0\’/>
</hostdev>
<hostdev mode=\’subsystem\’ type=\’pci\’ managed=\’yes\’>
<source>
<address domain=\’0x0000\’ bus=\’0xb1\’ slot=\’0x00\’ function=\’0x2\’/>
</source>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x08\’ function=\’0x0\’/>
</hostdev>
<hostdev mode=\’subsystem\’ type=\’pci\’ managed=\’yes\’>
<source>
<address domain=\’0x0000\’ bus=\’0xb1\’ slot=\’0x00\’ function=\’0x3\’/>
</source>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x09\’ function=\’0x0\’/>
</hostdev>
<memballoon model=\’virtio\’>
<address type=\’pci\’ domain=\’0x0000\’ bus=\’0x00\’ slot=\’0x05\’ function=\’0x0\’/>
</memballoon>
</devices>
</domain>