openstack透传GPU配置
一、普通gpu透传步骤
1: 在BIOS中打开硬件辅助虚拟化功能⽀持
对于intel cpu, 在主板中开启VT-x及VT-d选项,VT-x为开启虚拟化需要 VT-d为开启PCI passthrough。这两个选项⼀般在BIOS中Advance下CPU和System或相关条⽬中设置。例如:
- VT: Intel Virtualization Technology
- VT-d: Intel VT for Directed I/O
对于 amd cpu, 在主板中开启SVM及IOMMU选项,SVM为开启虚拟化需要,IOMMU为开启PCI passthrough
2:确认内核⽀持iommu
cat /proc/cmdline | grep iommu
如果没有输出, 则需要修改kernel启动参数。
对于intel cpu
1. 编辑 /etc/default/grub ⽂件, 在 GRUB_CMDLINE_LINUX ⾏后⾯添加: intel_iommu=on
例如:
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on"
如果没有 GRUB_CMDLINE_LINUX , 则使⽤ GRUB_CMDLINE_LINUX_DEFAULT
2. 更新
grub grub2-mkconfig -o /boot/grub2/grub.cfg
如果是uefi启动,需要修改启动文件,如下:
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
reboot
重启机器
对于amd cpu 与intel cpu的区别为, 添加的不是 intel_iommu=on , ⽽是 iommu=on , 其他步骤⼀样。
3:从默认驱动程序解绑GPU设备(如果设备之前没有提供其他程序使用可忽略)
echo "23e2 0100" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:81:00.0" > /sys/bus/pci/devices/0000:81:00.0/driver/unbind
echo "0000:81:00.0" > /sys/bus/pci/drivers/pci-stub/bind
至此,准备透传的GPU卡已准备就绪。
4:确认pci设备驱动信息
[root@compute01 ~]# lspci -nn |grep -i 23e2
31:00.0 Processing accelerators [1200]: Device [23e2:0100]
verdor ID为23e2 project ID为0100
二、配置openstack
1:配置nova-scheduler
在filter_scheduler中加⼊ PciPassthroughFilter , 同时添加 available_filters =nova.scheduler.filters.all_filters
vim /etc/nova/nova.conf
[filter_scheduler]
host_subset_size = 10
max_io_ops_per_host = 10
enabled_filters = RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,AggregateDiskFilter,DifferentHostFilter,SameHostFilter,PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters
2:配置nova-api
添加新的块pci
[pci]
alias = {"vendor_id":"23e2","product_id":"0100","device_type":"type-PCI","name":"a1"}
重启api以及scheduler服务
systemctl restart openstack-nova-api openstack-nova-scheduler
3:配置透传设备所在的计算节点
vim /etc/nova/nova.conf
[pci]
passthrough_whitelist = { "vendor_id":"23e2","product_id":"0100" }
alias = { "vendor_id":"23e2", "product_id":"0100", "device_type":"type-PCI", "name":"a1" }
重启nova-compute服务
systemctl restart libvirtd openstack-nova-compute
注意:如果是网卡"device_type":“type-PF”。如果是gpu则"device_type":“type-PCI”
4:配置带pci标签的flavor
openstack flavor set ml.large --property "pci_passthrough:alias"="a1:1"
可以使用如下命令创建
openstack flavor create --ram 2048 --disk 20 --vcpus 2 m1.large
openstack flavor set m1.large --property pci_passthrough:alias='a1:1,a2:1,a3:1,a4:1'
使⽤该flavor创建虚拟机, 虚拟机会⾃动调度到透传设备的节点上
openstack flavor set FLAVOR-NAME --property pci_passthrough:alias=ALIAS:COUNT
参考官网链接:
https://docs.openstack.org/nova/pike/admin/pci-passthrough.html
配置vfio驱动,如下:
配置系统加载模块
echo<<EOF>>/etc/modules-load.d/openstack-gpu.conf
vfio_pci
pci_stub
vfio
vfio_iommu_type1
kvm
kvm_intel
###############################
EOF
配置vfio加载的设备
配置使用vfio驱动的设备(这里的设备就是上面我们查到的设备的)
cat<<EOF>>/etc/modprobe.d/vfio.conf
options vfio-pci ids=23e2:0100
EOF
重启服务器查看vfio是否被加载
reboot
[root@compute01 ~]# dmesg|grep vfio
[ 28.537454] vfio_pci: add [23e2:0100[ffff:ffff]] class 0x000000/00000000
[ 180.486012] vfio-pci 0000:31:00.0: enabling device (0140 -> 0142)
[ 181.490071] vfio_ecap_init: 0000:31:00.0 hiding ecap 0x19@0x300
[ 181.490176] vfio_ecap_init: 0000:31:00.0 hiding ecap 0x1b@0x440
[ 181.490271] vfio_ecap_init: 0000:31:00.0 hiding ecap 0x1e@0x900
然后在去创建VM就可以了。