硬件要求：
IOMMU kernel 配置, 确保以下配置项已启用：
Device Drivers --->
[*] IOMMU Hardware Support --->
Generic IOMMU Pagetable Support ----
[*] AMD IOMMU support
<*> AMD IOMMU Version 2 driver
[*] Support for Intel IOMMU using DMA Remapping Devices
[*] Support for Shared Virtual Memory with Intel IOMMU
[*] Enable Intel DMA Remapping Devices by default
[*] Support for Interrupt Remapping
IOMMU kernel 启动参数配置：
$ sudo uname -a
Linux fc34 5.13.14-200.fc34.x86_64 #1 SMP Fri Sep 3 15:33:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ sudo grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on default_hugepagesz=1G"
reboot 后确认 IOMMU 已启用:
$ dmesg |grep -i -e DMAR -e IOMMU
[ 0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.13.14-200.fc34.x86_64 root=/dev/mapper/fedora_fedora-root ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off
[ 0.013021] ACPI: DMAR 0x000000007EF6A000 0000D8 (v01 INSYDE Romley 00000001 ACPI 00000001)
[ 0.013053] ACPI: Reserving DMAR table memory at [mem 0x7ef6a000-0x7ef6a0d7]
[ 0.257029] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.13.14-200.fc34.x86_64 root=/dev/mapper/fedora_fedora-root ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off
[ 0.257134] DMAR: IOMMU enabled
[ 0.469532] DMAR: Host address width 46
[ 0.469637] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[ 0.469750] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
[ 0.469886] DMAR: DRHD base: 0x000000ebffc000 flags: 0x1
[ 0.469997] DMAR: dmar1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
[ 0.470132] DMAR: RMRR base: 0x0000007ea8c000 end: 0x0000007ea94fff
[ 0.470246] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffe000 IOMMU 0
[ 0.470357] DMAR-IR: IOAPIC id 8 under DRHD base 0xebffc000 IOMMU 1
[ 0.470481] DMAR-IR: IOAPIC id 9 under DRHD base 0xebffc000 IOMMU 1
[ 0.470592] DMAR-IR: HPET id 0 under DRHD base 0xebffc000
[ 0.470700] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.471512] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.374638] iommu: Default domain type: Passthrough (set via kernel command line)
[ 1.475745] DMAR: No ATSR found
[ 1.475855] DMAR: No SATC found
[ 1.475862] DMAR: dmar0: Using Queued invalidation
[ 1.476206] DMAR: dmar1: Using Queued invalidation
[ 1.476483] pci 0000:00:00.0: Adding to iommu group 0
[ 1.476660] pci 0000:00:01.0: Adding to iommu group 1
确认 kernel 配置中 VFIO 选项都已启用:
Device Drivers --->
<M> VFIO Non-Privileged userpsace driver framework --->
[*] VFIO No-IOMMU support ----
<M> VFIO support for PCI devices
[*] VFIO PCI support for VGA devices
< > Mediated device driver framework
注：
fedora 默认不开启 VFIO No-IOMMU support.
配置 hugepages，安装 dpdk 并绑定 i350 网卡到 vfio-pci 驱动:
$ sudo dnf install dpdk dpdk-tools -y
dpdk-20.11-1.fc34.x86_64
dpdk-tools-20.11-1.fc34.x86_64
$ sudo echo 16 > /proc/sys/vm/nr_hugepages
$ sudo modprobe vfio-pci
$ sudo dpdk-hugepages.py -s
Node Pages Size Total
0 8 1Gb 8Gb
1 8 1Gb 8Gb
Hugepages mounted on /dev/hugepages
$ sudo dpdk-devbind.py -b vfio-pci 02:00.3
$ sudo dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:02:00.3 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=igb
Network devices using kernel driver
===================================
0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=vfio-pci *Active*
0000:05:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp5s0f0 drv=ixgbe unused=vfio-pci
0000:05:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp5s0f1 drv=ixgbe unused=vfio-pci
调整 CPU 模式为 performance 模式：
$ sudo cpupower frequency-set -g performance
由主机PCI 网卡直接分配（PCI Passthrough）:
-device vfio-pci,host=02:00.3
使用主机 hugepages:
-m 8G -object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \ -mem-prealloc -numa node,memdev=mem
$ sudo qemu-kvm -version
QEMU emulator version 3.0.1 (qemu-3.0.1-4.fc29)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
$ sudo qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split \
-nographic --enable-kvm \
-rtc base=localtime,clock=host \
-m 8G -object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-mem-prealloc -numa node,memdev=mem -overcommit mem-lock=on \
-smp 8 -cpu host \
-device intel-iommu,intremap=on,device-iotlb=on \
-device ioh3420,id=pcie.1,chassis=1 -no-hpet \
-netdev tap,id=net0,vhostforce,queues=4 \
-device virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on,vectors=10 \
-device vfio-pci,host=02:00.3 \
centos8-disk.qcow2
kernel 启动参数：
$ sudo grubby --update-kernel=ALL --args="default_hugepagesz=1G"
dpdk & hugepages & vfio 配置：
$ sudo echo 2 > /proc/sys/vm/nr_hugepages
$ sudo modprobe vfio enable_unsafe_noiommu_mode=1
$ sudo modprobe vfio-pci
$ sudo dpdk-devbind -b vfio-pci 00:03.0
$ sudo dpdk-hugepages.py -s
Node Pages Size Total
0 2 1Gb 2Gb
Hugepages mounted on /dev/hugepages
$ sudo dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:00:03.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=igb
Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
$ sudo cpupower frequency-set -g performance
主机 testpmd txonly 发包
$ sudo dpdk-testpmd -c 0xfc -m 4096 -a 02:00.3 -n 2 \
-- -i --rxd=1024 --txd=1024 \
--rxq=4 --txq=4 --nb-cores=4 \
--forward-mode=txonly \
--txpkts=64 \
--mlockall -a
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0
RX-errors: 0
RX-nombuf: 0
TX-packets: 25246875 TX-errors: 0 TX-bytes: 1615800204
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 1420592 Tx-bps: 727342616
############################################################################
testpmd> quit
虚拟机 testpmd rxonly 收包:
dpdk-testpmd -c 0xfc -m 2048 -a 00:03.0 -- -i -a \
--rxq=1 --txq=1 --nb-cores=4 \
--rxd=2048 --txd=2048 --enable-rx-cksum --mlockall \
--forward-mode=rxonly
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 327892831 RX-missed: 14493499 RX-bytes: 20970135168
RX-errors: 0
RX-nombuf: 0
TX-packets: 1 TX-errors: 0 TX-bytes: 60
Throughput (since last show)
Rx-pps: 1420609 Rx-bps: 727351720
Tx-pps: 0 Tx-bps: 0
############################################################################
虚拟机 testpmd txonly 发包:
$ sudo dpdk-testpmd -c 0xfc -m 2048 -a 00:03.0 -- -i -a \
--rxq=1 --txq=1 --nb-cores=4 \
--rxd=2048 --txd=2048 --enable-rx-cksum --mlockall \
--forward-mode=txonly --txpkts=64
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0
RX-errors: 0
RX-nombuf: 0
TX-packets: 5665824 TX-errors: 0 TX-bytes: 362613416
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 1420607 Tx-bps: 727350144
############################################################################
testpmd>
主机 testpmd rxonly 收包：
$ sudo dpdk-testpmd -c 0xfc -m 4096 -a 02:00.3 -n 2 \
-- -i --rxd=2048 --txd=2048 \
--rxq=4 --txq=4 --nb-cores=4 \
--forward-mode=rxonly --mlockall -a
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 34979352 RX-missed: 0 RX-bytes: 2238678868
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 1420598 Rx-bps: 727346200
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>
最终测试结果RX/TX都可达到千兆网络 64 字节小包线速.
CPU:
$ sudo lscpu
[root@centos-guest ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 1
Vendor ID: GenuineIntel
BIOS Vendor ID: QEMU
CPU family: 6
Model: 45
Model name: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
BIOS Model name: pc-q35-6.1
Stepping: 7
CPU MHz: 2599.998
BogoMIPS: 5199.99
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc archs
[root@centos-guest ~]#
guest rxonly:
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0
RX-errors: 0
RX-nombuf: 0
TX-packets: 239624814 TX-errors: 0 TX-bytes: 14377489160
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 1488316 Tx-bps: 714392640
############################################################################
testpmd>
guest txonly:
testpmd> show port stats all
######################## NIC statistics for port 0 ########################
RX-packets: 125837569 RX-missed: 0 RX-bytes: 7550254396
RX-errors: 1
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 1488207 Rx-bps: 714339808
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd>