MoonGen 在CentOS 7.4平台 内网编译支持 X722 网卡

|

外网,下载文件

先在外网机器, 把源码下载下来:

1
2
3
4
5
6
7
8
chunli@ubuntu:~$ git clone https://github.com/emmericp/MoonGen.git
Cloning into 'MoonGen'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 11846 (delta 0), reused 0 (delta 0), pack-reused 11845
Receiving objects: 100% (11846/11846), 20.62 MiB | 1.44 MiB/s, done.
Resolving deltas: 100% (7318/7318), done.
chunli@ubuntu:~$

接着把 项目的 git 依赖子模块 libmoon 下载下来

1
2
3
4
5
6
7
chunli@ubuntu:~$ cd MoonGen/
chunli@ubuntu:~/MoonGen$ cat build.sh
chunli@ubuntu:~/MoonGen$ git submodule update --init
Submodule 'libmoon' (https://github.com/libmoon/libmoon) registered for path 'libmoon'
Cloning into '/home/chunli/MoonGen/libmoon'...
Submodule path 'libmoon': checked out 'dfceafcbde59a6d50a793160382571ce45561b4d'
chunli@ubuntu:~/MoonGen$

libmoon 模块下载好后, 此模块会有新的依赖模块.
把新的依赖模块 一一下载下来

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
chunli@ubuntu:~/MoonGen$ git submodule update --init --recursive
Submodule 'deps/dpdk' (https://github.com/emmericp/dpdk) registered for path 'libmoon/deps/dpdk'
Submodule 'deps/highwayhash' (https://github.com/google/highwayhash.git) registered for path 'libmoon/deps/highwayhash'
Submodule 'deps/luajit' (https://github.com/emmericp/LuaJIT) registered for path 'libmoon/deps/luajit'
Submodule 'deps/pciids' (https://github.com/emmericp/pciids) registered for path 'libmoon/deps/pciids'
Submodule 'deps/tbb' (https://github.com/01org/tbb.git) registered for path 'libmoon/deps/tbb'
Submodule 'lua/lib/pf' (https://github.com/emmericp/pflua) registered for path 'libmoon/lua/lib/pf'
Submodule 'lua/lib/syscall' (https://github.com/emmericp/ljsyscall) registered for path 'libmoon/lua/lib/syscall'
Submodule 'lua/lib/turbo' (https://github.com/emmericp/turbo) registered for path 'libmoon/lua/lib/turbo'
Cloning into '/home/chunli/MoonGen/libmoon/deps/dpdk'...
Cloning into '/home/chunli/MoonGen/libmoon/deps/highwayhash'...
Cloning into '/home/chunli/MoonGen/libmoon/deps/luajit'...
Cloning into '/home/chunli/MoonGen/libmoon/deps/pciids'...
Cloning into '/home/chunli/MoonGen/libmoon/deps/tbb'...
Cloning into '/home/chunli/MoonGen/libmoon/lua/lib/pf'...
Cloning into '/home/chunli/MoonGen/libmoon/lua/lib/syscall'...
Cloning into '/home/chunli/MoonGen/libmoon/lua/lib/turbo'...
Submodule path 'libmoon/deps/dpdk': checked out '6d6c77b373757ec98010b4bf4fcb671423fa3ce2'
Submodule path 'libmoon/deps/highwayhash': checked out '14dedecd1de87cb662f7a882ea1578d2384feb2f'
Submodule path 'libmoon/deps/luajit': checked out '8271c643c21d1b2f344e339f559f2de6f3663191'
Submodule path 'libmoon/deps/pciids': checked out 'eea0f7199045fbaf3800a499e867ef1a4e5d1f9e'
Submodule path 'libmoon/deps/tbb': checked out 'd7e150913467f3c168ba6319c985cf341fd81012'
Submodule path 'libmoon/lua/lib/pf': checked out 'b16b31aed678a3b8ce809391c9e6bcd67f70be0e'
Submodule path 'libmoon/lua/lib/syscall': checked out '2a18b1b62947c597b4e573bb042483ff18434b6f'
Submodule path 'libmoon/lua/lib/turbo': checked out 'd69c85a255a6ceed329813f04e40bd2a9cefcf5e'
chunli@ubuntu:~/MoonGen$

看下版本分支

1
2
3
4
5
6
7
8
9
chunli@ubuntu:~/MoonGen$ git branch -avv
* master                    525d991 [origin/master] Merge pull request #283 from liweitianux/patch-1
  remotes/origin/HEAD       -> origin/master
  remotes/origin/dpdk-19.05 a51cdf1 fix include for dpdk 19.05
  remotes/origin/lacp       2ea978a implement lacp
  remotes/origin/master     525d991 Merge pull request #283 from liweitianux/patch-1
  remotes/origin/mooncli    2599af9 readme: fix links
  remotes/origin/phobos     d6e6ffa update build scripts
chunli@ubuntu:~/MoonGen$

看下日期

1
2
3
4
5
6
7
8
9
10
chunli@ubuntu:~/MoonGen$ git log  -n 1
commit 525d9917c98a4760db72bb733cf6ad30550d6669 (HEAD -> master, origin/master, origin/HEAD)
Merge: b11da03 c4af85a
Author: Paul Emmerich <emmericp@net.in.tum.de>
Date:   Sun Aug 9 13:15:04 2020 +0200

    Merge pull request #283 from liweitianux/patch-1

    interface/README.md: Fix a link markup
chunli@ubuntu:~/MoonGen$

压缩打包

1
2
3
4
5
6
chunli@ubuntu:~/MoonGen$ cd ..
chunli@ubuntu:~$ tar -zcf MoonGen_submodule_update.tar.gz MoonGen

chunli@ubuntu:~$ ll -h
-rw-rw-r--  1 chunli chunli 174M Apr  1 15:53 MoonGen_submodule_update.tar.gz
chunli@ubuntu:~$

得到: MoonGen_submodule_update.tar.gz
把: MoonGen_submodule_update.tar.gz 发到内网 CentOS 7.4机器上.

二, 内网编译

环境说明:

MoonGen 对 编译器, 汇编器, 链接器 要求较高,
CentOS 7.4 默认 gcc 4.8.5 版本太低

我的环境 X722 网卡

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@localhost MoonGen]# lshw -c network -businfo
Bus info          Device     Class          Description
=======================================================
pci@0000:03:00.0  enp3s0f0   network        I350 Gigabit Network Connection
pci@0000:03:00.1  enp3s0f1   network        I350 Gigabit Network Connection
pci@0000:03:00.2  enp3s0f2   network        I350 Gigabit Network Connection
pci@0000:03:00.3  enp3s0f3   network        I350 Gigabit Network Connection
pci@0000:1a:00.0  enp26s0f0  network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:1a:00.1  enp26s0f1  network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:1a:00.2  enp26s0f2  network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:1a:00.3  enp26s0f3  network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:3c:00.0  enp60s0    network        Ethernet Controller XL710 for 40GbE backplane
pci@0000:86:00.0  enp134s0   network        Ethernet Controller XL710 for 40GbE backplane
[root@localhost MoonGen]#

环境参考:

分别升级GCC 与 ld

GCC 升级方法1

GCC 版本  我编译升级的是 7.5, 可参考以下链接

CentOS7 升级GCC 7.5

binutils  我编译升级的是 2.36

GCC 升级方法2

同时升级 GCC 与 ld
yum install devtoolset-9

scl enable devtoolset-9 bash

解压:

1
2
3
4
[root@localhost source]# ll MoonGen_submodule_update.tar.gz
-rw-r--r-- 1 root root 181422070 Apr  1 16:10 MoonGen_submodule_update.tar.gz
[root@localhost source]#
[root@localhost source]# tar xf MoonGen_submodule_update.tar.gz

关闭C++11 编译特性

1
2
3
[root@localhost source]# vim MoonGen/CMakeLists.txt

set(CMAKE_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0 ...

修改 驱动代码(可选操作)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@localhost source]# vim MoonGen/libmoon/deps/dpdk/drivers/net/i40e/i40e_ethdev.c


添加两行代码, 以支持X722

[root@localhost i40e]# git diff
diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 5f26e24..e3e02b2 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -1126,6 +1126,10 @@ static inline void i40e_GLQF_reg_init(struct i40e_hw *hw)
                PMD_INIT_LOG(ERR, "Failed to init adminq: %d", ret);
                return -EIO;
        }
+
+    if(hw->device_id == I40E_DEV_ID_SFP_X722)
+        hw->flags &= ~I40E_HW_FLAG_802_1AD_CAPABLE;
+
        PMD_INIT_LOG(INFO, "FW %d.%d API %d.%d NVM %02d.%02d.%02d eetrack %04x",
                     hw->aq.fw_maj_ver, hw->aq.fw_min_ver,
                     hw->aq.api_maj_ver, hw->aq.api_min_ver,
[root@localhost i40e]#

使用 手动编译

1
2
[root@localhost source]# cd  MoonGen/libmoon/deps/dpdk/
[root@localhost dpdk]# make config T=x86_64-native-linuxapp-gcc && make install T=x86_64-native-linuxapp-gcc

验证编译结果

1
2
3
4
5
6
7
8
9
10
11
12
[root@localhost dpdk]# ll x86_64-native-linuxapp-gcc/app/
total 26764
-rwxr-xr-x 1 root root 3999352 Apr  1 16:26 dpdk-pdump
-rw-r--r-- 1 root root 2816123 Apr  1 16:26 dpdk-pdump.map
-rwxr-xr-x 1 root root   30992 Apr  1 16:24 dpdk-pmdinfogen
-rwxr-xr-x 1 root root 3977552 Apr  1 16:26 dpdk-procinfo
-rw-r--r-- 1 root root 2811419 Apr  1 16:26 dpdk-procinfo.map
-rwxr-xr-x 1 root root 4062296 Apr  1 16:26 dpdk-test-crypto-perf
-rw-r--r-- 1 root root 2826128 Apr  1 16:26 dpdk-test-crypto-perf.map
-rwxr-xr-x 1 root root 4027960 Apr  1 16:26 dpdk-test-eventdev
-rw-r--r-- 1 root root 2837602 Apr  1 16:26 dpdk-test-eventdev.map
[root@localhost dpdk]#

修改MoonGen编译脚本对DPDK的控制

1
2
3
4
[root@localhost dpdk]# cd ../../
[root@localhost libmoon]# pwd
/root/source/MoonGen/libmoon
[root@localhost libmoon]# vim build.sh

修改区域, 删除dpdk编译的相关代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@localhost libmoon]# git diff build.sh
diff --git a/build.sh b/build.sh
index c04c2fd..a67d9e2 100755
--- a/build.sh
+++ b/build.sh
@@ -55,18 +55,6 @@ make -j $NUM_CPUS BUILDMODE=static 'CFLAGS=-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LU
 make install DESTDIR=$(pwd)
 )

-(
-cd deps/dpdk
-#build DPDK with the right configuration
-make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
-if ${MLX5} ; then
-       sed -ri 's,(MLX5_PMD=).*,\1y,' x86_64-native-linuxapp-gcc/.config
-fi
-if ${MLX4} ; then
-       sed -ri 's,(MLX4_PMD=).*,\1y,' x86_64-native-linuxapp-gcc/.config
-fi
-EXTRA_CFLAGS="-Wno-error" make -j $NUM_CPUS O=x86_64-native-linuxapp-gcc
-)

 (
 cd lua/lib/turbo
[root@localhost libmoon]#

编译MoonGen

1
2
3
4
[root@localhost libmoon]# cd ..
[root@localhost MoonGen]# pwd
/root/source/MoonGen
[root@localhost MoonGen]# ./build.sh

编译结束后, 会提示绑定网卡的提示.

1
2
3
4
5
6
7
8
9
[root@localhost MoonGen]# ll build
total 10392
-rw-r--r-- 1 root root    13781 Apr  1 16:29 CMakeCache.txt
drwxr-xr-x 5 root root      234 Apr  1 16:29 CMakeFiles
-rw-r--r-- 1 root root     1775 Apr  1 16:29 cmake_install.cmake
drwxr-xr-x 4 root root      107 Apr  1 16:29 libmoon
-rw-r--r-- 1 root root    10736 Apr  1 16:29 Makefile
-rwxr-xr-x 1 root root 10605776 Apr  1 16:29 MoonGen
[root@localhost MoonGen]#

运行

绑定网卡

1
[root@localhost MoonGen]# cd  libmoon/deps/dpdk/usertools/

查看CPU 分布

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@localhost usertools]# ./cpu_layout.py
======================================================================
Core and Socket Information (as reported by '/sys/devices/system/cpu')
======================================================================
cores =  [0, 1, 2, 3, 4, 8, 9, 10, 11, 12]
sockets =  [0, 1]

        Socket 0        Socket 1
        --------        --------
Core 0  [0, 20]         [10, 30]
Core 1  [1, 21]         [11, 31]
Core 2  [2, 22]         [12, 32]
Core 3  [3, 23]         [13, 33]
Core 4  [4, 24]         [14, 34]
Core 8  [5, 25]         [15, 35]
Core 9  [6, 26]         [16, 36]
Core 10 [7, 27]         [17, 37]
Core 11 [8, 28]         [18, 38]
Core 12 [9, 29]         [19, 39]
[root@localhost usertools]#

安装 dpdk 网卡内核模块

1
2
3
4
5
6
7
8
9
10
11
12
[root@localhost dpdk]# cd x86_64-native-linuxapp-gcc/
[root@localhost x86_64-native-linuxapp-gcc]# cd kmod
[root@localhost kmod]# ll
total 984
-rw-r--r-- 1 root root 260360 Feb  3 14:10 igb_uio.ko
-rw-r--r-- 1 root root 741656 Feb  3 14:10 rte_kni.ko
[root@localhost kmod]# lsmod | grep igb_uio
[root@localhost kmod]# insmod ./igb_uio.ko
[root@localhost kmod]# lsmod | grep igb_uio
igb_uio                13307  0
uio                    19259  1 igb_uio
[root@localhost kmod]#

绑定网卡

1
2
[root@localhost usertools]# ./dpdk-devbind.py -b igb_uio 0000:1a:00.0
[root@localhost usertools]# ./dpdk-devbind.py -b igb_uio 0000:1a:00.1

查看绑定状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@localhost usertools]# ./dpdk-devbind.py  --status-dev net
Network devices using DPDK-compatible driver
============================================
0000:1a:00.0 'Ethernet Connection X722 for 10GbE SFP+ 37d0' drv=igb_uio unused=
0000:1a:00.1 'Ethernet Connection X722 for 10GbE SFP+ 37d0' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:03:00.0 'I350 Gigabit Network Connection 1521' if=enp3s0f0 drv=igb unused=igb_uio *Active*
0000:03:00.1 'I350 Gigabit Network Connection 1521' if=enp3s0f1 drv=igb unused=igb_uio
0000:03:00.2 'I350 Gigabit Network Connection 1521' if=enp3s0f2 drv=igb unused=igb_uio
0000:03:00.3 'I350 Gigabit Network Connection 1521' if=enp3s0f3 drv=igb unused=igb_uio
0000:1a:00.2 'Ethernet Connection X722 for 10GbE SFP+ 37d0' if=enp26s0f2 drv=i40e unused=igb_uio
0000:1a:00.3 'Ethernet Connection X722 for 10GbE SFP+ 37d0' if=enp26s0f3 drv=i40e unused=igb_uio
0000:3c:00.0 'Ethernet Controller XL710 for 40GbE backplane 1580' if=enp60s0 drv=i40e unused=igb_uio
0000:86:00.0 'Ethernet Controller XL710 for 40GbE backplane 1580' if=enp134s0 drv=i40e unused=igb_uio

Other Network devices
=====================
<none>
[root@localhost usertools]#

测试启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
[root@localhost MoonGen]# ./build/MoonGen  examples/l3-load-latency.lua  0 0
[INFO]  Initializing DPDK. This will take a few seconds...
EAL: Detected 40 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:03:00.2 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:03:00.3 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:1a:00.0 on NUMA socket 0
EAL:   probe driver: 8086:37d0 net_i40e
EAL: PCI device 0000:1a:00.1 on NUMA socket 0
EAL:   probe driver: 8086:37d0 net_i40e
EAL: PCI device 0000:1a:00.2 on NUMA socket 0
EAL:   probe driver: 8086:37d0 net_i40e
EAL: PCI device 0000:1a:00.3 on NUMA socket 0
EAL:   probe driver: 8086:37d0 net_i40e
EAL: PCI device 0000:3c:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1580 net_i40e
EAL: PCI device 0000:86:00.0 on NUMA socket 1
EAL:   probe driver: 8086:1580 net_i40e
[INFO]  Found 4 usable devices:
   Device 0: E4:95:6E:2F:D9:8F (Intel Corporation Ethernet Connection X722 for 10GbE SFP+)
   Device 1: E4:95:6E:2F:D9:90 (Intel Corporation Ethernet Connection X722 for 10GbE SFP+)
   Device 2: E4:95:6E:2F:D9:91 (Intel Corporation Ethernet Connection X722 for 10GbE SFP+)
   Device 3: E4:95:6E:2F:D9:92 (Intel Corporation Ethernet Connection X722 for 10GbE SFP+)
i40e_enable_pf_lb(): FW < v5.0, cannot enable loopback
[WARN]  Device 0 already configured, skipping initilization
[INFO]  Waiting for devices to come up...
[INFO]  Device 0 (E4:95:6E:2F:D9:8F) is up: 10000 MBit/s
[INFO]  1 device is up.
[WARN]  Per-queue rate limit is not supported on this device, setting per-device rate limit to 9999 Mbit/s instead (note: this may fail as well if the NIC doesn't support any rate limiting).
[INFO]  Performing ARP lookup on 10.1.0.10
[INFO]  Performing ARP lookup on 10.1.0.10
i40e_ethertype_filter_set(): Conflict with existing ethertype rules!
[WARN]  l2 filter error: Invalid argument
[INFO]  ARP lookup failed, using default destination mac address
[INFO]  ARP lookup failed, using default destination mac address
[WARN]  Packet size 60 is smaller than minimum timestamp size 84. Timestamped packets will be larger than load packets.
[Device: id=0] TX: 14.87 Mpps, 7615 Mbit/s (9995 Mbit/s with framing)
[Device: id=0] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing)
[Device: id=0] TX: 8.40 Mpps, 4300 Mbit/s (5644 Mbit/s with framing)
[Device: id=0] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing)
[Device: id=0] TX: 8.57 Mpps, 4390 Mbit/s (5762 Mbit/s with framing)
[Device: id=0] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing)
[Device: id=0] TX: 8.58 Mpps, 4394 Mbit/s (5767 Mbit/s with framing)