DPDK run in KVM Virtio

|

背景

上海疫情期间需要在家办公， 任务基于DPDK环境开发，
家里没有双路CPU的志强E5服务器， 更没有82599，X710多端口的网卡，小区封闭，快递停运，无从获取。

我想到了一个 Good idea!
虚拟机模拟实现.

在DPDK 源代码里面发现有 对Virtio网卡的支持。

在 KVM 虚拟机内虚拟4个 Virtio 互联网卡，以便于开发。

示意

在物理机 创建网桥镜像

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# 创建网桥
brctl addbr virtio1
brctl addbr virtio2
brctl addbr virtio3
brctl addbr virtio4

# 设置镜像
brctl setageing virtio1 0
brctl setageing virtio2 0
brctl setageing virtio3 0
brctl setageing virtio4 0

# 启动
ifconfig virtio1  up
ifconfig virtio2  up
ifconfig virtio3  up
ifconfig virtio4  up

安装开发机环境

先在KVM中安装一个CentOS 7.4， 公司的业务OS选型就是7.4
1个管理网卡
4个virtio网卡

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
root@debian:~/kvm# cat kvm_install_centos7.sh
#!/bin/bash
OS_DOMAIN='centos7'
OS_DISK="/home/OS_Disk_Image/${OS_DOMAIN}.disk"
OS_IMG_DIR='/mnt/data/08_各种镜像/OS_Linux/'

virsh  destroy  $OS_DOMAIN  > /dev/null 2>&1
virsh  undefine $OS_DOMAIN  > /dev/null 2>&1

rm -rf $OS_DISK
rm -rf /var/log/libvirt/qemu/${OS_DOMAIN}.log
rm -rf /etc/libvirt/storage

virt-install                                               \
--virt-type=kvm                                            \
--os-type=linux                                            \
--os-variant=centos7                                       \
--name ${OS_DOMAIN}                                        \
--cpu=host-passthrough                                     \
--accelerate                                               \
--autostart                                                \
--ram 8192                                                 \
--noautoconsole                                            \
--vcpus sockets=1,cores=4,threads=2                        \
--network mac=12:34:56:78:90:A4,bridge=br0,model=e1000     \
--network mac=12:34:56:78:90:A5,bridge=virtio1,model=virtio    \
--network mac=12:34:56:78:90:A6,bridge=virtio2,model=virtio    \
--network mac=12:34:56:78:90:A7,bridge=virtio3,model=virtio    \
--network mac=12:34:56:78:90:A8,bridge=virtio4,model=virtio    \
--graphics vnc,listen=0.0.0.0,port=5924,password=chunli    \
--disk path=${OS_DISK},size=60,format=raw,bus=virtio       \
--cdrom     ${OS_IMG_DIR}/CentOS-7-1708/CentOS-7-x86_64-DVD-1708.iso \

root@debian:~/kvm#

virtio 专用脚本

为了便于使用 查看、巨页，绑定系列脚本

开发机的 virtio 网卡 需要特别的设置

网卡信息脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

eth_info=$(lspci -Dvmmnnk | awk -v RS="\n\n" '/Virtio network device/{print gensub(/:\s([^\n]*)/, cnt++ "=\"\\1\"", "g");} END{print "dev_cnt=" cnt;}')

eval $eth_info

printf "%-14s%-15s%-10s%-16s %s\n" "Slot" "IF" "UP" "Driver" "Desc"

for ((i=0; i<$dev_cnt; i++))
do
 eval "Interface$i=---"
 eval "Linking$i=-"
 eval "dev_net_path=/sys/bus/pci/devices/\$Slot$i/"
 if [ -e $dev_net_path ]
 then
    find $dev_net_path/ | grep -w carrier > /dev/null 2>&1
    if [ 0 -eq $? ]
    then
        virtioX=$(dirname $(dirname $(dirname $(find $dev_net_path -name carrier))))
        eval "Driver$i=$(basename $(readlink $(echo $virtioX/driver)))"
        eval "Interface$i=$(basename $(dirname $(find $dev_net_path -name carrier)))"
        eval "read Linking$i < $(find $dev_net_path -name carrier)"
    fi


 fi
 eval "printf '%-14s%-15s%-10s%-16s %s\n' \$Slot$i \$Interface$i \$Linking$i \$Driver$i \"\$Device$i\""
done

巨页设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
#!/bin/bash


cd $(dirname $0)

# HUGEPAGES params
HUGEPGSZ_IN_KB=`cat /proc/meminfo  | grep Hugepagesize | cut -d : -f 2 | tr -d ' '`
NUMA_HUGE_PAGE_CNT=6

echo HUGEPGSZ_IN_KB $HUGEPGSZ_IN_KB
echo NUMA_HUGE_PAGE_CNT $NUMA_HUGE_PAGE_CNT

function LOG_ERR
{
    echo -e "\033[1;31;40m"$*"\033[0m"
}

function LOG_WAR
{
    echo -e "\033[1;33;40m"$*"\033[0m"
}

function LOG_INFO
{
    echo -e "\033[1;32;40m"$*"\033[0m"
}

function LOG_DEBUG
{
    echo -e "\033[1;30;40m"$*"\033[0m"
}

function step
{
    if [ $? -ne 0 ]
    then
        LOG_ERR "previous action failed, exit."
        exit
    fi

    echo -n "step $1: "

    shift
    eval $*
}

#
# Unloads igb_uio.ko.
#
remove_igb_uio_module()
{
    /sbin/lsmod | grep -s igb_uio > /dev/null
    if [ $? -eq 0 ] ; then
         /sbin/rmmod igb_uio
    fi

    LOG_INFO "Unloading any existing DPDK UIO module"
}

#
# Loads new igb_uio.ko (and uio module if needed).
#
load_igb_uio_module()
{
    modprobe igb_uio
    if [ $? -ne 0 ] ; then
        LOG_ERR "## ERROR: Could not load kmod/igb_uio.ko."
        exit
    fi
    LOG_INFO "Load DPDK UIO module Successful!"
}

#
# Unloads the rte_kni.ko module.
#
remove_kni_module()
{
    /sbin/lsmod | grep -s rte_kni > /dev/null
    if [ $? -eq 0 ] ; then
         /sbin/rmmod rte_kni
    fi

    LOG_INFO "Unloading any existing DPDK KNI module"
}

#
# Loads the rte_kni.ko module.
#
load_kni_module()
{
    modprobe rte_kni
    if [ $? -ne 0 ] ; then
        LOG_ERR "## ERROR: Could not load kmod/rte_kni.ko."
        exit
    fi
    LOG_INFO "Load  DPDK KNI module Successful!"
}

#
# Removes all reserved hugepages.
#
clear_huge_pages()
{
    echo > .echo_tmp
    for d in /sys/devices/system/node/node? ; do
        echo "echo 0 > $d/hugepages/hugepages-${HUGEPGSZ_IN_KB}/nr_hugepages" >> .echo_tmp
    done

    sh .echo_tmp
    rm -f .echo_tmp

    remove_mnt_huge

    LOG_INFO "Remove currently reserved hugepages, Unmounting /mnt/huge and removing directory"
}

#
# Creates hugepage filesystem.
#
create_mnt_huge()
{
    mkdir -p /mnt/huge

    grep -s '/mnt/huge' /proc/mounts > /dev/null
    if [ $? -ne 0 ] ; then
         mount -t hugetlbfs nodev /mnt/huge
    fi
}

#
# Removes hugepage filesystem.
#
remove_mnt_huge()
{
    grep -s '/mnt/huge' /proc/mounts > /dev/null
    if [ $? -eq 0 ] ; then
         umount /mnt/huge
    fi

    if [ -d /mnt/huge ] ; then
         rm -R /mnt/huge
    fi
}

# Creates hugepages on specific NUMA nodes.
#
set_numa_pages()
{
    # clear_huge_pages

    Pages=$1

    echo > .echo_tmp
    for d in /sys/devices/system/node/node? ; do
        node=$(basename $d)
        echo "echo $Pages > $d/hugepages/hugepages-${HUGEPGSZ_IN_KB}/nr_hugepages" >> .echo_tmp
    done

    sh .echo_tmp

    rm -f .echo_tmp

    create_mnt_huge

    LOG_INFO "Reserving hugepages: $Pages, Creating /mnt/huge and mounting as hugetlbfs"
}

#
# show hugepages infos from /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages and
# /proc/meminfo.
#
show_numa_pages()
{
    LOG_INFO "show hugepages config:"

    LOG_WAR "cmdline:"
    echo "$(cat /proc/cmdline)"

    LOG_WAR "hugepages info:"
    for d in /sys/devices/system/node/node? ; do
        pagesPath=$d/hugepages/hugepages-${HUGEPGSZ_IN_KB}/nr_hugepages
        echo $pagesPath "$(cat $pagesPath)"
    done

    LOG_WAR "hugepages in /proc/meminfo:"
    echo "$(grep -ri huge /proc/meminfo)"
}

#
# Uses dpdk-devbind.py to move devices to work with igb_uio
#
bind_devices_to_igb_uio()
{
    if [ -d /sys/module/igb_uio ]; then
        PCI_PATH=$1
        ${RTE_SDK}/usertools/dpdk-devbind.py --force -b igb_uio $PCI_PATH || exit 1
        LOG_INFO "bind $PCI_PATH to igb_uio Successfully!"
    else
        LOG_ERR "# Please load the 'igb_uio' kernel module before querying or "
        LOG_ERR "# adjusting device bindings"
    fi
}

#
# Uses dpdk-devbind.py to move devices to work with kernel drivers again
#
unbind_devices()
{
    DRV=$1
    PCI_PATH=$2
    ${RTE_SDK}/usertools/dpdk-devbind.py -b $DRV $PCI_PATH || LOG_ERR "[Fail] unbind"
    LOG_INFO "unbind $PCI_PATH, bind to driver $DRV Successfully !"
}

get_eth_original_info_by_name()
{
    dst_eth_name=$1
    pci_addr_of_this_eth=""
    driver_of_this_eth=""

    for((i=0; i<$dev_cnt; i++))
    do
        eval "eth_name_i=\$if$i"

        if [ -z $eth_name_i ]
        then
          return 1
        fi

        if [ $eth_name_i == $dst_eth_name ]
        then
            eval "pci_addr_of_this_eth=\$bus$i"
            eval "driver_of_this_eth=\$driver$i"
            return 0
        fi
    done
    return 1
}

#
# ret info at driver_of_this_eth
#
get_eth_original_info_by_pci()
{
    driver_of_this_eth=""

    for((i=0; i<$dev_cnt; i++))
    do
        eval "pci_i=\$bus$i"

        if [ -z $pci_i ]
        then
          return 1
        fi

        if [ $pci_i == $1 ]
        then
            eval "driver_of_this_eth=\$driver$i"
            return 0
        fi
    done
    return 1
}

#
# bind all given eth to dpdk driver.
# zero args will do nothing.
#
function bind_devices_by_name
{
    for eth in $*
    do
        get_eth_original_info_by_name $eth
        if [ $? -ne 0 ]
        then
            echo "can not get pci addr of $eth, bind failed."
            continue
        fi
        echo "bind eth $eth to dpdk driver igb_uio ..."
        bind_devices_to_igb_uio $pci_addr_of_this_eth
    done
}

#
# unbind a eth to it's original driver
#
function unbind_this_device_by_name
{
    get_eth_original_info_by_name $1
    if [ $? -ne 0 ]
    then
        echo "can not get pci addr of $eth, bind failed."
        return 0
    fi

    unbind_devices $driver_of_this_eth $pci_addr_of_this_eth
}

#
# 1) unbind given eth to their original driver.
# 2) if no eth given, unbind all eth binding dpdk driver.
#
function unbind_devices_by_name
{
    # unbind all given eth
    for eth in $*
    do
        echo "unbind dev for $eth"
        unbind_this_device_by_name $eth
    done

    # given at least one eth, bind done and ret
    if [ $# -gt 0 ]
    then
        return 0
    fi

    # given no eth, unbind all dev binding dpdk driver
    for((i=0; i<$dev_cnt; i++))
    do
        eval "Driver_i=\$Driver$i"

        if [ -z $Driver_i ]
        then
          return 1
        fi

        if [ $Driver_i != "igb_uio" ]
        then
            continue
        fi

        eval "get_eth_original_info_by_pci \$Slot$i"
        if [ $? -ne 0 ]
        then
            eval "echo can not get driver of \$Slot$i, unbind failed."
            continue
        fi

        eval "unbind_devices $driver_of_this_eth \$Slot$i"
    done
}

step 1 load_igb_uio_module
step 2 load_kni_module
step 3 set_numa_pages $NUMA_HUGE_PAGE_CNT
step 4 show_numa_pages

巨页清除

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
#!/bin/bash

HUGEPGSZ_IN_KB=1048576kB
function LOG_ERR
{
    echo -e "\033[1;31;40m"$*"\033[0m"
}

function LOG_WAR
{
    echo -e "\033[1;33;40m"$*"\033[0m"
}

function LOG_INFO
{
    echo -e "\033[1;32;40m"$*"\033[0m"
}

function LOG_DEBUG
{
    echo -e "\033[1;30;40m"$*"\033[0m"
}

function step
{
    if [ $? -ne 0 ]
    then
        LOG_ERR "previous action failed, exit."
        exit
    fi

    echo -n "step $1: "

    shift
    eval $*
}

#
# Unloads igb_uio.ko.
#
remove_igb_uio_module()
{
    /sbin/lsmod | grep -s igb_uio > /dev/null
    if [ $? -eq 0 ] ; then
         /sbin/rmmod igb_uio
    fi

    LOG_INFO "Unloading any existing DPDK UIO module"
}

#
# Loads new igb_uio.ko (and uio module if needed).
#
load_igb_uio_module()
{
    modprobe igb_uio
    if [ $? -ne 0 ] ; then
        LOG_ERR "## ERROR: Could not load kmod/igb_uio.ko."
        exit
    fi
    LOG_INFO "Load DPDK UIO module Successful!"
}

#
# Unloads the rte_kni.ko module.
#
remove_kni_module()
{
    /sbin/lsmod | grep -s rte_kni > /dev/null
    if [ $? -eq 0 ] ; then
         /sbin/rmmod rte_kni
    fi

    LOG_INFO "Unloading any existing DPDK KNI module"
}

#
# Loads the rte_kni.ko module.
#
load_kni_module()
{
    # Now try load the KNI module.
    # LOG_DEBUG "Loading DPDK KNI module"
    modprobe rte_kni
    if [ $? -ne 0 ] ; then
        LOG_ERR "## ERROR: Could not load kmod/rte_kni.ko."
        exit
    fi
    LOG_INFO "Load  DPDK KNI module Successful!"
}
#
# Removes all reserved hugepages.
#
clear_huge_pages()
{
    echo > .echo_tmp
    for d in /sys/devices/system/node/node? ; do
        echo "echo 0 > $d/hugepages/hugepages-${HUGEPGSZ_IN_KB}/nr_hugepages" >> .echo_tmp
    done

    sh .echo_tmp
    rm -f .echo_tmp

    remove_mnt_huge

    LOG_INFO "Remove currently reserved hugepages, Unmounting /mnt/huge and removing directory"
}

#
# Creates hugepage filesystem.
#
create_mnt_huge()
{
    mkdir -p /mnt/huge

    grep -s '/mnt/huge' /proc/mounts > /dev/null
    if [ $? -ne 0 ] ; then
         mount -t hugetlbfs nodev /mnt/huge
    fi
}

#
# Removes hugepage filesystem.
#
remove_mnt_huge()
{
    grep -s '/mnt/huge' /proc/mounts > /dev/null
    if [ $? -eq 0 ] ; then
         umount /mnt/huge
    fi

    if [ -d /mnt/huge ] ; then
         rm -R /mnt/huge
    fi
}

# Creates hugepages on specific NUMA nodes.
#
set_numa_pages()
{
    Pages=$1

    echo > .echo_tmp
    for d in /sys/devices/system/node/node? ; do
        node=$(basename $d)
        echo "echo $Pages > $d/hugepages/hugepages-${HUGEPGSZ_IN_KB}/nr_hugepages" >> .echo_tmp
    done

    sh .echo_tmp

    rm -f .echo_tmp

    create_mnt_huge

    LOG_INFO "Reserving hugepages: $Pages, Creating /mnt/huge and mounting as hugetlbfs"
}

#
# Uses dpdk-devbind.py to move devices to work with igb_uio
#
bind_devices_to_igb_uio()
{
    if [ -d /sys/module/igb_uio ]; then
        PCI_PATH=$1
        ${RTE_SDK}/usertools/dpdk-devbind.py --force -b igb_uio $PCI_PATH || exit 1
        LOG_INFO "bind $PCI_PATH to igb_uio Successfully!"
    else
        LOG_ERR "# Please load the 'igb_uio' kernel module before querying or "
        LOG_ERR "# adjusting device bindings"
    fi
}

#
# Uses dpdk-devbind.py to move devices to work with kernel drivers again
#
unbind_devices()
{
    DRV=$1
    PCI_PATH=$2
    ${RTE_SDK}/usertools/dpdk-devbind.py -b $DRV $PCI_PATH || LOG_ERR "[Fail] unbind"
    LOG_INFO "unbind $PCI_PATH, bind to driver $DRV Successfully !"
}

get_all_eth_info()
{
    if [ ! -f ./.if_tbl ]
    then
        LOG_ERR "ERROR: there is no .if_tbl, pleae do ./dpdk.install"
        exit
    fi

    # calc original eth info from .if_tbl
    original_eth_info=$(awk '{print gensub(/=(\S*)/, cnt++ "=\\1;", "g");}' ./.if_tbl)
    eval $original_eth_info

    # get eth dev info from lspci and format the output
    # as shell script's value assignment.
    eth_info=$(lspci -Dvmmnnk | awk -v RS="\n\n" '/Eth/{print gensub(/:\s([^\n]*)/, cnt++ "=\"\\1\"", "g");} END{print "dev_cnt=" cnt;}')

    # eval the shell script to get the variables.
    eval $eth_info
    for ((i=0; i<$dev_cnt; i++))
    do
        # calculate eth interface and linking status
        # if eth1's Slog is 0000:05:00.0
        # it's interface name is /sys/bus/pci/devices/0000:05:00.0/net/[interface]
        # it's linking status in /sys/bus/pci/devices/0000:05:00.0/net/[interface]/carrier
        eval "Interface$i=---"
        eval "Linking$i=-"
        eval "dev_net_path=/sys/bus/pci/devices/\$Slot$i/net"
        if [ -e $dev_net_path ]
        then
            eval "Interface$i=$(ls $dev_net_path)"
            eval "read Linking$i</sys/bus/pci/devices/\$Slot$i/net/\$Interface$i/carrier"
        fi
    done
}

#
# ret info at pci_addr_of_this_eth and driver_of_this_eth
#
get_eth_original_info_by_name()
{
    dst_eth_name=$1
    pci_addr_of_this_eth=""
    driver_of_this_eth=""

    for((i=0; i<$dev_cnt; i++))
    do
        eval "eth_name_i=\$if$i"

        if [ -z $eth_name_i ]
        then
          return 1
        fi

        if [ $eth_name_i == $dst_eth_name ]
        then
            eval "pci_addr_of_this_eth=\$bus$i"
            eval "driver_of_this_eth=\$driver$i"
            return 0
        fi
    done
    return 1
}

#
# ret info at driver_of_this_eth
#
get_eth_original_info_by_pci()
{
    driver_of_this_eth=""

    for((i=0; i<$dev_cnt; i++))
    do
        eval "pci_i=\$bus$i"

        if [ -z $pci_i ]
        then
          return 1
        fi

        if [ $pci_i == $1 ]
        then
            eval "driver_of_this_eth=\$driver$i"
            return 0
        fi
    done
    return 1
}

#
# bind all given eth to dpdk driver.
# zero args will do nothing.
#
function bind_devices_by_name
{
    for eth in $*
    do
        get_eth_original_info_by_name $eth
        if [ $? -ne 0 ]
        then
            echo "can not get pci addr of $eth, bind failed."
            continue
        fi
        echo "bind eth $eth to dpdk driver igb_uio ..."
        bind_devices_to_igb_uio $pci_addr_of_this_eth
    done
}

#
# unbind a eth to it's original driver
#
function unbind_this_device_by_name
{
    get_eth_original_info_by_name $1
    if [ $? -ne 0 ]
    then
        echo "can not get pci addr of $eth, bind failed."
        return 0
    fi

    unbind_devices $driver_of_this_eth $pci_addr_of_this_eth
}

#
# 1) unbind given eth to their original driver.
# 2) if no eth given, unbind all eth binding dpdk driver.
#
function unbind_devices_by_name
{
    # unbind all given eth
    for eth in $*
    do
        echo "unbind dev for $eth"
        unbind_this_device_by_name $eth
    done

    # given at least one eth, bind done and ret
    if [ $# -gt 0 ]
    then
        return 0
    fi

    # given no eth, unbind all dev binding dpdk driver
    for((i=0; i<$dev_cnt; i++))
    do
        eval "Driver_i=\$Driver$i"

        if [ -z $Driver_i ]
        then
          return 1
        fi

        if [ $Driver_i != "igb_uio" ]
        then
            continue
        fi

        eval "get_eth_original_info_by_pci \$Slot$i"
        if [ $? -ne 0 ]
        then
            eval "echo can not get driver of \$Slot$i, unbind failed."
            continue
        fi

        eval "unbind_devices $driver_of_this_eth \$Slot$i"
    done
}


step 1 remove_igb_uio_module
step 2 remove_kni_module
step 3 clear_huge_pages

网卡绑定

1
2
3
4
5
6
7
8
#!/bin/bash

# 示例
# 如下
# 0000:03:00.0  eth0           1         virtio_net
#./usertools/dpdk-devbind.py --force -b virtio_net  0000:03:00.0

./usertools/dpdk-devbind.py --force -b $*

网卡解绑

暂未实现
目前只能绑定， 不能解绑.

运行

查看 virtio 网卡信息

1
2
3
4
5
6
7
[root@localhost dpdk_virtio]# ./ethinfo.sh
Slot          IF             UP        Driver           Desc
0000:03:00.0  eth0           1         virtio_net       Virtio network device [1041]
0000:04:00.0  eth1           1         virtio_net       Virtio network device [1041]
0000:05:00.0  eth2           1         virtio_net       Virtio network device [1041]
0000:06:00.0  eth3           1         virtio_net       Virtio network device [1041]
[root@localhost dpdk_virtio]#

巨页设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@localhost dpdk_virtio]#
[root@localhost dpdk_virtio]# ./huge_mount
HUGEPGSZ_IN_KB 1048576kB
NUMA_HUGE_PAGE_CNT 6
step 1: Load DPDK UIO module Successful!
step 2: Load DPDK KNI module Successful!
step 3: Reserving hugepages: 6, Creating /mnt/huge and mounting as hugetlbfs
step 4: show hugepages config:
cmdline:
BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.el7.x86_64 root=UUID=66371ca9-9e2a-4922-9a94-aed6ed192959 ro crashkernel=auto rhgb quiet console=ttyS0,115200 default_hugepagesz=1G hugepagesz=1G hugepages=6
hugepages info:
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages 5
hugepages in /proc/meminfo:
AnonHugePages:      4096 kB
HugePages_Total:       5
HugePages_Free:        5
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
[root@localhost dpdk_virtio]#

网卡绑定

1
2
3
4
5
6
7
8
9
10
[root@localhost dpdk_virtio]# ./interface_bind igb_uio 0000:03:00.0
[root@localhost dpdk_virtio]# ./interface_bind igb_uio 0000:05:00.0
[root@localhost dpdk_virtio]# ./ethinfo.sh
Slot          IF             UP        Driver           Desc
0000:03:00.0  ---            -         igb_uio          Virtio network device [1041]
0000:04:00.0  eth1           1         virtio_net       Virtio network device [1041]
0000:05:00.0  ---            -         igb_uio          Virtio network device [1041]
0000:06:00.0  eth3           1         virtio_net       Virtio network device [1041]
[root@localhost dpdk_virtio]#
[root@localhost dpdk_virtio]#

运行 DPDK 程序

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
[root@localhost ~]# ./l2fwd -l 0-4 -- -p 0x0f
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:02:01.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:06:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1041 net_virtio
MAC updating enabled
Lcore 0: RX port 0
Lcore 1: RX port 1
Initializing port 0... done:
Port 0, MAC address: 12:34:56:78:90:A5

Initializing port 1... done:
Port 1, MAC address: 12:34:56:78:90:A7

Checking link statusdone
Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 1 Link Up - speed 10000 Mbps - full-duplex
L2FWD: entering main loop on lcore 1
L2FWD:  -- lcoreid=1 portid=1
L2FWD: lcore 2 has nothing to do
L2FWD: lcore 4 has nothing to do
L2FWD: lcore 3 has nothing to do
L2FWD: entering main loop on lcore 0
L2FWD:  -- lcoreid=0 portid=0

Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent:                  1101894
Packets received:               907488
Speed PPS sent:                1101894
Speed PPS received:             907488
Packets dropped:                   853
Statistics for port 1 ------------------------------
Packets sent:                   906641
Packets received:              1102747
Speed PPS sent:                 906641
Speed PPS received:            1102747
Packets dropped:                   847
Statistics for port 2 ------------------------------
Packets sent:                        0
Packets received:                    0
Speed PPS sent:                      0
Speed PPS received:                  0
Packets dropped:                     0
Statistics for port 3 ------------------------------
Packets sent:                        0
Packets received:                    0
Speed PPS sent:                      0
Speed PPS received:                  0
Packets dropped:                     0
Aggregate statistics ===============================
Total packets sent:            2008535
Total packets received:        2010235
Total packets dropped:            1700
====================================================
^C
Signal 2 received, preparing to exit...
Closing port 0... Done
Closing port 1... Done
Bye...
[root@localhost ~]#

创建发包机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
root@debian:~# cat kvm/kvm_install_slave.sh
#!/bin/bash
OS_DOMAIN='slave'
OS_DISK="/home/OS_Disk_Image/${OS_DOMAIN}.disk"
OS_IMG_DIR='/mnt/data/08_各种镜像/OS_Linux/'

virsh  destroy  $OS_DOMAIN  > /dev/null 2>&1
virsh  undefine $OS_DOMAIN  > /dev/null 2>&1

rm -rf $OS_DISK
rm -rf /var/log/libvirt/qemu/${OS_DOMAIN}.log
rm -rf /etc/libvirt/storage

virt-install                                                \
--virt-type=kvm                                             \
--os-type=linux                                             \
--os-variant=centos7                                        \
--name ${OS_DOMAIN}                                         \
--cpu=host-passthrough                                      \
--accelerate                                                \
--autostart                                                 \
--ram 1024                                                  \
--noautoconsole                                             \
--vcpus sockets=1,cores=1,threads=1                         \
--network mac=12:34:56:78:90:B4,bridge=br0,model=e1000      \
--network mac=12:34:56:78:90:B5,bridge=virtio1,model=virtio \
--network mac=12:34:56:78:90:B6,bridge=virtio2,model=virtio \
--network mac=12:34:56:78:90:B7,bridge=virtio3,model=virtio \
--network mac=12:34:56:78:90:B8,bridge=virtio4,model=virtio \
--graphics vnc,listen=0.0.0.0,port=5934,password=chunli     \
--disk path=${OS_DISK},size=40,format=raw,bus=virtio        \
--cdrom     ${OS_IMG_DIR}/CentOS-7-1708/CentOS-7-x86_64-DVD-1708.iso \

安装发包工具

关闭巨帧

1
2
3
4
5
6
7
[root@slave ~]# yum install ethtool

关闭巨帧
[root@slave ~]# ethtool -K eno1  gso off gro off tso off  lro off

验证 还有没有 巨帧
[root@slave ~]# tcpdump -i br0  greater 1500  -tnn

安装发包

tcpreplay-4.2.5-1.el7.x86_64.rpm CentOS7 安装包下载
tcpreplay-4.2.5-1.el7.x86_64.rpm 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@slave ~]# yum install ./tcpreplay-4.2.5-1.el7.x86_64.rpm

开发发包
[root@slave ~]# tcpreplay -i eth1  -M 200  packets_1.pcap
Actual: 137768 packets (110568244 bytes) sent in 4.42 seconds
Rated: 24999727.3 Bps, 199.99 Mbps, 31149.65 pps
Flows: 2204 flows, 498.32 fps, 137329 flow packets, 439 non-flow
Statistics for network device: eth1
    Successful packets:        137768
    Failed packets:            0
    Truncated packets:         0
    Retried packets (ENOBUFS): 0
    Retried packets (EAGAIN):  0
[root@slave ~]#

结束

到此， 端口互联， 开发机配置， 发包机配置，就全部配置好了。

可以安心的工作了。