https://zskjohn.blog.csdn.net/article/details/128846360 云计算|OpenStack|社区版OpenStack安装部署文档(六 --- 网络服务neutron的安装部署---Rocky版)
(######注:以上文章使用的是openstack的provider网络,也称之为提供者网络,对此网络的扩展selfservice,称之为自服务网络,这个文章没有涉及)
前面写的neutron部署安装,感觉很多地方还是没有写到,虽然网络很复杂,尤其是openstack的网络更为复杂,但一些目前我所了解的东西我想应该还是记录一下,备忘吧。
首先,前文部署的是Provider networks + Linux Bridge 方案, 此方案也就是官方文档的网络一,官方的方案还有第二个,是selfservice networks+Linuxbridge方案。
provider是一个半虚拟化的二层网络架构,只能经过桥接的方式实现,处于provider网络模式下vm获取到的ip地址与物理网络在同一网段,能够当作是物理网络的扩展,在该模式下,控制节点不须要安装L3 agent,也不须要网络节点,vm直接经过宿主机的NIC与物理网络通讯,provider网络只支持flat和vlan两种模式。
self-service模式容许租户本身建立网络,最终租户建立的网络借助provider网络以NAT方式访问外网,因此self-service模式能够当作是网络层级的延伸,要实现self-service模式必须先建立provider网络,self-service网络支持flat、vlan、vxlan、gre模式。
其中网络选项2:自助网络(Self-service networks)能够实现更加高级的网络功能,能够实现网络选项1的所有功能,
Neutron最为核心的工作是对二层物理网络的抽象与管理,物理服务器虚拟化后,虚拟机的网络功能由虚拟机网卡(vnic)提供,物理交换机也被虚拟化为虚拟交换机(vswitch),各个vnic连接再vswitch的端口上,最后这些vswitch通过物理服务器的物理网卡访问外部的物理网络。
OK,这里可能比较难以理解,但结合实际就很清楚了,以下是我使用VM虚拟机安装的openstack集群的controller节点的网络情况,
VM虚拟机的网卡ens33被虚拟化了,原来的IP 192.168.123.130落到了brq这个虚拟交换机上了:
[root@openstack1 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens33: mtu 1500 qdisc pfifo_fast master brq688a0356-4f state UNKNOWN qlen 1000link/ether 00:0c:29:de:be:e1 brd ff:ff:ff:ff:ff:ffinet6 fe80::20c:29ff:fede:bee1/64 scope link valid_lft forever preferred_lft forever
3: virbr0: mtu 1500 qdisc noqueue state DOWN qlen 1000link/ether 52:54:00:e8:93:4e brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000link/ether 52:54:00:e8:93:4e brd ff:ff:ff:ff:ff:ff
6: tap35bde514-d6@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000link/ether 16:23:e2:df:f1:59 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: tapdc2df59d-2b@if2: mtu 1500 qdisc noqueue master brq688a0356-4f state UP qlen 1000link/ether 9a:c6:e7:43:dd:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 1
8: tap29f235e4-7b@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000link/ether 76:81:a0:0a:0d:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 2
9: tapf4264f35-18@if3: mtu 1500 qdisc noqueue master brq688a0356-4f state UP qlen 1000link/ether 8e:f5:27:74:a3:53 brd ff:ff:ff:ff:ff:ff link-netnsid 2
10: brq688a0356-4f: mtu 1500 qdisc noqueue state UP qlen 1000link/ether 00:0c:29:de:be:e1 brd ff:ff:ff:ff:ff:ffinet 192.168.123.130/24 brd 192.168.123.255 scope global brq688a0356-4fvalid_lft forever preferred_lft foreverinet6 fe80::1843:29ff:fefe:1d7e/64 scope link valid_lft forever preferred_lft forever
11: vxlan-17: mtu 1450 qdisc noqueue master brq6ef09648-5c state UNKNOWN qlen 1000link/ether be:e1:5a:3d:ce:93 brd ff:ff:ff:ff:ff:ff
12: brq6ef09648-5c: mtu 1450 qdisc noqueue state UP qlen 1000link/ether 16:23:e2:df:f1:59 brd ff:ff:ff:ff:ff:ffinet6 fe80::dc29:e8ff:fe9c:d984/64 scope link valid_lft forever preferred_lft forever
3,
linux网卡虚拟化
TAP设备:模拟一个二层的网络设备,可以接收和发送二层网络数据包
如上的ip 命令可以看到:
6: tap35bde514-d6@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000link/ether 16:23:e2:df:f1:59 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: tapdc2df59d-2b@if2: mtu 1500 qdisc noqueue master brq688a0356-4f state UP qlen 1000link/ether 9a:c6:e7:43:dd:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 1
8: tap29f235e4-7b@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000link/ether 76:81:a0:0a:0d:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 2
TUN设备:模拟一个三层的网络设备,可以接收和发送三层网络数据包
在kubernetes的VXLAN网络模式下比较常见,openstack里目前还没发现
工作于二层的网络设备,功能类似物理交换机
brigde可以绑定linux上其他网络设备,并将这些设备虚拟化为端口,
端口
逻辑网络交换机上的虚拟交换端口
虚拟机通过port附着到network上
port可以分配ip地址和mac地址
例如:
####注:这里的端口和通常意义的端口不是一个概念
[root@openstack1 ~]# openstack port list
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+
| 29f235e4-7b56-4003-8cce-da3a55e6f49c | | fa:16:3e:c2:d0:0e | ip_address='172.16.1.1', subnet_id='1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68' | ACTIVE |
| 35bde514-d626-4b89-ab0b-e026da168ef2 | | fa:16:3e:8b:cc:69 | ip_address='172.16.1.2', subnet_id='1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68' | ACTIVE |
| 66ff50ec-4b9a-4c64-a60b-128c196bdf9d | | fa:16:3e:b2:6b:16 | ip_address='192.168.123.151', subnet_id='687b56a4-0672-451e-810c-ea3b895e3030' | N/A |
| b840246c-96e6-4385-97b8-b32fcc017fc1 | | fa:16:3e:98:68:14 | ip_address='172.16.1.26', subnet_id='1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68' | ACTIVE |
| bd360ee2-54bd-4590-979d-12996231d083 | | fa:16:3e:12:63:bb | ip_address='192.168.123.164', subnet_id='687b56a4-0672-451e-810c-ea3b895e3030' | N/A |
| c3456277-fb14-42d1-a86e-e4c1953fb15e | | fa:16:3e:15:a8:a3 | ip_address='192.168.123.179', subnet_id='687b56a4-0672-451e-810c-ea3b895e3030' | N/A |
| dc2df59d-2ba4-4d61-92ea-4832200fe129 | | fa:16:3e:e7:a0:74 | ip_address='192.168.123.150', subnet_id='687b56a4-0672-451e-810c-ea3b895e3030' | ACTIVE |
| e43e2f28-5bff-475b-8635-964ba90f4ee3 | | fa:16:3e:40:00:11 | ip_address='192.168.123.156', subnet_id='687b56a4-0672-451e-810c-ea3b895e3030' | N/A |
| f4264f35-1841-41c6-9b4f-5ca97eee162a | | fa:16:3e:30:aa:9c | ip_address='192.168.123.155', subnet_id='687b56a4-0672-451e-810c-ea3b895e3030' | ACTIVE |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+
相比linux bridge的小规模的主机内部通信场景,open vswitch更适合大规模的多主机通信场景
pytsical network,物理网络。
在物理网络环境中连接到openstack不同节点的网络,每个物理网络可以支持neutron中的一个或者多个虚拟网络。
连接租户内同一个network或者不同network之间的子网,以及连接内外网
固定ip,分配到每个端口上的ip,类似于物理环境中配置到网卡上的ip
floating ip(浮动ip)是external network创建的一种特殊的port,可以将floating ip绑定到任意network中的port上,底层会进行nat转发,将发送的浮动ip流量转发到该port上的对应固定ip上,外界可以通过浮动ip访问虚拟机,虚拟机也可以通过浮动ip访问外界
外部网络,也叫公共网络
它是一种特殊的provider network,连接的物理网络与数据中心或者internet相通,网络中的port可以访问外网
一般将租户的virtual router连接到该网络,并创建floating ip绑定虚拟机,实现虚拟机与外网通信
Exernal netwok类似于物理环境中直接使用公网ip网段,不同的是,openstack中external network对应的物理网络不一定能直连internet,有可能只是数据中心的一个内部私有网络。
由openstack管理员创建,直接对应数据中心现有物理网络的一个网段
providr network通常使用vlan或者flat模式,可以在多个租户之间共享
连接租户内同一个network或者不同network之间的子网,以及连接内外网(虚拟路由)
安全组,他的作用是在neutron port上的一组策略,规定了虚拟机入口和出口流量的规则
安全组基于linux iptables实现,默认拒绝所有流量,只有添加了放行规则的流量才允许通过
每个openstack项目中都有一个default默认安全组,默认包含如下规则-拒绝所有入口流量,允许所有出口流量
14.
南北向流量
虚拟机和外部网络通信的流量
15.
东西向流量
虚拟机之间的流量
二,
虚拟机实例(底层是libvirtd的kvm)的两个驱动---网卡驱动和磁盘驱动的属性
Manage images — glance 26.0.0.0b3.dev12 documentation (openstack官网文档里的)
Disk and CD-ROM bus model values
libvirt_type setting | Supported model values |
---|---|
qemu or kvm |
|
xen |
|
VIF model values
libvirt_type setting | Supported model values |
---|---|
qemu or kvm |
|
xen |
|
vmware |
|
由镜像制作的时候确定,该镜像使用哪种驱动,一般情况下,官方的kvm镜像使用的磁盘驱动是virtio,网卡驱动也是virtio,(例如,CentOS-7-x86_64-GenericCloud-1508.qcow2)但如果是在VM虚拟机里部署的openstack,那么,可能需要更换镜像磁盘驱动为IDE(物理机部署的openstack一般不存在此问题)
例如这么修改镜像属性:
openstack image set --property hw_disk_bus=ide --property hw_vif_model=e1000 867784df-9090-4ae0-ad25-564b941cc350
如果网卡属性设置错误,可能这个镜像转换成虚拟机实例的时候是成功的,但网络会出现问题,各种连接不通的情况,例如,我进入的一个设置了正确的网卡驱动的虚拟机实例,可以看到eth0网卡绑定有正确的IP(如果设置的是e1000网卡属性,那么,kvm虚拟机的网卡可能是ens3,这里就表示网卡没有虚拟成功,没有IP落在上面的):
仍然是使用上面在openstack内成功运行的虚拟机实例来讲解,每一个虚拟机启动后都会有一个配置文件,此文件存放路径为:/etc/libvirt/qemu
例如,这个实例可以看到是运行在openstack3这个计算节点的,名称是instance-0000001a
[root@openstack1 ~]# openstack server show centos7-init
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstack3 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstack3 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001a |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2023-02-10T08:35:06.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | selfservice=172.16.1.26, 192.168.123.156 |
| config_drive | True |
| created | 2023-02-10T08:34:26Z |
| flavor | m1.tiny (1) |
| hostId | cd6f0c1b6d4c4b38cdbdb156ba72eade7b3d1152222433c9565ed082 |
| id | 256bc90a-3e39-471c-b7a3-fa271aeb76da |
| image | Centos7 (2b672c8f-75bf-4fe0-8468-37309563025a) |
| key_name | mykey |
| name | centos7-init |
| progress | 0 |
| project_id | 205ce8addd9444c893bd62244bcdae78 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2023-02-11T01:05:29Z |
| user_id | 74bc206609e04092b698698d944e922a |
| volumes_attached | |
+-------------------------------------+----------------------------------------------------------+
OK,进入计算节点,查看虚拟机实例的配置文件:
[root@openstack3 ~]# cat /etc/libvirt/qemu/instance-0000001a.xml
instance-0000001a 256bc90a-3e39-471c-b7a3-fa271aeb76da centos7-init 2023-02-11 01:05:27 1024 50 0 0 1 admin admin 1048576 1048576 1 1024 RDO OpenStack Compute 18.3.0-1.el7 4d5cab7f-b3d7-45e6-9de6-5f0eb3e00498 256bc90a-3e39-471c-b7a3-fa271aeb76da Virtual Machine hvm destroy restart destroy /usr/libexec/qemu-kvm
其中的网络方面关键配置予以重点关注:
openstack3这个节点的网络概况:
可以看到第五个网卡是这个虚拟机使用的虚拟网卡,源网卡是第四个网卡,桥接模式
[root@openstack3 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens33: mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000link/ether 00:0c:29:c2:1f:10 brd ff:ff:ff:ff:ff:ffinet 192.168.123.132/24 brd 192.168.123.255 scope global ens33valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fec2:1f10/64 scope link valid_lft forever preferred_lft forever
4: brq6ef09648-5c: mtu 1450 qdisc noqueue state UP qlen 1000link/ether 5a:b2:95:2c:4d:ec brd ff:ff:ff:ff:ff:ff
5: tapb840246c-96: mtu 1450 qdisc pfifo_fast master brq6ef09648-5c state UNKNOWN qlen 1000link/ether fe:16:3e:98:68:14 brd ff:ff:ff:ff:ff:ffinet6 fe80::fc16:3eff:fe98:6814/64 scope link valid_lft forever preferred_lft forever
6: vxlan-17: mtu 1450 qdisc noqueue master brq6ef09648-5c state UNKNOWN qlen 1000link/ether 5a:b2:95:2c:4d:ec brd ff:ff:ff:ff:ff:ff
回到控制节点,查询port,可以看到mac是'fa:16:3e:98:68:14'的网卡绑定的IP是172.16.1.26:
[root@openstack1 ~]# openstack port list
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+
| 29f235e4-7b56-4003-8cce-da3a55e6f49c | | fa:16:3e:c2:d0:0e | ip_address='172.16.1.1', subnet_id='1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68' | ACTIVE |
| 35bde514-d626-4b89-ab0b-e026da168ef2 | | fa:16:3e:8b:cc:69 | ip_address='172.16.1.2', subnet_id='1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68' | ACTIVE |
| b840246c-96e6-4385-97b8-b32fcc017fc1 | | fa:16:3e:98:68:14 | ip_address='172.16.1.26', subnet_id='1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68' | ACTIVE |
在控制节点,进一步查询:
[root@openstack1 ~]# ip a |grep brq6ef09648-5c
7: tap35bde514-d6@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000
9: vxlan-17: mtu 1450 qdisc noqueue master brq6ef09648-5c state UNKNOWN qlen 1000
10: brq6ef09648-5c: mtu 1450 qdisc noqueue state UP qlen 1000
11: tap29f235e4-7b@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000
以上,可得brq6ef09648-5c是虚拟交换机,tapb840246c-96是虚拟机使用的虚拟网卡
OK,关闭虚拟机后,查看计算节点openstack3和控制节点openstack1的网络:
控制节点:
[root@openstack1 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens33: mtu 1500 qdisc pfifo_fast master brq688a0356-4f state UNKNOWN qlen 1000link/ether 00:0c:29:de:be:e1 brd ff:ff:ff:ff:ff:ffinet6 fe80::20c:29ff:fede:bee1/64 scope link valid_lft forever preferred_lft forever
3: virbr0: mtu 1500 qdisc noqueue state DOWN qlen 1000link/ether 52:54:00:e8:93:4e brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000link/ether 52:54:00:e8:93:4e brd ff:ff:ff:ff:ff:ff
6: tapdc2df59d-2b@if2: mtu 1500 qdisc noqueue master brq688a0356-4f state UP qlen 1000link/ether 32:c7:4f:44:8a:75 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: tap35bde514-d6@if2: mtu 1450 qdisc noqueue master brq6ef09648-5c state UP qlen 1000link/ether 3e:8e:f1:b5:b6:37 brd ff:ff:ff:ff:ff:ff link-netnsid 1
8: brq688a0356-4f: mtu 1500 qdisc noqueue state UP qlen 1000link/ether 00:0c:29:de:be:e1 brd ff:ff:ff:ff:ff:ffinet 192.168.123.130/24 brd 192.168.123.255 scope global brq688a0356-4fvalid_lft forever preferred_lft foreverinet6 fe80::e04b:63ff:fe8d:df13/64 scope link valid_lft forever preferred_lft forever
9: vxlan-17: mtu 1450 qdisc noqueue master brq6ef09648-5c state UNKNOWN qlen 1000link/ether 4e:20:be:fb:67:82 brd ff:ff:ff:ff:ff:ff
10: brq6ef09648-5c: mtu 1450 qdisc noqueue state UP qlen 1000link/ether 3e:8e:f1:b5:b6:37 brd ff:ff:ff:ff:ff:ffinet6 fe80::bc17:e5ff:fee8:c835/64 scope link valid_lft forever preferred_lft forever
计算节点:
[root@openstack3 ~]# ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens33: mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000link/ether 00:0c:29:c2:1f:10 brd ff:ff:ff:ff:ff:ffinet 192.168.123.132/24 brd 192.168.123.255 scope global ens33valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fec2:1f10/64 scope link valid_lft forever preferred_lft forever
4: brq6ef09648-5c: mtu 1450 qdisc noqueue state UP qlen 1000link/ether 5a:b2:95:2c:4d:ec brd ff:ff:ff:ff:ff:ff
6: vxlan-17: mtu 1450 qdisc noqueue master brq6ef09648-5c state UNKNOWN qlen 1000link/ether 5a:b2:95:2c:4d:ec brd ff:ff:ff:ff:ff:ff
可以看到,5号网卡没有了,印证了以上的结果,vxlan-17 到底是什么还是不太清楚。
四,
provider和self-service
self-service是对provider网络的扩展,也可以说是基于provider网络的扩展,意思就是必须要有provider
部署完成的self-service是这样的:
[root@openstack1 ~]# openstack network list
+--------------------------------------+-------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+-------------+--------------------------------------+
| 688a0356-4f2b-4029-b49e-a11bbdbedf0b | provider | 687b56a4-0672-451e-810c-ea3b895e3030 |
| 6ef09648-5ce7-4eb6-9aa2-735cb81567d3 | selfservice | 1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68 |
+--------------------------------------+-------------+--------------------------------------+
[root@openstack1 ~]# openstack subnet list
+--------------------------------------+-------------+--------------------------------------+------------------+
| ID | Name | Network | Subnet |
+--------------------------------------+-------------+--------------------------------------+------------------+
| 1bdfae33-5bcb-47ff-b3f0-aee2fc5e7f68 | selfservice | 6ef09648-5ce7-4eb6-9aa2-735cb81567d3 | 172.16.1.0/24 |
| 687b56a4-0672-451e-810c-ea3b895e3030 | provider1 | 688a0356-4f2b-4029-b49e-a11bbdbedf0b | 192.168.123.0/24 |
+--------------------------------------+-------------+--------------------------------------+------------------+
[root@openstack1 ~]# openstack router list
+--------------------------------------+--------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+--------+--------+-------+----------------------------------+-------------+-------+
| b208a515-e256-49c1-b939-15e47b9db07c | router | ACTIVE | UP | 205ce8addd9444c893bd62244bcdae78 | False | False |
+--------------------------------------+--------+--------+-------+----------------------------------+-------------+-------+
网络拓扑图:
图表显示,provider是外部网络,虚拟路由器是两网的连接点: