Lets create another network in the same tenant, and create 3 more VMS.
+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+ | ID | Name | Tenant ID | Status | Task State | Power State | Networks | +--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+ | ee98e356-e017-4cf1-93c7-7483370d4fe2 | vm201-1 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | dhcp-192-168-201=192.168.201.15 | | 756558c6-2064-4494-b432-26ac1c408f15 | vm201-2 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | dhcp-192-168-201=192.168.201.17 | | cecc9d0f-5301-4d5f-bcfd-63fb1f357fb5 | vm201-3 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | dhcp-192-168-201=192.168.201.16 | | f3d39ca9-3291-4aa3-b1e2-73279d1409bf | vx-1 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet=10.0.0.7 | | 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 | vx-2 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet=10.0.0.8 | | a0b423d0-a017-408e-9988-bb86abf1943f | vx-3 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet=10.0.0.9 | | 82fcbddc-f10a-41d9-b628-c558994d5e8c | vx2-1 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet-2=10.0.0.8 | | e71a3d7b-cc66-4ddd-82b3-c26554e43bcf | vx2-2 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet-2=10.0.0.10 | | 25745ee9-35f5-48e4-9c51-f35ac6988d95 | vx2-3 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet-2=10.0.0.9 | +--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+
Lets check where these VMs are running [you can also do nova show $uuid and get the host where its running currently]
MariaDB [nova]> select hostname,host,uuid from instances where vm_state='active'; +----------+------+--------------------------------------+ | hostname | host | uuid | +----------+------+--------------------------------------+ | vm201-1 | c25 | ee98e356-e017-4cf1-93c7-7483370d4fe2 | | vm201-2 | c24 | 756558c6-2064-4494-b432-26ac1c408f15 | | vm201-3 | c25 | cecc9d0f-5301-4d5f-bcfd-63fb1f357fb5 | | vx-1 | c25 | f3d39ca9-3291-4aa3-b1e2-73279d1409bf | | vx-2 | c24 | 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 | | vx-3 | c25 | a0b423d0-a017-408e-9988-bb86abf1943f | | vx2-1 | c24 | 82fcbddc-f10a-41d9-b628-c558994d5e8c | | vx2-2 | c25 | e71a3d7b-cc66-4ddd-82b3-c26554e43bcf | | vx2-3 | c24 | 25745ee9-35f5-48e4-9c51-f35ac6988d95 | +----------+------+--------------------------------------+
neutron net-list
+--------------------------------------+----------------------------------------------------+-------------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------------------------------------------------+-------------------------------------------------------+ | 720995e0-bc7c-4f1e-91b0-3a61f9254776 | HA network tenant adf998ea387e4fb684246431de401047 | 3d4c8a7f-cb08-45b7-8a6d-e5e4b177da81 169.254.192.0/18 | | c4520434-aea7-4263-b714-ac0b6fcdc57a | officeNet | 070ef3e3-5087-4d57-a907-2821af869575 10.0.0.0/24 | | 6bfcadcd-6251-4480-8b4e-c22a9151bf00 | HA network tenant 3b86f724614540bc902d3570b0bf86f1 | 8306c1e7-6d0f-46ec-bbc5-1c02362c1f44 169.254.192.0/18 | | f42a0fb8-0111-466a-b17a-9ff4b95d8c14 | floating-192-168-101 | 4ef2012a-f526-4e05-bce6-7f8d202d903c 192.168.101.0/24 | | a8ccf499-4ff3-44b3-9541-1d35693ecf80 | dhcp-192-168-201 | 7fe7c672-613e-440a-9780-547a94b38805 192.168.201.0/24 | | 52fcc1ec-86da-4ade-befa-fc81d868b44b | officeNet-2 | 0cd4831b-65a6-4be7-a74b-9efd3f74d826 10.0.0.0/24 | +--------------------------------------+----------------------------------------------------+-------------------------------------------------------+
neutron net-show c4520434-aea7-4263-b714-ac0b6fcdc57a
+---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | c4520434-aea7-4263-b714-ac0b6fcdc57a | | mtu | 0 | | name | officeNet | | port_security_enabled | True | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 87 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 070ef3e3-5087-4d57-a907-2821af869575 | | tenant_id | 3b86f724614540bc902d3570b0bf86f1 | +---------------------------+--------------------------------------+
check segmentation_id — its 87
neutron net-show officeNet-2 [using name also works ]
+---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | id | 52fcc1ec-86da-4ade-befa-fc81d868b44b | | mtu | 0 | | name | officeNet-2 | | port_security_enabled | True | | provider:network_type | vxlan | | provider:physical_network | | | provider:segmentation_id | 15 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 0cd4831b-65a6-4be7-a74b-9efd3f74d826 | | tenant_id | 3b86f724614540bc902d3570b0bf86f1 | +---------------------------+--------------------------------------+
check segmentation_id — its 15
lets take a closer look at these:
+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+ | ID | Name | Tenant ID | Status | Task State | Power State | Networks | +--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+ | 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 | vx-2 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet=10.0.0.8 | | 82fcbddc-f10a-41d9-b628-c558994d5e8c | vx2-1 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | - | Running | officeNet-2=10.0.0.8 | | +--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+
So I have vx-2 and vx-2-1 with the same IP address in the same host. (c24)
root@c14_utility_container-cc97b930:~# openstack server show 9f0fb1af-192b-4a30-a41b-8c8694ac9f05
+--------------------------------------+----------------------------------------------------------+
| Field | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | c24 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | c24 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000002a |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-04-04T21:53:09.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | officeNet=10.0.0.8 |
| config_drive | |
| created | 2016-04-04T21:52:55Z |
| flavor | m1.tiny (7dcb6a3f-6299-43d5-8634-660bab9f61cf) |
| hostId | 120b05fb25dc760cd00e26062cd3b678f39c37b82a57b19741350abc |
| id | 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 |
| image | cirros (166eb283-efc8-4a48-bbca-a9797c6b6136) |
| key_name | sda |
| name | vx-2 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | 3b86f724614540bc902d3570b0bf86f1 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | ACTIVE |
| updated | 2016-04-04T21:53:10Z |
| user_id | 40089a72571c48ab9817f209c262b38a |
+--------------------------------------+----------------------------------------------------------+
root@c14_utility_container-cc97b930:~# openstack server show 82fcbddc-f10a-41d9-b628-c558994d5e8c
+--------------------------------------+----------------------------------------------------------+
| Field | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | c24 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | c24 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000033 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-04-05T10:04:49.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | officeNet-2=10.0.0.8 |
| config_drive | |
| created | 2016-04-05T10:04:34Z |
| flavor | m1.tiny (7dcb6a3f-6299-43d5-8634-660bab9f61cf) |
| hostId | 120b05fb25dc760cd00e26062cd3b678f39c37b82a57b19741350abc |
| id | 82fcbddc-f10a-41d9-b628-c558994d5e8c |
| image | cirros (166eb283-efc8-4a48-bbca-a9797c6b6136) |
| key_name | sda |
| name | vx2-1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | 3b86f724614540bc902d3570b0bf86f1 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | ACTIVE |
| updated | 2016-04-05T10:04:50Z |
| user_id | 40089a72571c48ab9817f209c262b38a |
+--------------------------------------+----------------------------------------------------------+
root@c14_utility_container-cc97b930:~# ssh c24 virsh dumpxml instance-0000002a
<interface type='bridge'>
<mac address='fa:16:3e:cc:cc:a5'/>
<source bridge='brqc4520434-ae'/>
<target dev='tap3a901e7d-bb'/>
<model type='virtio'/>
<driver name='qemu'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
root@c14_utility_container-cc97b930:~# ssh c24 virsh dumpxml instance-00000033
<interface type='bridge'>
<mac address='fa:16:3e:54:14:68'/>
<source bridge='brq52fcc1ec-86'/>
<target dev='tap5f2653c4-19'/>
<model type='virtio'/>
<driver name='qemu'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
root@c14_utility_container-cc97b930:~# ssh c24 brctl show
br-vxlan 8000.525400314cee no eth4
brqc4520434-ae 8000.3a6872e704d4 no tap3a901e7d-bb
vxlan-87
brq52fcc1ec-86 8000.1aa8a59e8510 no tap5f2653c4-19
vxlan-15
As you can see, vx-2 is vlan-87 and vx2-1 is on vlan-15, so they are separated and treated differently.
If there are 2 instances of the same network (same vxlan), they would appear in the same brqXXX bridge.
root@c14_utility_container-cc97b930:~# ssh c25 brctl show
brq52fcc1ec-86 8000.aa6e9cf8e558 no tap7bc323f8-8e
vxlan-15
brqc4520434-ae 8000.6a859e839df6 no tap790aea7e-c0
tape2b6748f-8c
vxlan-87
So there is also something on c25 that is on vxlan-15
ifconfig vxlan-15 or ip link show vxlan-15 is not going to show anything special 🙂
you need to do
ip -d link show vxlan-15
XX: vxlan-15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq52fcc1ec-86 state UNKNOWN mode DEFAULT group default
link/ether 1a:a8:a5:9e:85:10 brd ff:ff:ff:ff:ff:ff promiscuity 1
vxlan id 15 group 239.1.1.1 dev br-vxlan port 32768 61000 ageing 300
and its going to show the vxlan id – 15 , that its a part of the multicast group and uses dev-vxlan.
So as you see, there are a lot of interfaces 🙂
eth0 (inside the VM) <=> tapXXXX (outside the VM, on compute node) <=> brqXXX(bridge) <=> vxlan-XX (vtep) <=> br-vxlan <=>eth4 (physical interface) <=> switch <=> [ the other side]
So when I ping between instances on different compute hosts and run tcpdump
I can see the packets directly on tapXXX and on brqXXX and also on vxlan-xx.
tcpdump on br-vxlan and eth4 shows the layer-3 br-vxlan ips on top.
When i ping 10.0.0.9(c25) from 10.0.0.8 (on c24),
inside the 10.0.0.8 instance
$ arp -n
(10.0.0.9) at fa:16:3e:da:3e:57 [ether] on eth0
From outside the instance:
All learned mac addresses are in the bridge (vxlan-87)
root@c24:~# brctl showmacs brqc4520434-ae port no mac addr is local? ageing timer 2 3a:68:72:e7:04:d4 yes 0.00 1 fa:16:3e:cc:cc:a5 no 0.08 2 fa:16:3e:da:3e:57 no 0.08 1 fe:16:3e:cc:cc:a5 yes 0.00
and
root@c24:~# bridge fdb show dev vxlan-87 3a:68:72:e7:04:d4 vlan 0 permanent fa:16:3e:da:3e:57 vlan 0 00:00:00:00:00:00 dst 239.1.1.1 via br-vxlan self permanent fa:16:3e:da:3e:57 dst 172.29.240.25 self
see that dst is 172.29.240.25 which is the vxlan ip of c25.