[NETVIRT-609] very small load brings odl/ovs system in cycle reconnect mood

Duplicate — Mon, 30 Oct 2017 19:52:44 +0000

I have only some few vm start in my deployment:

[root@overcloud-controller-0 ~]# nova list
neutron po+----------------------------------------------------------------------------------------------------------------------------------

Name

Status

Task State

Power State

Networks

----------------------------------------------------------------------------------------------------------------------------------+

ebac75a9-2471-4429-aa3b-16d42980d497	sdnvpn-3-2-quagga	ACTIVE	Running	sdnvpn-3-2-quagga-net=10.10.11.5, 192.168.37.202
348ff4e8-85da-478a-8cc5-bd04fda2f92d	sdnvpn-4-1	ACTIVE	Running	sdnvpn-4-1-net=10.10.10.7
e1adbebb-0a35-4ba9-bdc4-78d28dea5d9f	sdnvpn-4-2	ACTIVE	Running	sdnvpn-4-1-net=10.10.10.12
847734a2-af60-4852-b151-ab3c2e30f8ed	sdnvpn-4-3	ACTIVE	Running	sdnvpn-4-1-net=10.10.10.8
ee728f1a-ddb8-4ea9-9e6e-99164c064f92	sdnvpn-4-4	ACTIVE	Running	sdnvpn-4-2-net=10.10.11.3
1c03e896-340f-4626-98d5-265854ddb668	sdnvpn-4-5	ACTIVE	Running	sdnvpn-4-2-net=10.10.11.11
3593e96e-3fe6-45e1-8715-7b7ba239b45c	sdnvpn-8-1	ACTIVE	Running	sdnvpn-8-1=10.10.10.11, 192.168.37.203
bbd4f29b-0b18-459b-957b-b95f0ef73859	sdnvpn-8-2	ACTIVE	Running	sdnvpn-8-2=10.10.20.5

----------------------------------------------------------------------------------------------------------------------------------+
r[root@overcloud-controller-0 ~]# neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------------+

name

mac_address

fixed_ips

------------------------------------------------------------------------------------------------------------------------------------------------+

04e9f21e-0336-4773-9020-e9642149201e	fa:16:3e:32:99:a4	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.204"}
0680b874-d259-4250-a983-d88f779fcfd7	fa:16:3e:eb:d9:ff	{"subnet_id": "35d017d1-a098-4c29-9bcd-8bd9274e493f", "ip_address": "10.10.10.2"}
13de230f-5e18-4936-8c61-832f088174c8	fa:16:3e:d1:34:6d	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.215"}
2329e32a-888e-4a2c-adca-6fddcbffc179	fa:16:3e:16:25:be	{"subnet_id": "35d017d1-a098-4c29-9bcd-8bd9274e493f", "ip_address": "10.10.10.1"}
246ffaad-3f1f-48cd-8a15-c34c848ec2ea	fa:16:3e:a0:92:e5	{"subnet_id": "430a2296-c009-4b13-9984-2f532913acc4", "ip_address": "10.10.10.1"}
36c5d510-a339-4e35-a42e-8a295c9858e8	fa:16:3e:7e:77:8b	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.208"}
3acb8e14-c3ff-4925-83be-fdfecbf0dc65	fa:16:3e:f2:99:c0	{"subnet_id": "cd3bb114-9f4e-4936-9eff-8989b01427e5", "ip_address": "10.10.11.2"}
3d20beb3-9251-46fa-9cdd-ad8b8ef2a2bf	fa:16:3e:ae:17:5f	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.202"}
4b9e6f37-871a-4717-983d-b87be9b38d67	fa:16:3e:0a:d0:8a	{"subnet_id": "7c6b5f56-71fb-4728-bb95-5020b33b55c5", "ip_address": "10.10.20.5"}
5e92c202-d215-49b2-b9d7-1dd516d8e9fa	fa:16:3e:eb:02:87	{"subnet_id": "58399558-1fd9-43ba-a214-6236c0113194", "ip_address": "10.10.10.2"}
60549335-ff8c-4eaa-9c9d-f845c261b4ff	fa:16:3e:75:cb:83	{"subnet_id": "58399558-1fd9-43ba-a214-6236c0113194", "ip_address": "10.10.10.1"}
74c879d8-f94c-4df6-85a5-96f096b26990	fa:16:3e:65:58:fc	{"subnet_id": "ed189d60-b81b-4f05-87b3-5f436548345f", "ip_address": "10.10.11.5"}
75c558c8-b4b3-4c0f-a5a1-2266593b14a5	fa:16:3e:96:84:65	{"subnet_id": "430a2296-c009-4b13-9984-2f532913acc4", "ip_address": "10.10.10.2"}
7743129e-e28b-4aac-b0df-49b11aa9e450	fa:16:3e:e0:eb:79	{"subnet_id": "ed189d60-b81b-4f05-87b3-5f436548345f", "ip_address": "10.10.11.2"}
79be70a5-afe4-484a-ba2d-dda93c1fbcd4	fa:16:3e:61:1c:90	{"subnet_id": "430a2296-c009-4b13-9984-2f532913acc4", "ip_address": "10.10.10.7"}
81484fb1-4265-4d6d-a1c6-2a4aadc4f571	fa:16:3e:37:3d:57	{"subnet_id": "7c6b5f56-71fb-4728-bb95-5020b33b55c5", "ip_address": "10.10.20.2"}
99dbe740-a456-471b-ad85-ef708b9fac03	fa:16:3e:2b:14:38	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.203"}
9a72f914-2dac-4c0e-869e-d1cf84ab350b	fa:16:3e:02:59:42	{"subnet_id": "ed189d60-b81b-4f05-87b3-5f436548345f", "ip_address": "10.10.11.1"}
a89e432a-c485-4549-bdf0-9802a04efbca	fa:16:3e:26:a8:6d	{"subnet_id": "cd3bb114-9f4e-4936-9eff-8989b01427e5", "ip_address": "10.10.11.11"}
abc364c2-6560-4657-8402-ad890a049cb7	fa:16:3e:c5:ff:13	{"subnet_id": "430a2296-c009-4b13-9984-2f532913acc4", "ip_address": "10.10.10.8"}
b99dc5ff-0506-49fb-a54a-67606a6a438b	fa:16:3e:fd:f6:03	{"subnet_id": "cd3bb114-9f4e-4936-9eff-8989b01427e5", "ip_address": "10.10.11.3"}
be694def-9bff-461a-aea9-dfcb15b74497	fa:16:3e:fa:33:09	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.211"}
c9aadf0f-8608-4f39-96e1-af7b13178960	fa:16:3e:01:47:29	{"subnet_id": "850a01b8-4d98-4638-8e49-4fff4471c303", "ip_address": "192.168.37.201"}
e82cda88-ce29-4f0c-a519-80fc1686b6ca	fa:16:3e:8c:61:6c	{"subnet_id": "35d017d1-a098-4c29-9bcd-8bd9274e493f", "ip_address": "10.10.10.11"}
e9484795-4a9c-4267-8f8f-32baf275406c	fa:16:3e:37:18:96	{"subnet_id": "430a2296-c009-4b13-9984-2f532913acc4", "ip_address": "10.10.10.12"}

------------------------------------------------------------------------------------------------------------------------------------------------+

and as you can see in the appended jpg ovs connects and disconnects always from odl. ODL consume 100% cpu load:

1 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||98.1%] Tasks: 135, 1192 thr; 23 running
2 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||95.5%] Load average: 12.45 13.26 9.64
3 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||96.8%] Uptime: 02:11:36
4 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||97.4%]
Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||7.35G/9.61G]
Swp[ 0K/0K]

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
28511 odl 20 0 6177M 2597M 21064 S 362. 26.4 1h16:33 /usr/bin/java -Djava.security.properties=/opt/opendaylight/etc/odl.java.security -server -Xms128M -Xmx2048m
28679 odl 20 0 6177M 2597M 21064 R 92.6 26.4 13:36.81 /usr/bin/java -Djava.security.properties=/opt/opendaylight/etc/odl.java.security -server -Xms128M -Xmx2048m
28678 odl 20 0 6177M 2597M 21064 R 88.1 26.4 13:33.38 /usr/bin/java -Djava.security.properties=/opt/opendaylight/etc/odl.java.security -server -Xms128M -Xmx2048m
28680 odl 20 0 6177M 2597M 21064 R 84.9 26.4 13:37.51 /usr/bin/java -Djava.security.properties=/opt/opendaylight/etc/odl.java.security -server -Xms128M -Xmx2048m
28677 odl 20 0 6177M 2597M 21064 R 82.3 26.4 13:36.90 /usr/bin/java -Djava.security.properties=/opt/opendaylight/etc/odl.java.security -server -Xms128M -Xmx2048m

this is insane. 10 GB ram and 4 cores should be enough to run with 3 ovs.

OpenDaylight JIRA

[NETVIRT-609] very small load brings odl/ovs system in cycle reconnect mood