https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-589 netvirt <p>Hey,</p> <p>We see big issues in using stable/boron (nearly SR-3) in the OPNFV test pipeline. This is the test what is done:<br/> 2017-03-31 15:51:36,569 - openstack_utils - INFO - Creating neutron network sdnvpn-8-1...<br/> 2017-03-31 15:51:36,919 - openstack_utils - DEBUG - Network '130d6058-1321-4d4b-bc89-99c62e6dc971' created successfully<br/> 2017-03-31 15:51:36,919 - openstack_utils - DEBUG - Creating Subnet....<br/> 2017-03-31 15:51:37,166 - openstack_utils - DEBUG - Subnet '98a442fa-01d8-4241-a07d-af5dc22b7259' created successfully<br/> 2017-03-31 15:51:37,166 - openstack_utils - DEBUG - Creating Router...<br/> 2017-03-31 15:51:37,300 - openstack_utils - DEBUG - Router '80e53bf6-1426-46f8-9c2f-ee96d6efc644' created successfully<br/> 2017-03-31 15:51:37,301 - openstack_utils - DEBUG - Adding router to subnet...<br/> 2017-03-31 15:51:38,899 - openstack_utils - DEBUG - Interface added successfully.<br/> 2017-03-31 15:51:38,899 - openstack_utils - DEBUG - Adding gateway to router...<br/> 2017-03-31 15:51:40,276 - openstack_utils - DEBUG - Gateway added successfully.<br/> 2017-03-31 15:51:40,276 - sndvpn_test_utils - DEBUG - Creating network sdnvpn-8-2<br/> 2017-03-31 15:51:40,497 - sndvpn_test_utils - DEBUG - Creating subnet sdnvpn-8-2-subnet in network 773cd3bd-24a5-4886-9cb0-87fe7426c818 with cidr 10.10.20.0/24<br/> 2017-03-31 15:51:41,175 - openstack_utils - INFO - Creating security group 'sdnvpn-sg'...<br/> 2017-03-31 15:51:41,308 - openstack_utils - DEBUG - Security group 'sdnvpn-sg' with ID=65206fea-7be2-49b6-abce-b32c3c46a61a created successfully.<br/> 2017-03-31 15:51:41,309 - openstack_utils - DEBUG - Adding ICMP rules in security group 'sdnvpn-sg'...<br/> 2017-03-31 15:51:41,309 - openstack_utils - DEBUG - Security_group format set (no port range mentioned)<br/> 2017-03-31 15:51:41,455 - openstack_utils - DEBUG - Adding SSH rules in security group 'sdnvpn-sg'...<br/> 2017-03-31 15:51:41,456 - openstack_utils - DEBUG - Security_group format set (port range included)<br/> 2017-03-31 15:51:41,610 - openstack_utils - DEBUG - Security_group format set (port range included)<br/> 2017-03-31 15:51:41,773 - openstack_utils - DEBUG - Security_group format set (no port range mentioned) Neutron server returns request_ids: <span class="error">&#91;&#39;req-3afdd79b-09c5-41c0-a3d5-edc077407a21&#39;&#93;</span><br/> 2017-03-31 15:51:41,872 - openstack_utils - ERROR - Bad security group format.One of the port range is not properly set:range min: 80,range max: None<br/> 2017-03-31 15:51:41,872 - sndvpn_test_utils - INFO - Creating instance 'sdnvpn-8-2'...<br/> 2017-03-31 15:51:41,873 - sndvpn_test_utils - DEBUG - Configuration:<br/> name=sdnvpn-8-2<br/> flavor=m1.tiny<br/> image=1018ece1-b44c-4fef-857e-bb82f2a61f2f<br/> network=773cd3bd-24a5-4886-9cb0-87fe7426c818<br/> secgroup=65206fea-7be2-49b6-abce-b32c3c46a61a<br/> hypervisor=<br/> fixed_ip=None<br/> files=None<br/> userdata=<br/> None</p> <p>2017-03-31 15:51:41,889 - keystoneauth.identity.v2 - DEBUG - Making authentication request to <a href="http://192.168.37.10:5000/v2.0/tokens" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.37.10:5000/v2.0/tokens</a><br/> 2017-03-31 15:51:43,592 - keystoneauth.identity.v2 - DEBUG - Making authentication request to <a href="http://192.168.37.10:5000/v2.0/tokens" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.37.10:5000/v2.0/tokens</a><br/> 2017-03-31 15:51:51,896 - sndvpn_test_utils - DEBUG - Instance 'sdnvpn-8-2' booted successfully. IP='10.10.20.3'.<br/> 2017-03-31 15:51:51,896 - sndvpn_test_utils - DEBUG - Adding 'sdnvpn-8-2' to security group 'sdnvpn-sg'...<br/> 2017-03-31 15:51:51,905 - keystoneauth.identity.v2 - DEBUG - Making authentication request to <a href="http://192.168.37.10:5000/v2.0/tokens" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.37.10:5000/v2.0/tokens</a><br/> 2017-03-31 15:51:53,080 - sndvpn_test_utils - INFO - Creating instance 'sdnvpn-8-1'...<br/> 2017-03-31 15:51:53,081 - sndvpn_test_utils - DEBUG - Configuration:<br/> name=sdnvpn-8-1<br/> flavor=m1.tiny<br/> image=1018ece1-b44c-4fef-857e-bb82f2a61f2f<br/> network=130d6058-1321-4d4b-bc89-99c62e6dc971<br/> secgroup=65206fea-7be2-49b6-abce-b32c3c46a61a<br/> hypervisor=<br/> fixed_ip=None<br/> files=None<br/> userdata=<br/> #!/bin/sh<br/> set 10.10.20.3<br/> while true; do<br/> for i do<br/> ip=$i<br/> ping -c 1 $ip 2&gt;&amp;1 &gt;/dev/null<br/> RES=$?<br/> if [ "Z$RES" = "Z0" ] ; then<br/> echo ping $ip OK<br/> else echo ping $ip KO<br/> fi<br/> done<br/> sleep 1<br/> done</p> <p>2017-03-31 15:51:53,095 - keystoneauth.identity.v2 - DEBUG - Making authentication request to <a href="http://192.168.37.10:5000/v2.0/tokens" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.37.10:5000/v2.0/tokens</a><br/> 2017-03-31 15:51:54,357 - keystoneauth.identity.v2 - DEBUG - Making authentication request to <a href="http://192.168.37.10:5000/v2.0/tokens" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.37.10:5000/v2.0/tokens</a><br/> 2017-03-31 15:52:02,684 - sndvpn_test_utils - DEBUG - Instance 'sdnvpn-8-1' booted successfully. IP='10.10.10.7'.<br/> 2017-03-31 15:52:02,684 - sndvpn_test_utils - DEBUG - Adding 'sdnvpn-8-1' to security group 'sdnvpn-sg'...<br/> 2017-03-31 15:52:03,493 - sdnvpn-results - INFO - Create VPN with eRT==iRT<br/> 2017-03-31 15:52:03,649 - sdnvpn-testcase-8 - DEBUG - VPN created details: {u'bgpvpn': {u'export_targets': <span class="error">&#91;u&#39;88:88&#39;&#93;</span>, u'name': u'sdnvpn-7', u'route_distinguishers': <span class="error">&#91;u&#39;18:18&#39;&#93;</span>, u'routers': [], u'import_targets': <span class="error">&#91;u&#39;88:88&#39;&#93;</span>, u'networks': <br/> [], u'tenant_id': u'a94d4d97de9d4200ba9beca8d05c83d5', u'route_targets': [], u'project_id': u'a94d4d97de9d4200ba9beca8d05c83d5', u'type': u'l3', u'id': u'07c91f3f-df7d-4177-87c4-6320127359e8'}}<br/> 2017-03-31 15:52:03,649 - sdnvpn-results - INFO - Associate router 'sdnvpn-8-1-router' and net 'sdnvpn-8-2' to the VPN.<br/> 2017-03-31 15:52:04,965 - sndvpn_test_utils - DEBUG - Waiting for router 07c91f3f-df7d-4177-87c4-6320127359e8 to associate with BGPVPN 80e53bf6-1426-46f8-9c2f-ee96d6efc644<br/> 2017-03-31 15:52:06,129 - sndvpn_test_utils - DEBUG - Waiting for network 07c91f3f-df7d-4177-87c4-6320127359e8 to associate with BGPVPN 773cd3bd-24a5-4886-9cb0-87fe7426c818<br/> 2017-03-31 15:52:07,311 - sndvpn_test_utils - INFO - Waiting for instance f6e5c6f9-dd81-491a-aab5-dad988fc42b7 to get a DHCP lease...<br/> 2017-03-31 15:55:13,490 - sndvpn_test_utils - ERROR - Instance f6e5c6f9-dd81-491a-aab5-dad988fc42b7 seems to have failed leasing an IP.<br/> 2017-03-31 15:55:13,491 - sndvpn_test_utils - INFO - Waiting for instance 431b3a75-2aad-40b4-8d5b-bc16cf29a1a0 to get a DHCP lease...<br/> ^CTraceback (most recent call last):<br/> File "./run_tests.py", line 107, in &lt;module&gt;<br/> main()<br/> File "./run_tests.py", line 78, in main<br/> result = t.main()<br/> File "/home/opnfv/repos/sdnvpn/sdnvpn/test/functest/testcase_8.py", line 122, in main<br/> instances_up = test_utils.wait_for_instances_up(vm_1, vm_2)<br/> File "/home/opnfv/repos/sdnvpn/sdnvpn/lib/utils.py", line 270, in wait_for_instances_up<br/> check = <span class="error">&#91;wait_for_instance(instance) for instance in args&#93;</span><br/> File "/home/opnfv/repos/sdnvpn/sdnvpn/lib/utils.py", line 259, in wait_for_instance</p> <p>The VM we are using does 3 dhcp request. That is around 120 seconds. All internal transport tunnels are created before already.</p> <p>When I login into the vm alter then 120 seconds and I do "ifup eth0" then I get directly a ip.</p> <p>Attached you find the flows from ovs when it is not working and when we have waited 2 minutes. The flow from the controller and the flows from the compute are both shown. What can be seen is that not even table0 contains the inport flow. One more interesting thing is either the dhcp request goes through directly (less than 10 seconds) or it needs this long time.</p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-589

stable/boron not usable in OPNFV-test framwork - DHCP timesout

Bug Resolved Done Vyshakh Krishnan Nikolas Hermanns Mon, 3 Apr 2017 14:41:29 +0000 Thu, 20 Apr 2017 04:51:13 +0000 Thu, 20 Apr 2017 04:51:13 +0000 Boron General 0 11 <p>Attachment odl-dhcp-issue.zip has been added with description: tables</p> <p>Hi Nikolas,</p> <p>Could you please let us know how many VMs are being spawned. Is it failing for all the VMs initially and works fine when ifup eth0 is done after 2 minutes.</p> <p>&gt;&gt;One more interesting thing is either the dhcp request goes through directly (less than 10 seconds) or it needs this long time. &lt;&lt;</p> <p>Do you mean to say that DHCP request is actually being send within 10 secs and not within 120 secs as we expect?</p> <p>Hey Kency,</p> <p>thanks for looking into this.</p> <p>So I spawn 2 VMs both have the same issue is a probability from around 70%. Any additional VM has the same issue.</p> <p>If in the rare cases that the flows are pushed directly the DHCP request gets answered directly then I see that the VM is botted up in about 10 seconds.</p> <p>So 2 in short cases:<br/> 1. <span class="error">&#91;broken YANGTOOLS-14% of the times&#93;</span> No connectivity at all for the first ~2 minutest<br/> 2. <span class="error">&#91;working CONTROLLER-24% of the times&#93;</span> Direct connectivity and through that the DHCP request is answered directly.</p> <p>You see in the attachment that I added 2 times all the flow tables. That is both times from 1. The not working one is directly after the VM is spawned. The working on is after ~2 mins.</p> <p>Thanks! Nikolas</p> <p>I have a new finding!</p> <p>in ovs logs I can see a lot of this messages:<br/> 2017-04-05T10:32:59.834Z|00621|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:32:59.834Z|00622|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:32:59.924Z|00623|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.002Z|00624|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.013Z|00625|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.013Z|00626|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.014Z|00627|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.014Z|00628|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.026Z|00629|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.026Z|00630|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.045Z|00631|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.045Z|00632|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:00.045Z|00633|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPGMFC_GROUP_EXISTS error reply to OFPT_GROUP_MOD message<br/> 2017-04-05T10:33:10.045Z|00634|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: 316 flow_mods in the 9 s starting 10 s ago (316 adds)<br/> 2017-04-05T10:33:10.187Z|00635|connmgr|INFO|Dropped 8 log messages in last 11 seconds (most recently, 7 seconds ago) due to excessive rate<br/> 2017-04-05T10:33:10.187Z|00636|connmgr|INFO|br-int&lt;-&gt;tcp:192.0.2.7:6653: sending OFPBAC_BAD_OUT_GROUP error reply to OFPT_FLOW_MOD message</p> <p>This is related to this bug:<br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8132" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8132</a></p> <p>After a vm is delete groups and flows are not cleanup.</p> <p>On a fresh deployed system with a clean odl and a clean ovs everything seems to be working. When creating and removeing vms and networks after a while we get a group entry in ovs which is bad:<br/> group_id=210001,type=all<br/> in ODL:<br/> "buckets": {<br/> "bucket": [<br/> {<br/> "action": [<br/> {<br/> "openflowplugin-extension-nicira-action:nx-resubmit": </p> { "table": 55 } <p>,<br/> "order": 1<br/> },<br/> {<br/> "order": 0,<br/> "set-field": {<br/> "tunnel": </p> { "tunnel-id": 1 } <p> }<br/> }<br/> ],<br/> "bucket-id": 0,<br/> "watch_group": 4294967295,<br/> "watch_port": 4294967295,<br/> "weight": 0<br/> }<br/> ]<br/> },<br/> "group-id": 210001,<br/> "group-name": "3c94c809-663c-4ae0-9313-706d1aaf8310",<br/> "group-type": "group-all"</p> <p>So the group is not correctly synced!</p> <p>So I think we have still 2 issues here:<br/> 1. no cleanup when VMs, networks and bgpvpns are delete<br/> 2. a group cannot be modified.</p> <p>Just to state that one more time:<br/> in ovs the group is really empty!</p> <p>OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):<br/> group_id=210002,type=all,bucket=actions=group:210001,bucket=actions=load:0x200-&gt;NXM_NX_REG6[],resubmit(,220)<br/> group_id=150004,type=all,bucket=actions=set_field:fa:16:3e:12:a0:e5-&gt;eth_src,set_field:fa:16:3e:0a:c4:e1-&gt;eth_dst,load:0x100-&gt;NXM_NX_REG6[],resubmit(,220)<br/> group_id=5000,type=all,bucket=actions=CONTROLLER:65535,bucket=actions=resubmit(,17),bucket=actions=resubmit(,81)<br/> group_id=200002,type=all,bucket=actions=output:48<br/> group_id=210001,type=all<br/> 48(tun69773c7a1b0): addr:7a:48:85:85:fe:f9<br/> 49(tun441d8835c6b): addr:42:d9:3b:ca:c8:98<br/> 50(tape83f7b84-88): addr:fe:16:3e:0a:c4:e1<br/> LOCAL(br-int): addr:e2:4a:ad:35:bc:47</p> <p>check here the group_id=210001 it does not say anything behind. In odl it says:<br/> "openflowplugin-extension-nicira-action:nx-resubmit": {<br/> "table": 55<br/> },<br/> which means resubmit it to table 55 from where it is then matching on dhcp port and then send to the controller (Neutron dhcp).</p> <p>In tables_not_working log, I don't see any flows/groups for neutron VMs present in both compute nodes and seeing flows/groups only for transparent port created on the DPN for the flat provider network. Looks like there is no VMs spawned. At this stage, group_id=210001 is empty because of no VMs.</p> <p>What do you see in odl for group id 210001 ? Does it exist in inventory config ds or not ?</p> <p>when you add new VM back into this network, Are you seeing only ELAN Local BC group (210001) not updated with bucket? what do you see in wireshark trace ? is it group_mod or group-add ? what about other flows for the VM ?</p> <p>This is what I see in odl as already written in the comment above:<br/> in ODL:<br/> "buckets": {<br/> "bucket": [<br/> {<br/> "action": [<br/> {<br/> "openflowplugin-extension-nicira-action:nx-resubmit": </p> { "table": 55 } <p>,<br/> "order": 1<br/> },<br/> {<br/> "order": 0,<br/> "set-field": {<br/> "tunnel": </p> { "tunnel-id": 1 } <p> }<br/> }<br/> ],<br/> "bucket-id": 0,<br/> "watch_group": 4294967295,<br/> "watch_port": 4294967295,<br/> "weight": 0<br/> }<br/> ]<br/> },<br/> "group-id": 210001,<br/> "group-name": "3c94c809-663c-4ae0-9313-706d1aaf8310",<br/> "group-type": "group-all"</p> <p>Have in mind that I reproduced the error few times and what you do see here is not the original issue when the bug was found.</p> <p>So very short before the vm is stated I see the following in ovs:</p> <p>3 computes:<br/> OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):<br/> group_id=200000,type=all,bucket=actions=set_field:00:72:2b:16:87:6c-&gt;eth_dst,load:0x300-&gt;NXM_NX_REG6[],resubmit(,220)<br/> group_id=5000,type=all,bucket=actions=CONTROLLER:65535,bucket=actions=resubmit(,17),bucket=actions=resubmit(,81)<br/> group_id=210002,type=all,bucket=actions=group:210001,bucket=actions=load:0x300-&gt;NXM_NX_REG6[],resubmit(,220)<br/> group_id=210001,type=all<br/> group_id=150010,type=all,bucket=actions=set_field:fa:16:3e:33:a2:cc-&gt;eth_src,set_field:fa:16:3e:51:eb:2d-&gt;eth_dst,load:0x1000-&gt;NXM_NX_REG6[],resubmit(,220)<br/> group_id=150003,type=all,bucket=actions=set_field:fe:16:3e:0c:92:f0-&gt;eth_src,set_field:fa:16:3e:0c:92:f0-&gt;eth_dst,load:0x500-&gt;NXM_NX_REG6[],resubmit(,220)</p> <p>in odl:<br/> {<br/> "buckets": {<br/> "bucket": [<br/> {<br/> "action": [<br/> {<br/> "openflowplugin-extension-nicira-action:nx-resubmit": </p> { "table": 220 } <p>,<br/> "order": 2<br/> },<br/> {<br/> "openflowplugin-extension-nicira-action:nx-reg-load": {<br/> "dst": </p> { "end": 31, "nx-reg": "nicira-match:nxm-nx-reg6", "start": 0 } <p>,<br/> "value": 768<br/> },<br/> "order": 1<br/> },<br/> {<br/> "order": 0,<br/> "set-field": {<br/> "ethernet-match": {<br/> "ethernet-destination": </p> { "address": "00:72:2b:16:87:6c" } <p> }<br/> }<br/> }<br/> ],<br/> "bucket-id": 0,<br/> "watch_group": 4294967295,<br/> "watch_port": 4294967295,<br/> "weight": 0<br/> }<br/> ]<br/> },<br/> "group-id": 200000,<br/> "group-name": "7e763a79-bf3f-409a-80c7-6ea478b783eb",<br/> "group-type": "group-all"<br/> },<br/> {<br/> "buckets": {},<br/> "group-id": 210001,<br/> "group-name": "271c0841-0427-409f-b37c-94aea5175748",<br/> "group-type": "group-all"<br/> },</p> <p>see full inventory in logs.</p> <p>Attachment full-inventory.zip has been added with description: full-inventory before vm is booted</p> <p>group_id=210002,type=all,bucket=actions=group:210001,bucket=actions=load:0x300-&gt;NXM_NX_REG6[],resubmit(,220)<br/> group_id=210001,type=all</p> <p>It looks like the above groups are created for ELAN instance for flat provider network type and doesn't have any VMs in it. <br/> But when add VM, group_mod is sent for 210001 with a bucket to OVS, but not sure why OVS throws OFPGMFC_GROUP_EXISTS error. This should be thrown only for group add OFPGC_ADD request and not for OFPGC_MODIFY.</p> <p>Can we look at tcpdump to know what kind of group request is sent ?</p> <p>Reproduced again this time fetching logs and thread dump while and after hanging odl:</p> <p><a href="https://drive.google.com/open?id=0B_Rr7XjF0yoHc2EwT1psVGQwRVE" class="external-link" target="_blank" rel="nofollow noopener">https://drive.google.com/open?id=0B_Rr7XjF0yoHc2EwT1psVGQwRVE</a></p> <p>I see there are 6 threads which blocked in BgpConfigurationManager while advertising route to bgp. Looks like there is an issue with establishing neighbour with bgp peer. This causes a thread which invokes BgpConfigurationManager#replay holds bgpconfigmgr's instrinsic lock for longer time, eventually other threads trying to advertise routes, etc. are blocked on this lock.<br/> This makes around 7 threads to be unused by ODL (DjC, DCN notification etc.) and it leads to 2 mins hang and flows are not programmed in switches (in the attached log, table 17 flow rules are not programmed for some of the VMs)</p> <p>We need to address the following.</p> <p>1. Need to resolve establishing BGP neighbor issue. in karaf.log, i see lot of errors like: <br/> 2017-04-06 14:54:52,403 | ERROR | pool-46-thread-1 | BgpConfigurationManager | 318 - org.opendaylight.netvirt.bgpmanager-impl - 0.3.3.SNAPSHOT | Replay:startBgp() received exception: "org.apache.thrift.transport.TTransportException" </p> <p>Is there any issue with setting up 6wind quagga with ODL ? can you look into it ?<br/> If you resolve this, we may not see the thread blocking issue.</p> <p>2. Holding lock bgpconfigmgr's intrinsic lock for all method invocations is incorrect. This will make system to be unusable at this situation. It has to be addressed. Suneelu/Siva, Can you have a look ?</p> <p>Hey,</p> <p>godd finding! yes quagga bgp is not working yet on this machine. This was a bug raised internally for OPNFV. I will put more effort on this bug then now. You are right we need to remove this sync in addVRF. Can we still pull that in SR-3?</p> <p>Br Nikolas</p> <p>just for more information!</p> <ol> <li>cat overcloud-controller-0.odl.thread.dump |grep BL<br/> OCKED -B 1 -A 6 <br/> "ForkJoinPool-1-worker-0" #1072 daemon prio=5 os_prio=0 tid=0x00007f3fb4853000 nid=0x46521 waiting for monitor entry<br/> <span class="error">&#91;0x00007f3f34568000&#93;</span><br/> java.lang.Thread.State: BLOCKED (on object monitor)<br/> at org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager.addPrefix(BgpConfigurationManager.java:1918)</li> </ol> <ul class="alternate" type="square"> <li>waiting to lock &lt;0x000000008e1fe548&gt; (a org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager)<br/> at org.opendaylight.netvirt.bgpmanager.BgpManager.advertisePrefix(BgpManager.java:122)<br/> at Proxyf51ee27f_be1b_4724_a075_2d313a52fe71.advertisePrefix(Unknown Source)<br/> at Proxy4be9080d_62c9_4555_b8c8_5c3f04181a72.advertisePrefix(Unknown Source)<br/> at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.addPrefixToBGP(VpnInterfaceManager.java:965)<br/> &#8211;<br/> "ForkJoinPool-1-worker-2" #1070 daemon prio=5 os_prio=0 tid=0x00007f3fb4856800 nid=0x46520 waiting for monitor entry<br/> <span class="error">&#91;0x00007f3f0c347000&#93;</span><br/> java.lang.Thread.State: BLOCKED (on object monitor)<br/> at org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager.addPrefix(BgpConfigurationManager.java:1918)</li> <li>waiting to lock &lt;0x000000008e1fe548&gt; (a org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager)<br/> at org.opendaylight.netvirt.bgpmanager.BgpManager.advertisePrefix(BgpManager.java:122)<br/> at Proxyf51ee27f_be1b_4724_a075_2d313a52fe71.advertisePrefix(Unknown Source)<br/> at Proxy4be9080d_62c9_4555_b8c8_5c3f04181a72.advertisePrefix(Unknown Source)<br/> at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.addPrefixToBGP(VpnInterfaceManager.java:965)<br/> &#8211;<br/> "ForkJoinPool-1-worker-3" #1071 daemon prio=5 os_prio=0 tid=0x00007f3f78206000 nid=0x4651f waiting for monitor entry<br/> <span class="error">&#91;0x00007f3f1bdfe000&#93;</span><br/> java.lang.Thread.State: BLOCKED (on object monitor)<br/> at org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager.addPrefix(BgpConfigurationManager.java:1918)</li> <li>waiting to lock &lt;0x000000008e1fe548&gt; (a org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager)<br/> at org.opendaylight.netvirt.bgpmanager.BgpManager.advertisePrefix(BgpManager.java:122)<br/> at Proxyf51ee27f_be1b_4724_a075_2d313a52fe71.advertisePrefix(Unknown Source)<br/> at Proxy4be9080d_62c9_4555_b8c8_5c3f04181a72.advertisePrefix(Unknown Source)<br/> at org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.addPrefixToBGP(VpnInterfaceManager.java:965)</li> </ul> <p>I checked. if effects master branch as well in the same amount.</p> <p>Vyshakh, Can you get <a href="https://git.opendaylight.org/gerrit/#/c/54578/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54578/</a> merged ? Also it has to be cherry picked into stable/boron.</p> Development External issue ID 8142 External issue URL Priority Rank 0|i01sqf: