stable/boron not usable in OPNFV-test framwork - DHCP timesout

Hey,

We see big issues in using stable/boron (nearly SR-3) in the OPNFV test pipeline. This is the test what is done:
2017-03-31 15:51:36,569 - openstack_utils - INFO - Creating neutron network sdnvpn-8-1...
2017-03-31 15:51:36,919 - openstack_utils - DEBUG - Network '130d6058-1321-4d4b-bc89-99c62e6dc971' created successfully
2017-03-31 15:51:36,919 - openstack_utils - DEBUG - Creating Subnet....
2017-03-31 15:51:37,166 - openstack_utils - DEBUG - Subnet '98a442fa-01d8-4241-a07d-af5dc22b7259' created successfully
2017-03-31 15:51:37,166 - openstack_utils - DEBUG - Creating Router...
2017-03-31 15:51:37,300 - openstack_utils - DEBUG - Router '80e53bf6-1426-46f8-9c2f-ee96d6efc644' created successfully
2017-03-31 15:51:37,301 - openstack_utils - DEBUG - Adding router to subnet...
2017-03-31 15:51:38,899 - openstack_utils - DEBUG - Interface added successfully.
2017-03-31 15:51:38,899 - openstack_utils - DEBUG - Adding gateway to router...
2017-03-31 15:51:40,276 - openstack_utils - DEBUG - Gateway added successfully.
2017-03-31 15:51:40,276 - sndvpn_test_utils - DEBUG - Creating network sdnvpn-8-2
2017-03-31 15:51:40,497 - sndvpn_test_utils - DEBUG - Creating subnet sdnvpn-8-2-subnet in network 773cd3bd-24a5-4886-9cb0-87fe7426c818 with cidr 10.10.20.0/24
2017-03-31 15:51:41,175 - openstack_utils - INFO - Creating security group 'sdnvpn-sg'...
2017-03-31 15:51:41,308 - openstack_utils - DEBUG - Security group 'sdnvpn-sg' with ID=65206fea-7be2-49b6-abce-b32c3c46a61a created successfully.
2017-03-31 15:51:41,309 - openstack_utils - DEBUG - Adding ICMP rules in security group 'sdnvpn-sg'...
2017-03-31 15:51:41,309 - openstack_utils - DEBUG - Security_group format set (no port range mentioned)
2017-03-31 15:51:41,455 - openstack_utils - DEBUG - Adding SSH rules in security group 'sdnvpn-sg'...
2017-03-31 15:51:41,456 - openstack_utils - DEBUG - Security_group format set (port range included)
2017-03-31 15:51:41,610 - openstack_utils - DEBUG - Security_group format set (port range included)
2017-03-31 15:51:41,773 - openstack_utils - DEBUG - Security_group format set (no port range mentioned) Neutron server returns request_ids: ['req-3afdd79b-09c5-41c0-a3d5-edc077407a21']
2017-03-31 15:51:41,872 - openstack_utils - ERROR - Bad security group format.One of the port range is not properly set:range min: 80,range max: None
2017-03-31 15:51:41,872 - sndvpn_test_utils - INFO - Creating instance 'sdnvpn-8-2'...
2017-03-31 15:51:41,873 - sndvpn_test_utils - DEBUG - Configuration:
name=sdnvpn-8-2
flavor=m1.tiny
image=1018ece1-b44c-4fef-857e-bb82f2a61f2f
network=773cd3bd-24a5-4886-9cb0-87fe7426c818
secgroup=65206fea-7be2-49b6-abce-b32c3c46a61a
hypervisor=
fixed_ip=None
files=None
userdata=
None

2017-03-31 15:51:41,889 - keystoneauth.identity.v2 - DEBUG - Making authentication request to http://192.168.37.10:5000/v2.0/tokens
2017-03-31 15:51:43,592 - keystoneauth.identity.v2 - DEBUG - Making authentication request to http://192.168.37.10:5000/v2.0/tokens
2017-03-31 15:51:51,896 - sndvpn_test_utils - DEBUG - Instance 'sdnvpn-8-2' booted successfully. IP='10.10.20.3'.
2017-03-31 15:51:51,896 - sndvpn_test_utils - DEBUG - Adding 'sdnvpn-8-2' to security group 'sdnvpn-sg'...
2017-03-31 15:51:51,905 - keystoneauth.identity.v2 - DEBUG - Making authentication request to http://192.168.37.10:5000/v2.0/tokens
2017-03-31 15:51:53,080 - sndvpn_test_utils - INFO - Creating instance 'sdnvpn-8-1'...
2017-03-31 15:51:53,081 - sndvpn_test_utils - DEBUG - Configuration:
name=sdnvpn-8-1
flavor=m1.tiny
image=1018ece1-b44c-4fef-857e-bb82f2a61f2f
network=130d6058-1321-4d4b-bc89-99c62e6dc971
secgroup=65206fea-7be2-49b6-abce-b32c3c46a61a
hypervisor=
fixed_ip=None
files=None
userdata=
#!/bin/sh
set 10.10.20.3
while true; do
for i do
ip=$i
ping -c 1 $ip 2>&1 >/dev/null
RES=$?
if [ "Z$RES" = "Z0" ] ; then
echo ping $ip OK
else echo ping $ip KO
fi
done
sleep 1
done

2017-03-31 15:51:53,095 - keystoneauth.identity.v2 - DEBUG - Making authentication request to http://192.168.37.10:5000/v2.0/tokens
2017-03-31 15:51:54,357 - keystoneauth.identity.v2 - DEBUG - Making authentication request to http://192.168.37.10:5000/v2.0/tokens
2017-03-31 15:52:02,684 - sndvpn_test_utils - DEBUG - Instance 'sdnvpn-8-1' booted successfully. IP='10.10.10.7'.
2017-03-31 15:52:02,684 - sndvpn_test_utils - DEBUG - Adding 'sdnvpn-8-1' to security group 'sdnvpn-sg'...
2017-03-31 15:52:03,493 - sdnvpn-results - INFO - Create VPN with eRT==iRT
2017-03-31 15:52:03,649 - sdnvpn-testcase-8 - DEBUG - VPN created details: {u'bgpvpn': {u'export_targets': [u'88:88'], u'name': u'sdnvpn-7', u'route_distinguishers': [u'18:18'], u'routers': [], u'import_targets': [u'88:88'], u'networks':
[], u'tenant_id': u'a94d4d97de9d4200ba9beca8d05c83d5', u'route_targets': [], u'project_id': u'a94d4d97de9d4200ba9beca8d05c83d5', u'type': u'l3', u'id': u'07c91f3f-df7d-4177-87c4-6320127359e8'}}
2017-03-31 15:52:03,649 - sdnvpn-results - INFO - Associate router 'sdnvpn-8-1-router' and net 'sdnvpn-8-2' to the VPN.
2017-03-31 15:52:04,965 - sndvpn_test_utils - DEBUG - Waiting for router 07c91f3f-df7d-4177-87c4-6320127359e8 to associate with BGPVPN 80e53bf6-1426-46f8-9c2f-ee96d6efc644
2017-03-31 15:52:06,129 - sndvpn_test_utils - DEBUG - Waiting for network 07c91f3f-df7d-4177-87c4-6320127359e8 to associate with BGPVPN 773cd3bd-24a5-4886-9cb0-87fe7426c818
2017-03-31 15:52:07,311 - sndvpn_test_utils - INFO - Waiting for instance f6e5c6f9-dd81-491a-aab5-dad988fc42b7 to get a DHCP lease...
2017-03-31 15:55:13,490 - sndvpn_test_utils - ERROR - Instance f6e5c6f9-dd81-491a-aab5-dad988fc42b7 seems to have failed leasing an IP.
2017-03-31 15:55:13,491 - sndvpn_test_utils - INFO - Waiting for instance 431b3a75-2aad-40b4-8d5b-bc16cf29a1a0 to get a DHCP lease...
^CTraceback (most recent call last):
File "./run_tests.py", line 107, in <module>
main()
File "./run_tests.py", line 78, in main
result = t.main()
File "/home/opnfv/repos/sdnvpn/sdnvpn/test/functest/testcase_8.py", line 122, in main
instances_up = test_utils.wait_for_instances_up(vm_1, vm_2)
File "/home/opnfv/repos/sdnvpn/sdnvpn/lib/utils.py", line 270, in wait_for_instances_up
check = [wait_for_instance(instance) for instance in args]
File "/home/opnfv/repos/sdnvpn/sdnvpn/lib/utils.py", line 259, in wait_for_instance

The VM we are using does 3 dhcp request. That is around 120 seconds. All internal transport tunnels are created before already.

When I login into the vm alter then 120 seconds and I do "ifup eth0" then I get directly a ip.

Attached you find the flows from ovs when it is not working and when we have waited 2 minutes. The flow from the controller and the flows from the compute are both shown. What can be seen is that not even table0 contains the inport flow. One more interesting thing is either the dhcp request goes through directly (less than 10 seconds) or it needs this long time.