Race-like behavior in programming of OpenStack instance flows causes flood+egress rules to not being programmed

Scenario:

creating 2 instances of same tenant across 2 nodes. Using ML2->ODL with GRE tunnels. About 1/2 of the times, the flows that need to be programmed to connect the 2 instances is not performed. Removing and re-creating the instance is a way to work around this issue.

Logs when flow does NOT get programmed:

osgi> 2014-06-11 17:18:23.299 EDT [pool-19-thread-1] WARN o.o.o.neutron.provider.OF10Provider - Tunnel Port gre-769-172.16.150.144 on node OVS|172.16.150.143:33558: OFPort = -1 . Don't panic. It might get converged soon...
2014-06-11 17:18:23.299 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - (FF) debug outside Table<?> row : intfs.values()
2014-06-11 17:18:23.346 EDT [pool-19-thread-1] WARN o.o.o.neutron.provider.OF10Provider - Tunnel Port gre-769-172.16.150.143 on node OVS|172.16.150.144:45945: OFPort = -1 . Don't panic. It might get converged soon...
2014-06-11 17:18:23.346 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - (FF) debug outside Table<?> row : intfs.values()
2014-06-11 17:18:23.348 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Identified Tunnel port gre-769-172.16.150.144 -> OF (4) on OVS|172.16.150.143:33558
2014-06-11 17:18:23.353 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Programmed ingress tunnel and flodd rules for node OVS|172.16.150.143:33558 tunnel gre-769-172.16.150.144 tunnelOFPort 4 internalVlan 2 patchOFPort 1
2014-06-11 17:18:23.353 EDT [pool-19-thread-1] WARN o.o.o.neutron.provider.OF10Provider - Tunnel Port gre-769-172.16.150.143 on node OVS|172.16.150.144:45945: OFPort = -1 . Don't panic. It might get converged soon...
2014-06-11 17:18:23.353 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - (FF) debug outside Table<?> row : intfs.values()

Deleting partially populated instance:

osgi> 2014-06-11 17:19:23.886 EDT [pool-19-thread-1] INFO o.o.ovsdb.neutron.SouthboundHandler - Delete interface tap8f0c5f6f-be
2014-06-11 17:19:23.951 EDT [pool-19-thread-1] INFO o.o.ovsdb.neutron.SouthboundHandler - Delete interface gre-769-172.16.150.144
2014-06-11 17:19:23.951 EDT [pool-19-thread-1] INFO o.o.ovsdb.neutron.SouthboundHandler - Delete interface gre-769-172.16.150.143

Logs when flow does get properly programmed:

osgi> 2014-06-11 17:19:34.970 EDT [pool-19-thread-1] ERROR o.o.o.neutron.TenantNetworkManager - Unable to get an internalVlan for Network NeutronNetwork [networkUUID=e6bcc184-e438-4a52-99b7-0eed3f3867eb, networkName=pepsigre2, adminStateUp=true, shared=false, tenantID=2a4efe11ba414c429b4fdcec87bb7617, routerExternal=false, providerNetworkType=gre, providerPhysicalNetwork=null, providerSegmentationID=769, status=ACTIVE, subnets=[9e17cadc-454f-4597-b1a0-0075786b5ca9], myPorts=[NeutronPort [portUUID=3c91a822-6687-4140-9192-120615764195, networkUUID=e6bcc184-e438-4a52-99b7-0eed3f3867eb, name=, adminStateUp=true, status=null, macAddress=FA:16:3E:E9:E7:DC, fixedIPs=[org.opendaylight.controller.networkconfig.neutron.Neutron_IPs@5228b8e9], deviceID=22d439cd-8bc8-4702-b6bd-ae9968375edc, deviceOwner=network:router_interface, tenantID=79907b74e8e24eb7a77e15fbb7f59cdc, floatingIPMap={}, securityGroups=[]], NeutronPort [portUUID=6c06c2df-9573-457d-966f-3d91b58216a9, networkUUID=e6bcc184-e438-4a52-99b7-0eed3f3867eb, name=, adminStateUp=true, status=null, macAddress=FA:16:3E:2D:11:80, fixedIPs=[org.opendaylight.controller.networkconfig.neutron.Neutron_IPs@3550d33c], deviceID=cc7b6576-a827-4f91-9b14-746daf89b198, deviceOwner=compute:None, tenantID=2a4efe11ba414c429b4fdcec87bb7617, floatingIPMap={}, securityGroups=[NeutronSecurityGroup{securityGroupUUID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityGroupName='default', securityGroupDescription='default', securityGroupTenantID='2a4efe11ba414c429b4fdcec87bb7617', securityRules=[NeutronSecurityRule

{securityRuleUUID='3df65578-9f18-40b6-876b-7028d90e6fca', securityRuleDirection='egress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv6', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}, NeutronSecurityRule{securityRuleUUID='52b56b5a-ca55-4d5f-a058-27f2bf3703cd', securityRuleDirection='egress', securityRuleProtocol='tcp', securityRulePortMin=22, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='66c4d22f-4e77-49af-9b14-9103bc0fa593', securityRuleDirection='ingress', securityRuleProtocol='icmp', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='8ce9f4b9-2583-496f-b217-72d82dae6141', securityRuleDirection='egress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}, NeutronSecurityRule{securityRuleUUID='9c9b7f83-a864-4302-8d07-245b7e4c6633', securityRuleDirection='ingress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv6', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}, NeutronSecurityRule{securityRuleUUID='c0218615-3ef2-444e-b274-b06fb51c37eb', securityRuleDirection='egress', securityRuleProtocol='icmp', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='e502dacc-363f-479f-a71d-01714757e945', securityRuleDirection='ingress', securityRuleProtocol='tcp', securityRulePortMin=22, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='f8a48bd3-534f-427c-be10-4a13ec1c08af', securityRuleDirection='ingress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}]]]], NeutronPort [portUUID=dbecc03a-e4c3-4227-99fa-330d6abeacc8, networkUUID=e6bcc184-e438-4a52-99b7-0eed3f3867eb, name=, adminStateUp=true, status=null, macAddress=FA:16:3E:A0:67:6D, fixedIPs=[org.opendaylight.controller.networkconfig.neutron.Neutron_IPs@4d5c237c], deviceID=dhcp2c92e20d-e8d5-5ad2-afca-c87bc946c73e-e6bcc184-e438-4a52-99b7-0eed3f3867eb, deviceOwner=network:dhcp, tenantID=2a4efe11ba414c429b4fdcec87bb7617, floatingIPMap={}, securityGroups=[]], NeutronPort [portUUID=8f0c5f6f-be6c-4e14-af0d-04bfe8fe8ffd, networkUUID=e6bcc184-e438-4a52-99b7-0eed3f3867eb, name=, adminStateUp=true, status=null, macAddress=FA:16:3E:99:EF:DC, fixedIPs=[org.opendaylight.controller.networkconfig.neutron.Neutron_IPs@3d79332c], deviceID=db6fd2c6-1862-4ee5-b09e-65a45734df21, deviceOwner=compute:None, tenantID=2a4efe11ba414c429b4fdcec87bb7617, floatingIPMap={}, securityGroups=[NeutronSecurityGroup{securityGroupUUID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityGroupName='default', securityGroupDescription='default', securityGroupTenantID='2a4efe11ba414c429b4fdcec87bb7617', securityRules=[NeutronSecurityRule{securityRuleUUID='3df65578-9f18-40b6-876b-7028d90e6fca', securityRuleDirection='egress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv6', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}

, NeutronSecurityRule

{securityRuleUUID='52b56b5a-ca55-4d5f-a058-27f2bf3703cd', securityRuleDirection='egress', securityRuleProtocol='tcp', securityRulePortMin=22, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='66c4d22f-4e77-49af-9b14-9103bc0fa593', securityRuleDirection='ingress', securityRuleProtocol='icmp', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='8ce9f4b9-2583-496f-b217-72d82dae6141', securityRuleDirection='egress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}, NeutronSecurityRule{securityRuleUUID='9c9b7f83-a864-4302-8d07-245b7e4c6633', securityRuleDirection='ingress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv6', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}, NeutronSecurityRule{securityRuleUUID='c0218615-3ef2-444e-b274-b06fb51c37eb', securityRuleDirection='egress', securityRuleProtocol='icmp', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='e502dacc-363f-479f-a71d-01714757e945', securityRuleDirection='ingress', securityRuleProtocol='tcp', securityRulePortMin=22, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}, NeutronSecurityRule{securityRuleUUID='f8a48bd3-534f-427c-be10-4a13ec1c08af', securityRuleDirection='ingress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}]]]], NeutronPort [portUUID=e2f823d1-3f30-4451-949b-92d5f86f4acf, networkUUID=e6bcc184-e438-4a52-99b7-0eed3f3867eb, name=, adminStateUp=true, status=null, macAddress=FA:16:3E:9A:D7:03, fixedIPs=[org.opendaylight.controller.networkconfig.neutron.Neutron_IPs@600ead3b], deviceID=f55eba60-137e-4728-95a4-507ced2d8a33, deviceOwner=compute:None, tenantID=2a4efe11ba414c429b4fdcec87bb7617, floatingIPMap={}, securityGroups=[NeutronSecurityGroup{securityGroupUUID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityGroupName='default', securityGroupDescription='default', securityGroupTenantID='2a4efe11ba414c429b4fdcec87bb7617', securityRules=[NeutronSecurityRule{securityRuleUUID='3df65578-9f18-40b6-876b-7028d90e6fca', securityRuleDirection='egress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv6', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}, NeutronSecurityRule{securityRuleUUID='52b56b5a-ca55-4d5f-a058-27f2bf3703cd', securityRuleDirection='egress', securityRuleProtocol='tcp', securityRulePortMin=22, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}

, NeutronSecurityRule

{securityRuleUUID='66c4d22f-4e77-49af-9b14-9103bc0fa593', securityRuleDirection='ingress', securityRuleProtocol='icmp', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}

, NeutronSecurityRule

{securityRuleUUID='8ce9f4b9-2583-496f-b217-72d82dae6141', securityRuleDirection='egress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}

, NeutronSecurityRule

{securityRuleUUID='9c9b7f83-a864-4302-8d07-245b7e4c6633', securityRuleDirection='ingress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv6', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}

, NeutronSecurityRule

{securityRuleUUID='c0218615-3ef2-444e-b274-b06fb51c37eb', securityRuleDirection='egress', securityRuleProtocol='icmp', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}

, NeutronSecurityRule

{securityRuleUUID='e502dacc-363f-479f-a71d-01714757e945', securityRuleDirection='ingress', securityRuleProtocol='tcp', securityRulePortMin=22, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=null, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='79907b74e8e24eb7a77e15fbb7f59cdc'}

, NeutronSecurityRule

{securityRuleUUID='f8a48bd3-534f-427c-be10-4a13ec1c08af', securityRuleDirection='ingress', securityRuleProtocol='null', securityRulePortMin=null, securityRulePortMax=null, securityRuleEthertype='IPv4', securityRuleRemoteIpPrefix='null', securityRemoteGroupID=425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d, securityRuleGroupID='425c9ae3-230b-4d6f-9d18-66fc2ecd0b5d', securityRuleTenantID='2a4efe11ba414c429b4fdcec87bb7617'}

]]]]]]
2014-06-11 17:19:35.033 EDT [pool-19-thread-1] WARN o.o.o.neutron.provider.OF10Provider - Could not Identify Tunnel port gre-769-172.16.150.144 on OVS|172.16.150.141:41895. Don't panic. It might get converged soon...
2014-06-11 17:19:35.033 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - (FF) debug outside Table<?> row : intfs.values()
2014-06-11 17:19:35.079 EDT [pool-19-thread-1] WARN o.o.o.neutron.provider.OF10Provider - Tunnel Port gre-769-172.16.150.141 on node OVS|172.16.150.144:45945: OFPort = -1 . Don't panic. It might get converged soon...
2014-06-11 17:19:35.079 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - (FF) debug outside Table<?> row : intfs.values()
2014-06-11 17:19:35.096 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Identified Tunnel port gre-769-172.16.150.144 -> OF (16) on OVS|172.16.150.141:41895
2014-06-11 17:19:35.202 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Programmed ingress tunnel and flodd rules for node OVS|172.16.150.141:41895 tunnel gre-769-172.16.150.144 tunnelOFPort 16 internalVlan 3 patchOFPort 1
2014-06-11 17:19:35.202 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Identified Tunnel port gre-769-172.16.150.141 -> OF (14) on OVS|172.16.150.144:45945
2014-06-11 17:19:35.206 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Programmed (non-local) EgressTunnelBridgeRules for attachedMac fa:16:3e:9a:d7:03 internalVlan 3 tunnelOFPort 14
2014-06-11 17:19:35.212 EDT [pool-19-thread-1] INFO o.o.o.neutron.provider.OF10Provider - Programmed ingress tunnel and flodd rules for node OVS|172.16.150.144:45945 tunnel gre-769-172.16.150.141 tunnelOFPort 14 internalVlan 3 patchOFPort 1