OVS Nodes Connect Quickly, Not All Get TransportZone TEPs

When OVS nodes connect quickly, one after another, some nodes will not get all the TEPs. The cause of this is as follows:

In org.opendaylight.genius.itm.confighelpers.ItmTepAddWorker we have the following code:

this.meshedDpnList = dpnTEPsInfoCache.getAllPresent();
LOG.debug("Invoking Internal Tunnel build method with Configured DpnList {} ; Meshed DpnList {} ",
cfgdDpnList, meshedDpnList);
futures.addAll(itmInternalTunnelAddWorker.buildAllTunnels(mdsalManager, cfgdDpnList,
meshedDpnList, itmConfig)) ;

Note that "meshedDpnList" is retrieved from a cache. This cache is a subclass of mdsalutil's DataObjectCache which waits for notifications of type DpnTEPsInfo and caches them in memory. What I observed is that if the switches connect in quick succession, sometimes that cache is not updated when the code above is executed, resulting in that new switch never getting the TEP to connect to the missing DPN.

Consider the following two log lines, separated by less than 0.1 of a second:

2018-02-22T09:35:57,918 | DEBUG | ForkJoinPool-1-worker-3 | ItmInternalTunnelAddWorker | 176 - org.opendaylight.genius.itm-impl - 0.4.0.SNAPSHOT | Updating CONFIGURATION datastore with DPN DPNTEPsInfo [_dPNID=37528269029441, _key=DPNTEPsInfoKey [_dPNID=37528269029441], _tunnelEndPoints=[TunnelEndPoints [_gwIpAddress=IpAddress [_ipv4Address=Ipv4Address [_value=0.0.0.0]], _interfaceName=37528269029441::0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=172.17.0.2]], _key=TunnelEndPointsKey [_portname=, _vLANID=0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=172.17.0.2]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan], _optionTunnelTos=0, _portname=, _subnetMask=IpPrefix [_ipv4Prefix=Ipv4Prefix [_value=255.255.255.255/32]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan, _tzMembership=[TzMembership [_key=TzMembershipKey [_zoneName=default-transport-zone], _zoneName=default-transport-zone, augmentation=[]]], _vLANID=0, _optionOfTunnel=false, augmentation=[]]], augmentation=[]]

And

2018-02-22T09:35:57,987 | DEBUG | ForkJoinPool-1-worker-3 | ItmTepAddWorker | 176 - org.opendaylight.genius.itm-impl - 0.4.0.SNAPSHOT | Invoking Internal Tunnel build method with Configured DpnList [DPNTEPsInfo [_dPNID=116901406912847, _key=DPNTEPsInfoKey [_dPNID=116901406912847], _tunnelEndPoints=[TunnelEndPoints [_gwIpAddress=IpAddress [_ipv4Address=Ipv4Address [_value=0.0.0.0]], _interfaceName=116901406912847::0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=172.17.0.3]], _key=TunnelEndPointsKey [_portname=, _vLANID=0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=172.17.0.3]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan], _optionTunnelTos=0, _portname=, _subnetMask=IpPrefix [_ipv4Prefix=Ipv4Prefix [_value=255.255.255.255/32]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan, _tzMembership=[TzMembership [_key=TzMembershipKey [_zoneName=default-transport-zone], _zoneName=default-transport-zone, augmentation=[]]], _vLANID=0, _optionOfTunnel=false, augmentation=[]]], augmentation=[]]] ; Meshed DpnList []

Note that the DpnTEPsInfo for 172.17.0.2 is created in the first line but is absent in the meshed list on the second.