L3 VMs sometimes unreachable because TunnelStateChangeListener can't read DPNTEPsInfo

The flow :

• Setup: ODL + CONTROL node + 2 compute nodes
• Creating ext-net and private networks
• Creating a router
• Connecting the router to networks
• Launch 2 vm’s

The result: sporadic failure while installing T21 rules for one of VM’s .

It seems to be side-effect of the review [1] where storing ITM models to DS is replaced to batching utilities instead of individual transaction submit
Please review and decide if it’s possible to remove using BatchingUtils for updating DPNTEPsInfo, like it is done for InternalTunnels list [2]

[1] https://git.opendaylight.org/gerrit/#/c/52059/3/itm/itm-impl/src/main/java/org/opendaylight/genius/itm/confighelpers/ItmInternalTunnelAddWorker.java
[2] https://git.opendaylight.org/gerrit/#/c/53963/3

Please see below detailed analysis according to the attached log:

2017-03-29 11:41:56,940 | DEBUG | nPool-1-worker-2 | ItmInternalTunnelAddWorker | 312 - org.opendaylight.genius.itm-impl - 0.1.3.SNAPSHOT |
Creating Trunk Interface with parameters trunk I/f Name - tun0da28bcfe11, parent I/f name - 233992411000049:tunnel_port:0, source IP - IpAddress [_ipv4Address=Ipv4Address [_value=10.0.77.228]], destination IP - IpAddress [_ipv4Address=Ipv4Address [_value=10.0.97.144]] gateway IP - IpAddress [_ipv4Address=Ipv4Address [_value=0.0.0.0]]

2017-03-29 11:41:56,941 | DEBUG | nPool-1-worker-2 | ItmInternalTunnelAddWorker | 312 - org.opendaylight.genius.itm-impl - 0.1.3.SNAPSHOT |
Updating CONFIGURATION datastore with DPN DPNTEPsInfo [_dPNID=216456726482219, _key=DPNTEPsInfoKey [_dPNID=216456726482219], _tunnelEndPoints=[TunnelEndPoints [_gwIpAddress=IpAddress [_ipv4Address=Ipv4Address [_value=0.0.0.0]], _interfaceName=216456726482219:tunnel_port:0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.97.144]], _key=TunnelEndPointsKey [_portname=tunnel_port, _vLANID=0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.97.144]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan], _portname=tunnel_port, _subnetMask=IpPrefix [_ipv4Prefix=Ipv4Prefix [_value=0.0.0.0/0]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan, _tzMembership=[TzMembership [_key=TzMembershipKey [_zoneName=fe72e073-0b56-4c93-a432-7dd744c61894], _zoneName=fe72e073-0b56-4c93-a432-7dd744c61894, augmentation=[]]], _vLANID=0, augmentation=[]]], augmentation=[]]

2017-03-29 11:41:57,128 | TRACE | pool-47-thread-1 | TunnelInterfaceStateListener | 319 - org.opendaylight.netvirt.vpnmanager-impl - 0.3.3.SNAPSHOT |
Tunnel addition---- StateTunnelList{getDstInfo=DstInfo{getTepDeviceId=216456726482219, getTepDeviceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.TepTypeInternal, getTepIp=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.97.144]], augmentations={}}, getOperState=Up, getSrcInfo=SrcInfo{getTepDeviceId=233992411000049, getTepDeviceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.TepTypeInternal, getTepIp=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.77.228]], augmentations={}}, getTransportType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan, getTunnelInterfaceName=tun0da28bcfe11, isTunnelState=true, augmentations={}}

2017-03-29 11:41:57,128 | TRACE | pool-47-thread-1 | TunnelInterfaceStateListener | 319 - org.opendaylight.netvirt.vpnmanager-impl - 0.3.3.SNAPSHOT |
Handle tunnel event for srcDpn 233992411000049 SrcTepIp 10.0.77.228 DestTepIp 10.0.97.144

2017-03-29 11:41:57,130 | TRACE | nPool-1-worker-3 | VrfEntryListener | 321 - org.opendaylight.netvirt.fibmanager-impl - 0.3.3.SNAPSHOT |
manageRemoteRouteOnDPN :: action true, DpnId 233992411000049, vpnId 70001, rd fe72e073-0b56-4c93-a432-7dd744c61894, destPfx 10.0.120.12/32

2017-03-29 11:41:57,130 | WARN | nPool-1-worker-3 | NexthopManager | 321 - org.opendaylight.netvirt.fibmanager-impl - 0.3.3.SNAPSHOT | RPC Call to getTunnelInterfaceName returned with Errors []

2017-03-29 11:41:57,131 | WARN | nPool-1-worker-3 | VrfEntryListener | 321 - org.opendaylight.netvirt.fibmanager-impl - 0.3.3.SNAPSHOT | Failed to add Route: 10.0.120.12/32 in vpn: fe72e073-0b56-4c93-a432-7dd744c61894

2017-03-29 11:41:57,800 | DEBUG | nPool-1-worker-0 | ItmInternalTunnelAddWorker | 312 - org.opendaylight.genius.itm-impl - 0.1.3.SNAPSHOT |
Updating CONFIGURATION datastore with DPN DPNTEPsInfo [_dPNID=216456726482219, _key=DPNTEPsInfoKey [_dPNID=216456726482219], _tunnelEndPoints=[TunnelEndPoints [_gwIpAddress=IpAddress [_ipv4Address=Ipv4Address [_value=0.0.0.0]], _interfaceName=216456726482219:tunnel_port:0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.97.144]], _key=TunnelEndPointsKey [_portname=tunnel_port, _vLANID=0, _ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.97.144]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan], _portname=tunnel_port, _subnetMask=IpPrefix [_ipv4Prefix=Ipv4Prefix [_value=0.0.0.0/0]], _tunnelType=class org.opendaylight.yang.gen.v1.urn.opendaylight.genius.interfacemanager.rev160406.TunnelTypeVxlan, _tzMembership=[TzMembership [_key=TzMembershipKey [_zoneName=6f835afc-ed6a-49a2-83b9-589cdef548f8], _zoneName=6f835afc-ed6a-49a2-83b9-589cdef548f8, augmentation=[]]], _vLANID=0, augmentation=[]]], augmentation=[]]