[NETVIRT-1474] Race condition between default SNAT group and flow

Mon, 25 Nov 2019 07:56:25 +0000

Observed a problem where the default SNAT flow fails to add
because openflowplugin misses a group used on the flow that is added
just before the flow, which makes sense that it could happen as both
things are not synchronized in any way.

Relevant code is org.opendaylight.netvirt.natservice.internal.SNATDefaultRouteProgrammer#addOrDelDefaultFibRouteToSNATForSubnet

No idea why this is not happening in CSIT but happens to me in a devstack environment. But is related to timing.

Traces

2018-10-23T07:39:55,149 | INFO | jobcoordinator-main-task-0 |
ExternalNetworkGroupInstaller    | 345 -
org.opendaylight.netvirt.natservice-impl - 0.7.1.SNAPSHOT |
installExtNetGroupEntry : Installing ext-net group 225000 entry for
subnet 9832209c-aad6-4c76-aadc-23b4bca18e0e with macAddress null
(extInterface: 48479895235134:br-ex-patch:trunk)
2018-10-23T07:39:55,150 | INFO | jobcoordinator-main-task-0 |
RouterDpnChangeListener          | 345 -
org.opendaylight.netvirt.natservice-impl - 0.7.1.SNAPSHOT |
installDefaultNatRouteForRouterExternalSubnets : Installing default
routes in FIB on dpn 48479895235134 for subnetId Uuid{_value=9832209c-
aad6-4c76-aadc-23b4bca18e0e} with vpnId 100004
2018-10-23T07:39:55,179 | INFO | jobcoordinator-main-task-0 |
RouterDpnChangeListener          | 345 -
org.opendaylight.netvirt.natservice-impl - 0.7.1.SNAPSHOT | add :
Installing default route in FIB on dpn 48479895235134 for router
04b29abd-f633-4c15-87e7-80481e766eb0 with vpn 100000
2018-10-23T07:39:55,179 | INFO | jobcoordinator-main-task-0 |
RouterDpnChangeListener          | 345 -
org.opendaylight.netvirt.natservice-impl - 0.7.1.SNAPSHOT | add : SNAT
enabled for router 04b29abd-f633-4c15-87e7-80481e766eb0
2018-10-23T07:39:55,206 | INFO | nc-exe-4         |
FlowForwarder                    | 367 -
org.opendaylight.openflowplugin.applications.forwardingrules-manager -
0.7.1.SNAPSHOT | Reading the group from config inventory: 225000
2018-10-23T07:39:55,206 | ERROR | nc-exe-4         |
FlowForwarder                    | 367 -
org.opendaylight.openflowplugin.applications.forwardingrules-manager -
0.7.1.SNAPSHOT | Flow add with id SNAT.48479895235134.21.0.0.0.0.100004
failed for node Uri{_value=openflow:48479895235134} with error
[RpcError [message=Group 225000 not present in the config inventory,
severity=ERROR, errorType=APPLICATION, tag=operation-failed,
applicationTag=null, info=null, cause=null]]

OpenDaylight JIRA

[NETVIRT-1474] Race condition between default SNAT group and flow