Test flow:

• Setup with two compute nodes
• To create one vm instance on each compute
• Setting the floating ip for 2nd vm instance (allocated on d1)
• Restart OVS in loop 10 times on d1, capture group table before and after the reset;

Issue: Missing groups after restart .
Аfter initial analysis looks there is a race between the external group installation (group table) before and after OVS reset.
The screenshots showing the issue and log are attached.
Note: happened twice out of 10 attempts.

These are suspicious prints I can see in the karaf log, however these errors appear after successful reboots as well. Should it be cleaned up ?
Do you see a difference between the unsuccessful reboot in (09:12:36) vs the successful ones (for example 09:12:40)?
In both cases it seems like the group is "installed" even though the port is not up according to the log message - is there any way to actually verify this? Any traces to enable?

2016-10-13 09:12:25,847 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Not yet received the node-connector updated for 3 for the group with id GroupId [_value=210000]
2016-10-13 09:12:25,847 | INFO | n-dispatcher-333 | ControllerUpdateCommand | 293 - org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | Register ODL controllers : {} bridges detail : {}
2016-10-13 09:12:25,847 | TRACE | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | awaitGroups() completed: node=openflow:132572872974439
2016-10-13 09:12:25,847 | WARN | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Installing the group for node connector 3
2016-10-13 09:12:25,847 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Not yet received the node-connector updated for 3 for the group with id GroupId [_value=210002]
2016-10-13 09:12:25,848 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | These Groups are pointing to node-connectors that are not up yet [Group{getBuckets=Buckets{getBucket=[Bucket{getAction=[Action{getAction=GroupActionCase{getGroupAction=GroupAction{getGroupId=209999, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}], getBucketId=BucketId [_value=0], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}, Bucket{getAction=[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri [_value=3], augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Action{getAction=SetFieldCase{getSetField=SetField{getTunnel=Tunnel{getTunnelId=5000, augmentations={}}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], getBucketId=BucketId [_value=1], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}], augmentations={}}, getGroupId=GroupId [_value=210000], getGroupName=7a73adf3-f6b2-488e-b2b5-c3b7513fb9f0, getGroupType=GroupAll, augmentations={}}, Group{getBuckets=Buckets{getBucket=[Bucket{getAction=[Action{getAction=GroupActionCase{getGroupAction=GroupAction{getGroupId=210001, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}], getBucketId=BucketId [_value=0], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}, Bucket{getAction=[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri [_value=3], augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Action{getAction=SetFieldCase{getSetField=SetField{getTunnel=Tunnel{getTunnelId=5001, augmentations={}}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], getBucketId=BucketId [_value=1], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}], augmentations={}}, getGroupId=GroupId [_value=210002], getGroupName=62344e07-e250-45e9-a778-34608ea6889b, getGroupType=GroupAll, augmentations={}}]
2016-10-13 09:12:25,848 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Installing the group GroupId [_value=210000] finally although the port is not up after checking for 5 times
2016-10-13 09:12:25,848 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Installing the group GroupId [_value=210002] finally although the port is not up after checking for 5 times
2016-10-13 09:12:25,857 | ERROR | pool-47-thread-1 | SubnetOpDpnManager | 357 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.0.SNAPSHOT | Cannot get, portOp for port 388b1e67-1f7d-4f1f-86b7-6ca596b88197 is not available in datastore