Flow installation in OVS switch causes ofplugin regression failures in jenkins

The latest regression run reports failure in normal flow installation. Flows are pushed via Bulkomatic and verified in mininet switch.

GERRIT_BRANCH: stable/boron
GERRIT_REFSPEC: refs/heads/stable/boron
BRANCH: stable/boron
CONTROLLERFEATURES: odl-jolokia,odl-openflowplugin-flow-services-ui,odl-openflowplugin-app-bulk-o-matic

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/70/robot/

This issue is seen from last 3 days consistently in regression (boron + Lithium plugin), below given the console logs, one can observe out of 3 DPN in one of the DPN 1000 flows are not installed.

==============================================================================
openflowplugin-bulkomatic-clustering-daily.txt.Cluster Reconcilliation Mult...
==============================================================================
Check Shards Status And Initialize Variables :: Check Status for a... | PASS |
------------------------------------------------------------------------------
Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS |
------------------------------------------------------------------------------
Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS |
------------------------------------------------------------------------------
Add Bulk Flow From Follower :: 3000 Flows (1K per DPN) in 3 DPN ad... | PASS |
------------------------------------------------------------------------------
Get Bulk Flows and Verify In Cluster :: Initiate get operation and... | PASS |
------------------------------------------------------------------------------
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... | FAIL |
Keyword 'MininetKeywords.Mininet Sync Status' failed after retrying for 1 minute 40 seconds. The last error was: '*** s1 ------------------------------------------------------------------------
OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=0

• • • s2 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s3 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      mininet>' contains 'flow_count=1000' 2 times, not 3 times.
      ------------------------------------------------------------------------------

Also the same issue is observed in 2 node cluster suite, steps given as below

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/70/console

==============================================================================
openflowplugin-bulkomatic-clustering-daily.txt.Cluster Reconcilliation Mult...
==============================================================================
Check Shards Status And Initialize Variables :: Check Status for a... | PASS |
------------------------------------------------------------------------------
Get Inventory Follower Before Cluster Restart :: Find a follower i... | PASS |
------------------------------------------------------------------------------
Start Mininet Connect To Follower Node1 :: Start mininet with conn... | PASS |
------------------------------------------------------------------------------
Add Bulk Flow From Follower :: 3000 Flows (1K per DPN) in 3 DPN ad... | PASS |
------------------------------------------------------------------------------
Get Bulk Flows and Verify In Cluster :: Initiate get operation and... | PASS |
------------------------------------------------------------------------------
Verify Flows In Switch Before Cluster Restart :: Verify flows are ... | FAIL |
Keyword 'MininetKeywords.Mininet Sync Status' failed after retrying for 1 minute 40 seconds. The last error was: '*** s1 ------------------------------------------------------------------------
OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=0

• • • s2 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s3 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      mininet>' contains 'flow_count=1000' 2 times, not 3 times.
      ------------------------------------------------------------------------------

Thanks
Sanjib