Uploaded image for project: 'OpenFlowPlugin'
  1. OpenFlowPlugin
  2. OPNFLWPLUG-798

[GROUP RECONCILIATION] Group reconciliation fails during OVS restart

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Cannot Reproduce
    • None
    • None
    • General
    • None
    • Operating System: Linux
      Platform: All

    • 6933

    Description

      Test flow:

      • Setup with two compute nodes
      • To create one vm instance on each compute
      • Setting the floating ip for 2nd vm instance (allocated on d1)
      • Restart OVS in loop 10 times on d1, capture group table before and after the reset;

      Issue: Missing groups after restart .
      Аfter initial analysis looks there is a race between the external group installation (group table) before and after OVS reset.
      The screenshots showing the issue and log are attached.
      Note: happened twice out of 10 attempts.

      These are suspicious prints I can see in the karaf log, however these errors appear after successful reboots as well. Should it be cleaned up ?
      Do you see a difference between the unsuccessful reboot in (09:12:36) vs the successful ones (for example 09:12:40)?
      In both cases it seems like the group is "installed" even though the port is not up according to the log message - is there any way to actually verify this? Any traces to enable?

      2016-10-13 09:12:25,847 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Not yet received the node-connector updated for 3 for the group with id GroupId [_value=210000]
      2016-10-13 09:12:25,847 | INFO | n-dispatcher-333 | ControllerUpdateCommand | 293 - org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | Register ODL controllers : {} bridges detail : {}
      2016-10-13 09:12:25,847 | TRACE | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | awaitGroups() completed: node=openflow:132572872974439
      2016-10-13 09:12:25,847 | WARN | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Installing the group for node connector 3
      2016-10-13 09:12:25,847 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Not yet received the node-connector updated for 3 for the group with id GroupId [_value=210002]
      2016-10-13 09:12:25,848 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | These Groups are pointing to node-connectors that are not up yet [Group{getBuckets=Buckets{getBucket=[Bucket{getAction=[Action{getAction=GroupActionCase{getGroupAction=GroupAction{getGroupId=209999, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}], getBucketId=BucketId [_value=0], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}, Bucket{getAction=[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri [_value=3], augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Action{getAction=SetFieldCase{getSetField=SetField{getTunnel=Tunnel{getTunnelId=5000, augmentations={}}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], getBucketId=BucketId [_value=1], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}], augmentations={}}, getGroupId=GroupId [_value=210000], getGroupName=7a73adf3-f6b2-488e-b2b5-c3b7513fb9f0, getGroupType=GroupAll, augmentations={}}, Group{getBuckets=Buckets{getBucket=[Bucket{getAction=[Action{getAction=GroupActionCase{getGroupAction=GroupAction{getGroupId=210001, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}], getBucketId=BucketId [_value=0], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}, Bucket{getAction=[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri [_value=3], augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Action{getAction=SetFieldCase{getSetField=SetField{getTunnel=Tunnel{getTunnelId=5001, augmentations={}}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], getBucketId=BucketId [_value=1], getWatchGroup=4294967295, getWatchPort=4294967295, getWeight=0, augmentations={}}], augmentations={}}, getGroupId=GroupId [_value=210002], getGroupName=62344e07-e250-45e9-a778-34608ea6889b, getGroupType=GroupAll, augmentations={}}]
      2016-10-13 09:12:25,848 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Installing the group GroupId [_value=210000] finally although the port is not up after checking for 5 times
      2016-10-13 09:12:25,848 | ERROR | pool-45-thread-3 | FlowNodeReconciliationImpl | 342 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.4.0.SNAPSHOT | Installing the group GroupId [_value=210002] finally although the port is not up after checking for 5 times
      2016-10-13 09:12:25,857 | ERROR | pool-47-thread-1 | SubnetOpDpnManager | 357 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.0.SNAPSHOT | Cannot get, portOp for port 388b1e67-1f7d-4f1f-86b7-6ca596b88197 is not available in datastore

      Attachments

        1. Output1.PNG
          Output1.PNG
          332 kB
        2. Output2.PNG
          Output2.PNG
          357 kB
        3. screen-karaf.log.rar
          91 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Unassigned Unassigned
              olga.schukin@hpe.com Olga Schukin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: