[NETVIRT-1599] Upgrade Failures:Connectivity check fails after upgrade and ovsdb egress flows missing Created: 07/May/19  Updated: 04/Jun/20  Resolved: 04/Jun/20

Status: Verified
Project: netvirt
Component/s: None
Affects Version/s: None
Fix Version/s: Magnesium

Type: Bug Priority: Medium
Reporter: Jaya Priyadarshini Assignee: Srinivas Rachakonda
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Upgrade Failures:Connectivity check fails after upfrade and ovdb egress flows missing

Logs:

All the logs from this job

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-neon/

 Comments   
Comment by Srinivas Rachakonda [ 21/Jan/20 ]

Hi,

Steps to reproduce:

  1. Stop ODL.
  2. Disconnect OVS
  3. Wipe ODL Local data: data, journal, snapshot
  4. Start ODL and get full sync.
  5. Set upgrade flag.
  6. Connect back OVS to ODL <<< errors seen in ovs-vswitchd.log

 

Error messages:

 

2020-01-20T13:32:26.913Z|00101|connmgr|INFO|br-int: removed primary controller "tcp:192.168.56.105:6653"

2020-01-20T13:38:44.387Z|00102|connmgr|INFO|br-int: added primary controller "tcp:192.168.56.105:6653"

2020-01-20T13:38:44.387Z|00103|rconn|INFO|br-int<->tcp:192.168.56.105:6653: connecting...

2020-01-20T13:38:44.480Z|00104|rconn|INFO|br-int<->tcp:192.168.56.105:6653: connected

2020-01-20T13:38:44.757Z|00105|connmgr|INFO|br-int<->tcp:192.168.56.105:6653: sending OFPBFC_BAD_ID error reply to ONFT_BUNDLE_CONTROL message

2020-01-20T13:38:57.418Z|00106|connmgr|INFO|br-int<->tcp:192.168.56.105:6653: sending OFPBFC_TIMEOUT error reply to ONFT_BUNDLE_CONTROL message

 

 

Thanks,

Srinivas

Comment by dheenadayalan b [ 28/Jan/20 ]

As per the analysis, Code fix is not required for this change only configuration change required at switch end. Please increase bundle-idle-timeout to  minimum of 3600 seconds (60 minutes) on switch, and please verify  and confirm, if bundle commit after upgrading is successful.

switch command:

sudo ovs-vsctl set Open_vSwitch . other_config:bundle-idle-timeout=3600

--Dheena

Comment by Srinivas Rachakonda [ 30/Jan/20 ]

The suite is still failing with the changes made.

Below are the steps performed:

 

  1. Stop ODL.
  2. Disconnect OVS
  3. Wipe Local data : data, journal, snapshots
  4. Start ODL
  5. Set Upgrade Flag
  6. Connect OVS
  7. Get Active Bundle: /restconf/operations/arbitrator-reconcile:get-active-bundle
  8. Commit Active bundle: /restconf/operations/arbitrator-reconcile:commit-active-bundle
  9. Check for flows on OVS <<<<<<<<<<<<< Fail as no flows are seen.

 

JOB:

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-sodium/12/

 

ROBOT LOGS:

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-sodium/12/robot-plugin/log_full.html.gz

 

Update the same In JIRA.

Comment by dheenadayalan b [ 01/Jun/20 ]

During upgrade, Bundle messages are being pushed to switch. However, when same group
added with different flows in the switch resulted in OFPGMFC_GROUP_EXIST error and Bundle commit failed due to this. Modified flag <group-add-mod-enabled>true</group-add-mod-enabled> in \etc\opendaylight\datastore\initial\config\default-openflow-switch-connection-config.xml and issue is resolved. Same steps need to incorporated in script and need to be reverted once test case is successful.

Comment by Srinivas Rachakonda [ 04/Jun/20 ]

The script is fixed with suggestions from design.

Comment by Srinivas Rachakonda [ 04/Jun/20 ]

Passed in Magnesium.

https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-magnesium/262/

Generated at Wed Feb 07 20:24:28 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.