Operating System: All
Platform: All

Hi Itzik,

I have a couple of questions:

1) Do you use packstack or devstack to deploy openstack& ODL?
2) How do you restart Openstack after server reboot?

Thanks, Vinh

Devstack is used as development setup and it never recovers completely after node reboot. The issue is invalid if devstack is used.

Will try with packstack

However, I think it is an Openstack deployment platform issue, not ODL

hi,

I used the Ocata Official instalaltion guide to install every package required for Openstack and configured them to use ODL. Then restarted the Control node. When the System was up again, I see that the qdhcp NS getting created but the previously created instances in compute node were not reachable. Continuiung to debug further.

Can you please mention the steps tried out from your end

(In reply to Venkatrangan Govindarajan from comment #3)
> hi,
>
> I used the Ocata Official instalaltion guide to install every package
> required for Openstack and configured them to use ODL. Then restarted the
> Control node. When the System was up again, I see that the qdhcp NS getting
> created but the previously created instances in compute node were not
> reachable. Continuiung to debug further.
>
>
> Can you please mention the steps tried out from your end

The steps are in 'How to reproduce' in the bug itself.
I checked it also with Pike.

I have reproduced the issue and have observed 2 observation.

Setup:

• Control node with ODL
• Compute node

Topo:

• Create a network
• Create a VM
  DO A REBOOT of control with ODL setup

Observation#1

• DHCP tap intf port number has been changed after the ODL reboot. (port:1 -> port:3)
• VM tap intf port number has been changed after the ODL reboot. (port:1 -> port:3)
• In control node, ct_table flows and actions are changed in dump-flows. But while checking DS, the flows are not changed.

In Control node: After ODL reboot

• Table 213 and 243, missing ct_state flows.
• Table 220, instead of output:3, the action is changed to DROP
• In most of the tables, the action are changed from resubmit,17 to DROP
  In Compute node: After ODL reboot
• Flows are installed freshly. But, no flows are missing. Neither no actions are changed.
• 1 packet drop has been observed in table 211

In Control node: After VM soft reboot

• Table 0 flow for earlier DHCP port was still present. (port:1)
  In Compute node: After VM soft reboot
• Packet count varies due to newly installed flows.
• Drop packet count raised in table 211

Observation#2
In dump flows, the actions are updated correctly. But in the control node, output port for network was not updated after reboot.
before reboot : Network port is 1
After reboot : Network port is 3
But in dump flow, output action port is 1
Thus, manually deleted the output action port 1 and added new flow with port 3. Thereafter did a soft reboot to the VM. IP got assigned to the VM.

Itzik,
Can you please share your dump-flows, data store info and karaf logs
Thanks in advance!

Attachment observation_1.zip has been added with description: Observation_1

Attachment observation_2.zip has been added with description: Observation_2

Tested below scenarios
Setup:
1 node ODL
1 Control Node
1 Compute Node

Topo:

• Create a network
• Create a VM

Scenario#1 : Reboot only ODL node
-> The issue is NOT reproduced.
-> VM got IP after ODL was rebooted and the respective VM was reachable.

Scenario#2 : Reboot Control node
-> Once the control node is rebooted, console was unreachable.
(openstack) console log show VM1
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-8a6568e0-6676-40ac-a494-23e35739c089)
-> VM reachability was failed too.

Itzik,

Workaround for this bug: Restart OpenVswitch
[sudo systemctl restart openvswitch]
The correct flows are updated in control node as expected.

Distro Version: Carbon

Observation:
Once we do a reboot,
-> In control node, the resubmit actions are changed to drop. And addition to that, some flow's match conditions are missing (Example: reg6)
REF: Please check control node flows (before and after reboot)

Say, we are analysing single flow Table:55,
-> The config datastore for Inventory nodes and network topology are updated.
-> Flows were added from Openflowpluin resubmit action is written in the instruction.
-> In tcpdump, table 55 flows are written twice. and the instruction has action as drop rather than, resubmit.
-> In openvswitch log, the table 55 flows are written twice. And the action is drop too. (REF: openvswitch line#13848, 13849, 13906 and 13907)

Attachment ovs.zip has been added with description: OVS_log

Attachment DataStrore_tcpdump.zip has been added with description: DataStrore and tcpdump

Addition to that, issue occurs in Carbon too.

There seems to be some errors in OFPT_GROUP_MOD that needs to be analyzed.

Resubmit flow was pushed to the device and once I performed reboot, the same issue occurred in openflowplugin.

Setup:

• Control node with ODL

Features to be installed:

• odl-openflowplugin-flow-services
• odl-openflowplugin-nxm-extensions
• odl-mdsal-apidocs

Steps:

• Push a resubmit flow using RESTAPI
• Reboot the control node with ODL.

Observation:

• The flow with action resubmit had been changed to action drop after reboot.
• In config datastore, the action is not been changed to drop. But in device the action has been changed to drop.

Please ref bug: https://bugs.opendaylight.org/show_bug.cgi?id=9137

Moved to Bertrand.

Hi Itzik and Arthi, this bug appears to be resolved in stable/nitrogen and master branches (see the https://jira.opendaylight.org/browse/OPNFLWPLUG-942 comments). Are you able to test this bug out using the latest nitrogen or master branch build?

Any update?

No update on this bug for more then 2 months. Closing the bug. Please re-open the bug if it's not resolved and you are planning to work on it.