Operating System: All
Platform: All

A situation I'm running into consistently is that the Openstack control node which is stacking (and using ODL) will end up not setting the 'controller' for br-int. As I understand it, devstack is creating br-int - presumably before the 'manager' has been set to ODL. (br-int is not present prior to stacking)

Anyway, once stacking is complete, the 'manager' is set to ODL, but the 'controller' for br-int (and br-ex) is not set. And so the pipeline flows are not present.

After manually setting the controller to point to ODL, the pipeline flows get added.

So, is this the same bug, or should I file a separate bug?

I am investigating the issue reported by Eric. IMO, the two may be same. What Sam saw is the effect of controller not getting added.

I think there are different issues:

1. The bug I reported here is valid. The pipeline flows will only get added when a node event comes in for a bridge. That does not happen in the case for this bug. In this case you can see that the pipeline flows are never added to config.

2. The pipeline flows are programmed but the controller is never set so the bridge never connects as a openflow switch and flows pushed to it. In this case you can see the pipeline flows in config and the controller address is set.

We need to be clear on which case is being hit. I will create another bug to cover case 2.

There might also be a third case: https://bugs.opendaylight.org/show_bug.cgi?id=3974. In that case br-int is not even created.

I tried first 3 steps mentioned here with the patch for 4135 and the flow came up as -

[stack@controller ~]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=21.818s, table=0, n_packets=0, n_bytes=0, priority=0 actions=goto_table:20
cookie=0x0, duration=23.783s, table=0, n_packets=0, n_bytes=0, dl_type=0x88cc actions=CONTROLLER:65535
cookie=0x0, duration=21.751s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:30
cookie=0x0, duration=21.668s, table=30, n_packets=0, n_bytes=0, priority=0 actions=goto_table:40
cookie=0x0, duration=21.584s, table=40, n_packets=0, n_bytes=0, priority=0 actions=goto_table:50
cookie=0x0, duration=21.493s, table=50, n_packets=0, n_bytes=0, priority=0 actions=goto_table:60
cookie=0x0, duration=21.409s, table=60, n_packets=0, n_bytes=0, priority=0 actions=goto_table:70
cookie=0x0, duration=21.325s, table=70, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80
cookie=0x0, duration=21.259s, table=80, n_packets=0, n_bytes=0, priority=0 actions=goto_table:90
cookie=0x0, duration=21.183s, table=90, n_packets=0, n_bytes=0, priority=0 actions=goto_table:100
cookie=0x0, duration=21.133s, table=100, n_packets=0, n_bytes=0, priority=0 actions=goto_table:110
cookie=0x0, duration=21.083s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop

I get the same flow if I do not manually add br-int.

Am I missing something?

(In reply to Sharad Mishra from comment #4)
> I tried first 3 steps mentioned here with the patch for 4135 and the flow
> came up as -
>
> [stack@controller ~]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
> OFPST_FLOW reply (OF1.3) (xid=0x2):
> cookie=0x0, duration=21.818s, table=0, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:20
> cookie=0x0, duration=23.783s, table=0, n_packets=0, n_bytes=0,
> dl_type=0x88cc actions=CONTROLLER:65535
> cookie=0x0, duration=21.751s, table=20, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:30
> cookie=0x0, duration=21.668s, table=30, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:40
> cookie=0x0, duration=21.584s, table=40, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:50
> cookie=0x0, duration=21.493s, table=50, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:60
> cookie=0x0, duration=21.409s, table=60, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:70
> cookie=0x0, duration=21.325s, table=70, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:80
> cookie=0x0, duration=21.259s, table=80, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:90
> cookie=0x0, duration=21.183s, table=90, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:100
> cookie=0x0, duration=21.133s, table=100, n_packets=0, n_bytes=0, priority=0
> actions=goto_table:110
> cookie=0x0, duration=21.083s, table=110, n_packets=0, n_bytes=0, priority=0
> actions=drop
>
>
> I get the same flow if I do not manually add br-int.
>
> Am I missing something?

Sharad,

not sure what the question is. Are you just saying that you can't reproduce the bug with your patch? That's possible since all these bugs are timing related. The key is to isolate the scenario which is hard. The 4135 fix may or may not have any impact of the problem here. The only way to verify is if the flows are in config and not operational. If you see the flows on the switch then that means they are in operational.

It is this line in PipelineOrchestratorImpl that is failing for this bug:
if (southbound.getBridge(node) != null) {

In this case the node event does not have a BridgeAugmentation so it fails and the pipeline flows don't get programmed.

Sam

(In reply to Sam Hague from comment #5)
> (In reply to Sharad Mishra from comment #4)
> > I tried first 3 steps mentioned here with the patch for 4135 and the flow
> > came up as -
> >
> > [stack@controller ~]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
> > OFPST_FLOW reply (OF1.3) (xid=0x2):
> > cookie=0x0, duration=21.818s, table=0, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:20
> > cookie=0x0, duration=23.783s, table=0, n_packets=0, n_bytes=0,
> > dl_type=0x88cc actions=CONTROLLER:65535
> > cookie=0x0, duration=21.751s, table=20, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:30
> > cookie=0x0, duration=21.668s, table=30, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:40
> > cookie=0x0, duration=21.584s, table=40, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:50
> > cookie=0x0, duration=21.493s, table=50, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:60
> > cookie=0x0, duration=21.409s, table=60, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:70
> > cookie=0x0, duration=21.325s, table=70, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:80
> > cookie=0x0, duration=21.259s, table=80, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:90
> > cookie=0x0, duration=21.183s, table=90, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:100
> > cookie=0x0, duration=21.133s, table=100, n_packets=0, n_bytes=0, priority=0
> > actions=goto_table:110
> > cookie=0x0, duration=21.083s, table=110, n_packets=0, n_bytes=0, priority=0
> > actions=drop
> >
> >
> > I get the same flow if I do not manually add br-int.
> >
> > Am I missing something?
>
> Sharad,
>
> not sure what the question is. Are you just saying that you can't reproduce
> the bug with your patch? That's possible since all these bugs are timing
> related. The key is to isolate the scenario which is hard. The 4135 fix may
> or may not have any impact of the problem here. The only way to verify is if
> the flows are in config and not operational. If you see the flows on the
> switch then that means they are in operational.
>
> It is this line in PipelineOrchestratorImpl that is failing for this bug:
> if (southbound.getBridge(node) != null) {
>
> In this case the node event does not have a BridgeAugmentation so it fails
> and the pipeline flows don't get programmed.
>
> Sam

Sam,

I am trying to recreate the bug. Since it is timing related .. it is intermittent. And if I understand you correctly, since in my case I got the flows on switch, it worked this time. Is that correct?

-Sharad

(In reply to Sharad Mishra from comment #6)
> (In reply to Sam Hague from comment #5)
> > (In reply to Sharad Mishra from comment #4)
> > > I tried first 3 steps mentioned here with the patch for 4135 and the flow
> > > came up as -
> > >
> > > [stack@controller ~]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
> > > OFPST_FLOW reply (OF1.3) (xid=0x2):
> > > cookie=0x0, duration=21.818s, table=0, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:20
> > > cookie=0x0, duration=23.783s, table=0, n_packets=0, n_bytes=0,
> > > dl_type=0x88cc actions=CONTROLLER:65535
> > > cookie=0x0, duration=21.751s, table=20, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:30
> > > cookie=0x0, duration=21.668s, table=30, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:40
> > > cookie=0x0, duration=21.584s, table=40, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:50
> > > cookie=0x0, duration=21.493s, table=50, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:60
> > > cookie=0x0, duration=21.409s, table=60, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:70
> > > cookie=0x0, duration=21.325s, table=70, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:80
> > > cookie=0x0, duration=21.259s, table=80, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:90
> > > cookie=0x0, duration=21.183s, table=90, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:100
> > > cookie=0x0, duration=21.133s, table=100, n_packets=0, n_bytes=0, priority=0
> > > actions=goto_table:110
> > > cookie=0x0, duration=21.083s, table=110, n_packets=0, n_bytes=0, priority=0
> > > actions=drop
> > >
> > >
> > > I get the same flow if I do not manually add br-int.
> > >
> > > Am I missing something?
> >
> > Sharad,
> >
> > not sure what the question is. Are you just saying that you can't reproduce
> > the bug with your patch? That's possible since all these bugs are timing
> > related. The key is to isolate the scenario which is hard. The 4135 fix may
> > or may not have any impact of the problem here. The only way to verify is if
> > the flows are in config and not operational. If you see the flows on the
> > switch then that means they are in operational.
> >
> > It is this line in PipelineOrchestratorImpl that is failing for this bug:
> > if (southbound.getBridge(node) != null) {
> >
> > In this case the node event does not have a BridgeAugmentation so it fails
> > and the pipeline flows don't get programmed.
> >
> > Sam
>
>
> Sam,
>
> I am trying to recreate the bug. Since it is timing related .. it is
> intermittent. And if I understand you correctly, since in my case I got the
> flows on switch, it worked this time. Is that correct?
>
> -Sharad

Sharad,

yes, if the flows are on the switch then it passed and the bug wasn't hit.

4014 and 4135 would look similar on the bridge in that you would not see flows. That could happen for different reasons on the ODL side.

1. The line of code I mentioned above where an bridge augmentationis not found and causes the flows to not be added to config. In this case there could be a br-int in operational but somehow NetVirt doesn't see the node update and hence doesn't add the flows. Since the flows are not added to config, they are never pushed to the switch. Here you would need to see if the flows are in config and not on the switch and if the controller is set and connected.

2. The issue which 4135 fixed where the controller was never added to the bridge. In this case the pipeline flows would be programmed and added to config, but since the set-controller never worked the flows would never get pushed to the switch. You would need to chck if the controller is set in the switch to verify if this was hit.

For 1, when I hit the issue, it was during the devstack networking-odl adding br-int and then connecting to ODL. I don't really know how this could happen, though, because I thought if the ovsdb node connects to ODL then those operational updates should make it to NetVirt - one of the updates should have been a BridgeAugmentation. But that did not happen. I could see the ovsdb node connected to ODL and I saw the br-int bridge connected to ODL OpenFlow - but not flows. Looking deeper I saw that the flows were not in config.

Sam

forgot to add, in your testing are you using devstack? And without the fixes we pushed to make devstack not add br-int?

Sam, I am using devstack and latest ODL master.