Operating System: All
Platform: All

Attachment captures.zip has been added with description: captures of before and after network restart

Hi Sandeep,

Can you reproduce the issue with odl-openflowplugin-flow-services-ui feature only? If not I think we should submit this bug to l2switch folks.

BR/Luis

(In reply to Luis Gomez from comment #1)
> Hi Sandeep,
>
> Can you reproduce the issue with odl-openflowplugin-flow-services-ui feature
> only? If not I think we should submit this bug to l2switch folks.
>
> BR/Luis

Hi Luis
Links are not learned until odl-l2switch-loopremover is installed. This is the behavior in Lithium and Helium.

Regards
Sandeep

(In reply to SANDEEP GANGADHARAN from comment #2)
> (In reply to Luis Gomez from comment #1)
> > Hi Sandeep,
> >
> > Can you reproduce the issue with odl-openflowplugin-flow-services-ui feature
> > only? If not I think we should submit this bug to l2switch folks.
> >
> > BR/Luis
>
> Hi Luis
> Links are not learned until odl-l2switch-loopremover is installed.
> This is the behavior in Lithium and Helium.
>
> Regards
> Sandeep

Sandeep,

I didn't follow all the way. Are you saying that your environment doesn't learn links at all until you install loopremover? If so, I think something is wrong as we should be getting link learning with just ofp-flow-services-ui

Thanks,
JamO

(In reply to Jamo Luhrsen from comment #3)
> (In reply to SANDEEP GANGADHARAN from comment #2)
> > (In reply to Luis Gomez from comment #1)
> > > Hi Sandeep,
> > >
> > > Can you reproduce the issue with odl-openflowplugin-flow-services-ui feature
> > > only? If not I think we should submit this bug to l2switch folks.
> > >
> > > BR/Luis
> >
> > Hi Luis
> > Links are not learned until odl-l2switch-loopremover is installed.
> > This is the behavior in Lithium and Helium.
> >
> > Regards
> > Sandeep
>
>
> Sandeep,
>
> I didn't follow all the way. Are you saying that your environment doesn't
> learn links at all until you install loopremover? If so, I think something
> is wrong as we should be getting link learning with just ofp-flow-services-ui
>
> Thanks,
> JamO

Jamo/Luis
Yes. With only ofp-flow-services-ui enabled controller does not learn links. Should this be another defect?

Regards
Sandeep

so after further investigation with Sandeep, we have narrowed things down a little more.

the first point is that this problem is coming if you only install odl-openflowplugin-flow-services-ui. The confusion between us is that with
just that feature I was seeing links being learned and Sandeep was not.
The reason is that my ovs v2.0 was punting packets (so lldp's) to the
controller. By default, Sandeeps v2.3 ovs was dropping those so no
links are learned. If you use v2.3 ovs you can then program a catchall
flow to punt packets to the controller in order to have lldp's sent
there and links learned.

so, with just ofp-flow-services-ui installed, the basic steps of starting
mininet with a linear,2,0 topo, stopping then restarting will show that
the expected 2 links are not learned. just one.

to be clear, we wanted to rule out that l2switch-loopremover was getting
in the way. it does not appear to be the case.

Cool, then we can say this is an ofplugin bug and I personally would like to see this fix in Lithium as even if a configuration with no hosts is unusual, this bug could uncover some design problem.

So Jamo and I were concerned this small issue was hiding something big and it actually does. After more analysis and test here are my conclusions:

1) LLDP speaker does not start/stop sending LLDP packets on port up/down events (similar to current l2switch not reacting to port events)

2) However when a switch connects for the first time to ODL, the LLDP speaker ONLY sends LLDP packets to those ports that report up during registration. This would be right if 1) was working.

So, the combination of 1) and 2) behaviors makes topology to fail to update any new connection you create between switches after initial registration.

BR/Luis

Adding Anil to this one.

Anil fixed similar issue when a switch adds/removes an openflow port. This is a similar case where the port goes up/down.

Hi, here is the experimental fix. I pushed it on master. Please test it whenever you have spared cycles.
https://git.opendaylight.org/gerrit/#/c/22811/

What is this to do with l2switch? the issue is in openflowplugin helium code at least (I have not tested with Li code).

(In reply to Luis Gomez from comment #11)
> What is this to do with l2switch? the issue is in openflowplugin helium code
> at least (I have not tested with Li code).

To reinforce Luis' point, see comment #5. The issue detailed in this
bug does not involve l2switch (those features are not even installed)

OK, I see the point now, this issue is not happening with Li redesign, only with He code.

Since all projects in Li are based in He code we need a workaround and here is my suggest: if the port event notifications are difficult to address in He code, lets change the LLDP speaker to send LLDP packets at connection time to all ports regarless whether they are up or down. This should be easy to achieve in code and will alleviate very much this bug.

BR/Luis

(In reply to Luis Gomez from comment #13)
> OK, I see the point now, this issue is not happening with Li redesign, only
> with He code.
>
> Since all projects in Li are based in He code we need a workaround and here
> is my suggest: if the port event notifications are difficult to address in
> He code, lets change the LLDP speaker to send LLDP packets at connection
> time to all ports regarless whether they are up or down. This should be easy
> to achieve in code and will alleviate very much this bug.
>
> BR/Luis

This could produce a lot of useless noise in the mgmt network. Imagine a few high port density switches. The controller will be sending LLDP packet outs on the wire a lot.

but if it's determined to be the only way for now, it's better to do that, than lose a link in the topology.

I understand Jamo. My workaround is in case nothing can be done or too much effort to fix. Better too chatty protocol than not working right?

(In reply to Luis Gomez from comment #15)
> I understand Jamo. My workaround is in case nothing can be done or too much
> effort to fix. Better too chatty protocol than not working right?

Probably not too chatty - if the ports are not up nothing will come on the wire - right?

(In reply to Abhijit Kumbhare from comment #16)
> (In reply to Luis Gomez from comment #15)
> > I understand Jamo. My workaround is in case nothing can be done or too much
> > effort to fix. Better too chatty protocol than not working right?
>
> Probably not too chatty - if the ports are not up nothing will come on the
> wire - right?

No, the suggestion is that if we cannot send LLDP packet-out based on ports being UP (via notifications) then we just send LLDP on all ports that the switch advertises, even if they are down. So, packet-out on a down port is the chatty part.

I agree with Luis. Chatty is better than broken.

More reasons to fix: this works in Helium so it could be seen as regression

I was asked by Michal Rehak to continue with his remarks.
As he identified, the probable place for change shoud be in class
org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator

I prepared this patch
https://git.opendaylight.org/gerrit/#/c/22882/1

I have tested it aproximately 10 times (mininet up-down, with 2 switches with no hosts) and in logs I have seen both links.
I installed only
>>>>feature:install odl-openflowplugin-flow-services-ui

and debugging was set as following
>>>>>>log:set TRACE org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPSpeaker
>>>>>>log:set TRACE org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator

Luis: I pushed a fix to stable/helium branch as well

https://git.opendaylight.org/gerrit/23795

Can you please verify it.

Anil, your patch actually fixes:

https://bugs.opendaylight.org/show_bug.cgi?id=3233

which is a different issue

This was actually fixed before we released Lithium.

Actually it is not, I just checked with latest stable/lithium.

This bug is still pending for resolution...

This one is tricky to reproduce with mininet becasue you need a switch with a port in state DOWN when it connects to controller and then bring the port UP. I just did with a HW switch and the issue is still there.

Adding Kamal for a quick check of this.

Adding Anil for a quick check of this. Apparently this is happening only with the He design.

Just tested and LLDP speaker works good in the case of port UP. For the port DOWN the issue is still there but it is not that critical so lowering the priority.

BR/Luis

I tried to reproduce this problem on latest master branch (carbon) and it is working perfectly fine (at least when I tried to reproduce it with steps described in original description, but + added table miss entries, because otherwise links are just not showing)

1. Start mininet with linear topo with 2 switches and no hosts
2. Check topology
— NODES [2]---
[openflow:1]
openflow:1:1
openflow:1:LOCAL
[openflow:2]
openflow:2:LOCAL
openflow:2:1
— LINKS [2]---
openflow:2:1 --> openflow:1:1
openflow:1:1 --> openflow:2:1
3. Exit mininet and check topology
— NODES [2]---
[openflow:1]
openflow:1:1
openflow:1:LOCAL
[openflow:2]
openflow:2:LOCAL
openflow:2:1
— LINKS [0]---
4. Start mininet again, and check topology (result should be same as in step 2)
5. Set port 1 of switch 1 DOWN and check topology (result should be same as in step 3)
6. Set port 1 of switch 1 UP and check topology (result should be same as in step 2)

Attached also pcap from Wireshark.

Attachment lldp.pcapng has been added with description: Switch port UP/DOWN capture

I do not see PORT STATUS events in the capture, normally to verify this bug with wireshark we should see a port DOWN event and after controller should not be sending any LLDP packet-out to that port.

Adding capture also with PORT_STATUS messages, with these steps

1. Start mininet
2. Set port 1 on switch s1 DOWN
3. Set port 1 on switch s1 UP
4. Stop mininet
5. Repeat steps 1 - 4

PORT_STATUS is correctly send on DOWN event, but I think, even when we received this DOWN event, PACKET_OUTS are still sent to port.

Attachment lldp.pcapng has been added with description: PORT_STATUS+ LLDP capture

I believe this bug was fixed in helium plugin. Given that helium plugin is now deprecated, please re-open the bug if you see this issue with carbon/nitrogen/oxygen branch.