[#OPNFLWPLUG-939] Regression in Switch scalability test (carbon)

[OPNFLWPLUG-939] Regression in Switch scalability test (carbon) Created: 04/Sep/17 Updated: 27/Sep/21 Resolved: 27/Sep/17
Status:	Resolved
Project:	OpenFlowPlugin
Component/s:	General
Affects Version/s:	None
Fix Version/s:	None

Type:

Bug

Reporter:

Luis Gomez

Assignee:

Luis Gomez

Resolution:

Done

Votes:

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Environment:

Operating System: All
Platform: All

External issue ID:

9089

Description

It seems we can hardly do more than 100 switches in carbon/nitrogen:

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/plot/Switch%20Scalability/

However boron seems pretty good:

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability/

Comments

Comment by Luis Gomez [ 04/Sep/17 ]

From the karaf log, it seems switches fail to connect:

Comment by Luis Gomez [ 05/Sep/17 ]

Here is the karaf log with INFO level:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/129/odl1_karaf.log.gz

It seems like in second iteration of 200 switches, many of the OF connection remain slave (never move to master).

Comment by Tomas Slusny [ 06/Sep/17 ]

Can you try it with this patch: https://git.opendaylight.org/gerrit/#/c/62607/8 Luis?

Comment by Luis Gomez [ 06/Sep/17 ]

Sure, I just triggered the build. It will take some time to start given existing jenkins queue.

Comment by Luis Gomez [ 07/Sep/17 ]

The result from yesterday run with the patch does not show better:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/133/odl1_karaf.log.gz

Comment by Luis Gomez [ 14/Sep/17 ]

OK, after looking at the karaf log in detail, this is why scalability test fails and controller breaks. See below scenarios:

1) In normal scenario with few switches the device is processed immediately after it connects:

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]}

2) When more switches start to connect simultaneously, controller takes more time to process the switches and after 200 switches it is very likely this time overpasses the 20 secs max time to set up slave, so in this case there is first a slave role set and then a master role set after switch is served:

3) Finally, this is the problematic scenario that happens when a switch is processed just before the 20 secs max time to set up slave, controller in this case mess ups all as it tries to setup master and slave roles simultaneously as you can see below:

Since this can happen with relatively few switches connecting at the same time and the impact is really bad (only solution is restart controller), I am raising this issue as BLOCKER for nitrogen.

Comment by Tomas Slusny [ 14/Sep/17 ]

Can you test it with this patch Luis? https://git.opendaylight.org/gerrit/#/c/63123/

Comment by Luis Gomez [ 14/Sep/17 ]

Result is bit better but I still see the issue:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/147/odl1_karaf.log.gz

— Device disconnects because load —

— Device Reconnects —

— Master process starts —

— 20 secs slave role kicks in the middle and problem happens —

Comment by Tomas Slusny [ 18/Sep/17 ]

Thank you for logs, it helped a lot. The failure case was because SLAVE role future was cancelled because we started sending MASTER role, and we sent slave role not aquired notification to mastership service, so I added check for CancellationException so this will not happend anymore. Updated both patches (master, stable/nitrogen), so can you re-test it Louis?

Comment by Luis Gomez [ 18/Sep/17 ]

Good news last patchset:

https://git.opendaylight.org/gerrit/#/c/63123/

gets scalability as should be:

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/plot/Switch%20Scalability/

Comment by Luis Gomez [ 18/Sep/17 ]

Nitrogen patch: https://git.opendaylight.org/gerrit/#/c/63146

Comment by Tomas Slusny [ 19/Sep/17 ]

Carbon patch: https://git.opendaylight.org/gerrit/#/c/63275/
It is a bit different because there is no RoleContext that centralizes role management in Carbon, what complicates things.

Comment by Luis Gomez [ 20/Sep/17 ]

From your comment: what do we want to do here? are we not going to cherry-pick nitrogen cluster fixes to carbon?

Comment by Luis Gomez [ 20/Sep/17 ]

Also the carbon patch does not work:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/409/odl1_karaf.log.gz

From the log we see connection ongoing but actually nothing shows in operational for switch openflow:146

Comment by Tomas Slusny [ 21/Sep/17 ]

Hmm that is weird, logs looks fine. I noticed some weird thing (that slave task did not got fully cancelled for some reason) so I uploaded new patch.

Comment by Luis Gomez [ 22/Sep/17 ]

Same issue, still mess up roles in carbon:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/411/odl1_karaf.log.gz

Master and Slave at the same time and scale test fails on this switch:

Comment by Luis Gomez [ 22/Sep/17 ]

Change title to reflect only carbon has the issue now.

Comment by Tomas Slusny [ 26/Sep/17 ]

Alright, I tried to reproduce what was changed in role context in nitrogen and master, can you try this issue with this patch: https://git.opendaylight.org/gerrit/#/c/63536/ luis?

Comment by Luis Gomez [ 27/Sep/17 ]

OK, the last patch seems to work in carbon but now we have new issue preventing switch scale test to work properly:

https://bugs.opendaylight.org/show_bug.cgi?id=9217

Comment by Tomas Slusny [ 27/Sep/17 ]

Alright the patch on carbon got merged, so should we wait until the new bug is closed before we close this one?

Comment by Luis Gomez [ 27/Sep/17 ]

No thats fine, I already tested it.

Generated at Wed Feb 07 20:33:47 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.

[OPNFLWPLUG-939] Regression in Switch scalability test (carbon) Created: 04/Sep/17 Updated: 27/Sep/21 Resolved: 27/Sep/17