[OPNFLWPLUG-248] DroptestRPC doesn't work with Cbench, When using more than 140 switches Created: 19/Aug/14  Updated: 27/Sep/21  Due: 12/Sep/14  Resolved: 11/Sep/14

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Christopher O'Shea Assignee: Christopher O'Shea
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC

External issue ID: 1581

 Description   

Try to run Cbench with dropAllPacketsRpc On,

First round i get some returns but all test after return zero.

Error seen on OSGi console
2014-08-19 16:08:56.398 UTC [md-sal-binding-notification-12] ERROR o.o.o.droptest.AbstractDropTest - Failed to process packet
java.lang.IllegalStateException: No default provider is available
at org.opendaylight.yang.gen.v1.urn.opendaylight.flow.service.rev130819.SalFlowService$$Broker$DirectProxy.addFlow(SalFlowService$$Broker$DirectProxy.java) ~[na:na]
at org.opendaylight.openflowplugin.droptest.DropTestRpcSender.processPacket(DropTestRpcSender.java:71) ~[bundlefile:na]
at org.opendaylight.openflowplugin.droptest.AbstractDropTest.onPacketReceived(AbstractDropTest.java:121) ~[bundlefile:na]
at org.opendaylight.openflowplugin.droptest.DropTestRpcSender$$Broker$ListenerInvoker.onNotification(DropTestRpcSender$$Broker$ListenerInvoker.java) [bundlefile:na]
at org.opendaylight.controller.sal.binding.impl.AbstractNotificationListenerRegistration.notify(AbstractNotificationListenerRegistration.java:38) [bundlefile:na]
at org.opendaylight.controller.sal.binding.impl.NotifyTask.run(NotifyTask.java:42) [bundlefile:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

 Comments   
Comment by RichardHill [ 20/Aug/14 ]

Using SP edition Command dropAllPacketsRpc was not found:

Version

org.opendaylight.controller.version = 0.1
org.opendaylight.controller.build.scm.version = 79e737d256d5d3ccb7dd52893caa8dce26a4e17b
org.opendaylight.controller.build.user = jenkins-controller
org.opendaylight.controller.build.workspace = **********
org.opendaylight.controller.build.timestamp = 1408520473664
org.opendaylight.controller.build.machine = **********

Steps

  1. telnet localhost 2400
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.
    osgi> dropAllPacketsRpc on
    gogo: CommandNotFoundException: Command not found: dropAllPacketsRpc
Comment by Christopher O'Shea [ 20/Aug/14 ]

The droptest bundle isn't found in any of the normal release you have to have to download the bundle see step 3

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test

Comment by RichardHill [ 20/Aug/14 ]

https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/1999/

Comment by Vaclav Demcak [ 20/Aug/14 ]

https://git.opendaylight.org/gerrit/#/c/10106/

Problem is a command dropAllPacketsRpc On ... osgi consolle is case insensitive but provider start DropTestRpcSender only for "on" command exactly.

Change:
start DropTestRpcSender for case insensitive command.

Comment by Christopher O'Shea [ 20/Aug/14 ]

Hi Vaclav,

I don't believe this will fix the issue as i get back
"dropAllPacketsRpc on
DropAllFlows transitions to on"

When we started the performance testing we see reply from Cbench see output
https://jenkins.opendaylight.org/integration/job/integration-master-performance-plugin/50/console

No we see we get some reply for the first run then we just see the Errors in OSGi console, below is the output from cbench.
https://jenkins.opendaylight.org/integration/job/integration-master-performance-plugin/65/console

Comment by Vaclav Demcak [ 21/Aug/14 ]

I can see. The problem should be a bad NodeConnector lifecycle.

Next patches could help:
https://git.opendaylight.org/gerrit/#/c/10091/
https://git.opendaylight.org/gerrit/#/c/10063/
https://git.opendaylight.org/gerrit/#/c/10064/
https://git.opendaylight.org/gerrit/#/c/10141/

Comment by Christopher O'Shea [ 26/Aug/14 ]

I'm not sure what's going on.

When running with the base edition, and the download the drop test bundle.
https://jenkins.opendaylight.org/integration/job/integration-master-project-centralized-integration/lastSuccessfulBuild/artifact/distributions/base/target/distributions-base-0.2.0-SNAPSHOT-osgipackage.zip
I see the issue.

But when i run with the build from OFPlugin private build
https://jenkins.opendaylight.org/openflowplugin/job/openflowplugin-merge/lastSuccessfulBuild/org.opendaylight.openflowplugin$distributions-openflowplugin-base/artifact/org.opendaylight.openflowplugin/distributions-openflowplugin-base/0.0.3-SNAPSHOT/distributions-openflowplugin-base-0.0.3-SNAPSHOT-osgipackage.zip

Comment by Michal Rehak [ 28/Aug/14 ]

Hi Christopher,
drop-test bundle contains the "repeater" for cbench. You should remove arphandler, simple-forwarding-manager and statisticsmanager in order to get the best performance.

In openflowplugin there is specialized distribution: distribution/cbench having those bundles removed.

In case you get the "no provider" exception it means that drop-test is missing or is not loaded.

Could you inspect loaded bundles when using the integration build and eventually logs if there is anything stinky regarding drop-test bundle?

Comment by Christopher O'Shea [ 29/Aug/14 ]

After talking to Michael Rehak, we found it was related to have a large number of switches, anything above 140 the controller got stuck.

This Cbench test was with 256 switch and ran on Aug 14, 2014 7:08:19 PM
https://jenkins.opendaylight.org/integration/job/integration-master-performance-plugin/39/console

Comment by Timotej Kubas [ 05/Sep/14 ]

TopologyManager and InventoryManager were reworked and now they use transaction chaining what solved the problem.

Mentioned patches can be found here:
https://git.opendaylight.org/gerrit/#/c/10759/
https://git.opendaylight.org/gerrit/#/c/10064/

Cbench tests with 900 switches passed with no problems.
Please retest.

Comment by Christopher O'Shea [ 11/Sep/14 ]

Cbench testing does run with 256 switches.

Generated at Wed Feb 07 20:31:57 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.