[OPNFLWPLUG-426] [Lithium redesign] cbench throughput spikes CPU to 100% and no results are reported Created: 04/May/15  Updated: 27/Sep/21  Resolved: 05/May/15

Status: Verified
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Jamo Luhrsen Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: JPEG File 3114_CPU_usage.jpg    
Issue Links:
Duplicate
is duplicated by OPNFLWPLUG-425 NPE at org.opendaylight.openflowplugi... Resolved
External issue ID: 3114

 Description   

Investigation is ongoing....

features:
odl-openflowplugin-flow-services-ui-li
odl-openflowplugin-drop-test-li

config:
"dropallpacketsrpc on"

cbench cmdLine:
/usr/local/bin/cbench -c 172.17.7.23 -t -m 12000 -M 10000 -s 8 -l 3

There is no logging that indicates a deadlock is being noticed, but that was one comment
I saw in email:
https://lists.opendaylight.org/pipermail/openflowplugin-dev/2015-May/003012.html

One a 4 core system, all CPUs are running near the 100% limit. Have not isolated the exact
consumer of those cpu cycles yet.

There is an NPE seen in the karaf log when cbench is started:
2015-05-04 16:22:12,033 | ERROR | lt-dispatcher-24 | DataChangeListener | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Error notifying listener org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl
java.lang.NullPointerException
at org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl.reconciliation(FlowNodeReconciliationImpl.java:162)[188:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.1.0.SNAPSHOT]
at org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl.flowNodeConnected(FlowNodeReconciliationImpl.java:137)[188:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.1.0.SNAPSHOT]
at org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl.onDataChanged(FlowNodeReconciliationImpl.java:123)[188:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.1.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:134)[157:org.opendaylight.controller.sal-binding-broker-impl:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[164:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[164:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[164:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[164:com.typesafe.akka.actor:2.3.10]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]



 Comments   
Comment by Jamo Luhrsen [ 04/May/15 ]

after starting cbench, the following screen shot is what my
profiler looks like for CPU usage.

Comment by Jamo Luhrsen [ 04/May/15 ]

Attachment 3114_CPU_usage.jpg has been added with description: screenshot of cpu sampler from jvisual

Comment by Jozef Gloncak [ 05/May/15 ]

https://git.opendaylight.org/gerrit/#/c/19569/ - check for null reference

Comment by Jamo Luhrsen [ 05/May/15 ]

verified with build from nexus:
distribution-karaf-0.3.0-20150505.173521-1449.zip

Generated at Wed Feb 07 20:32:26 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.