[CONTROLLER-1342] Uncaught error from thread [opendaylight-cluster-data-akka.actor.default-dispatcher-90] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data] Uncaught error from thread [opendaylight-cluster-data-akka

Blocks — Thu, 19 Oct 2017 15:30:03 +0000

opendaylight-user@root>Uncaught error from thread [opendaylight-cluster-data-akka.actor.default-dispatcher-90] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data]
Uncaught error from thread [opendaylight-cluster-data-akka.actor.default-dispatcher-23] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data]
java.lang.NoClassDefFoundError: com/google/common/collect/Iterables
at org.opendaylight.controller.liblldp.LLDP.getLLDPPacketLength(LLDP.java:288)
at org.opendaylight.controller.liblldp.LLDP.serialize(LLDP.java:254)
at org.opendaylight.controller.liblldp.Packet.serialize(Packet.java:161)
at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPUtil.buildLldpFrame(LLDPUtil.java:97)
at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPSpeaker.nodeConnectorAdded(LLDPSpeaker.java:152)
at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.notifyNodeConnectorAppeared(NodeConnectorInventoryEventTranslator.java:104)
at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.onDataChanged(NodeConnectorInventoryEventTranslator.java:71)
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:134)
at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)
at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Uncaught error from thread [java.lang.NoClassDefFoundError: com/google/common/collect/Iterables
at org.opendaylight.controller.liblldp.LLDP.getLLDPPacketLength(LLDP.java:288)
at org.opendaylight.controller.liblldp.LLDP.serialize(LLDP.java:254)
at org.opendaylight.controller.liblldp.Packet.serialize(Packet.java:161)
at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPUtil.buildLldpFrame(LLDPUtil.java:97)
at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPSpeaker.nodeConnectorAdded(LLDPSpeaker.java:152)
at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.notifyNodeConnectorAppeared(NodeConnectorInventoryEventTranslator.java:104)
at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.onDataChanged(NodeConnectorInventoryEventTranslator.java:71)
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:134)
at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)
at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
opendaylight-cluster-data-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data]
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Iterables
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 23 more
java.lang.NoClassDefFoundError: com/google/common/collect/Iterables
at org.opendaylight.controller.liblldp.LLDP.getLLDPPacketLength(LLDP.java:288)
at org.opendaylight.controller.liblldp.LLDP.serialize(LLDP.java:254)
at org.opendaylight.controller.liblldp.Packet.serialize(Packet.java:161)
at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPUtil.buildLldpFrame(LLDPUtil.java:97)
at org.opendaylight.openflowplugin.applications.lldpspeaker.LLDPSpeaker.nodeConnectorAdded(LLDPSpeaker.java:152)
at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.notifyNodeConnectorAppeared(NodeConnectorInventoryEventTranslator.java:104)
at org.opendaylight.openflowplugin.applications.lldpspeaker.NodeConnectorInventoryEventTranslator.onDataChanged(NodeConnectorInventoryEventTranslator.java:71)
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:134)
at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)
at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

====

[14:56:40] <flaviof> folks ever seen an issue like this: https://gist.github.com/67d6c0f67d4597862492 ?
[14:56:49] <flaviof> vishnoianil_: shague: edwarnicke: ^^
[14:58:23] <vishnoianil_> flaviof, looks like your jvm is running out of memory
[14:58:48] <flaviof> vishnoianil_: ok! I can see that.... lots of open windows here
[14:58:50] <flaviof> thanks!
[14:59:10] <vishnoianil_> flaviof, but i thought it used to die silently before and there were some discussion on catching those events and try to gracefully shut down the JVM
[14:59:14] <vishnoianil_> and looks like that's implemented
[14:59:24] <vishnoianil_> Moiz or Tom can give you more details about it
[14:59:54] <flaviof> ack. Let me kill some windows and see if that helps
[15:00:10] <flaviof> thanks vishnoianil_ !
[15:41:08] <alagalah> flaviof: No but we just saw this
[15:41:09] <alagalah> https://gist.github.com/7b9c49ebf8a4ce3d45fd
[15:41:26] <alagalah> flaviof: Happens if you connect a switch, or if the switch is already set to connect and you fire up controller, it goes boom
[15:42:44] <flaviof> alagalah: that is exactly the trace I'm seeing. funny how I just started seeing it in the last hour or so, and changed nothing
[15:43:07] <flaviof> alagalah: so is this a bug in akka?
[15:48:21] <alagalah> flaviof: Yeah controller
[15:48:21] <alagalah> I think
[15:48:28] <alagalah> flaviof: like theres 20 patches this morning
[15:48:34] <edwarnicke> flaviof: Looking
[15:48:49] <edwarnicke> flaviof: Could you create a bug with reproduction steps (so we can have folks poke it )
[15:49:01] <alagalah> flaviof: I was just building with -U to clear out some crud from where I was playing with UT and ka-bloom
[15:49:30] <flaviof> edwarnicke: ty. yes, if that is the way to go. alagalah: is there no open bugs on that bad boy?
[15:50:11] <alagalah> flaviof: Just hit it and using controller for something else on webex at the mo'
[15:50:31] <alagalah> flaviof: So no, haven't logged a bug, just reached out to see if you hit this and you had
[15:50:43] <flaviof> alagalah: ack. any chance you are running out of memory (like me)?
[15:50:47] <edwarnicke> alagalah: flaviof Random Data: liblldp hasn't changed since Mar 17
[15:51:06] <flaviof> edwarnicke: that is good info
[15:51:17] <edwarnicke> flaviof: How would I reproduce it?
[15:54:16] <edwarnicke> vishnoianil_: See my comment here: https://git.opendaylight.org/gerrit/#/c/21242/
[15:54:19] <edwarnicke> vishnoianil_: I think you are fine
[15:54:20] <flaviof> edwarnicke:heh, simply 1)start odl; 2) doing a stack
[15:54:27] <edwarnicke> vishnoianil_: In fact I think you are doing good at the place I commented
[15:54:35] <edwarnicke> vishnoianil_: Just wanted to call out a dangerous curve
[15:54:42] <edwarnicke> flaviof: Do you have something simpler
[15:55:07] <flaviof> edwarnicke: i wish.
[15:55:12] <edwarnicke> flaviof: From the error, it sounds like simply pointing a OF switch at it might do it
[15:55:14] <edwarnicke> flaviof: Let me try that
[15:55:45] <flaviof> edwarnicke:it is not 100% reproducible; man I hate those....
[15:56:05] <edwarnicke> flaviof: Good to know
[15:56:32] <edwarnicke> flaviof: Has ovsdb cut stable/lithium yet (not calling you out, just want to know what branch to try on)
[15:57:27] <flaviof> edwarnicke: correct. I'm on latest master in ovsdb.
[15:57:39] <edwarnicke> flaviof: Cool, that's what I am building now
[15:57:43] <flaviof> edwarnicke: thinking we will merge you gerrits and then cut the branch.
[15:57:55] <flaviof> edwarnicke: will make you click less, right?
[15:58:00] <edwarnicke> flaviof: I'm fine with that overall, but again, just looking to be reproducing the right way
[15:58:09] <flaviof> ack
[15:58:12] edwarnicke fails to whine about the five clicks
[15:58:18] <flaviof> lol
[15:58:49] <edwarnicke> flaviof: Also chasing a small bug in my patch stack... I forgot the where clause when iid stamping bridges
[15:59:19] <flaviof> edwarnicke:ack. thanks for the chase
[15:59:31] <edwarnicke> flaviof: My pleasure
[15:59:36] <edwarnicke> flaviof: Hopefully I'll figure something out
[15:59:44] <edwarnicke> flaviof: I can think of several places we might be having the issue
[15:59:58] <edwarnicke> flaviof: I honestly don't think its in clustering (I suspect they are just the ones catching the exception and logging it)
[16:00:02] <edwarnicke> flaviof: But I could be wrong
[16:00:04] <vishnoianil_> edwarnicke : just responded to your comment
[16:00:16] <vishnoianil_> you mean not to use */bridge/<bridge-name> format?
[16:00:37] <vishnoianil_> and use something else */net-virt-bridge/<bridge-name>?
[16:03:01] <edwarnicke> vishnoianil_: I responded to your response
[16:03:08] <edwarnicke> vishnoianil_: I do want to be crystal clear though
[16:03:15] <edwarnicke> vishnoianil_: I am not saying what you did there was wrong
[16:03:21] <edwarnicke> vishnoianil_: In fact I like what you did there
[16:03:32] <edwarnicke> vishnoianil_: I am just calling out a point to be cautious about in the future
[16:06:22] <vishnoianil_> edwarnicke, okay got it, we kind of take care of that in net-virt code

OpenDaylight JIRA

[CONTROLLER-1342] Uncaught error from thread [opendaylight-cluster-data-akka.actor.default-dispatcher-90] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data] Uncaught error from thread [opendaylight-cluster-data-akka