[BGPCEP-689] ConcurrentModificationException when closing bgp-state-provider-service-group service instance

Sun, 3 Mar 2019 11:50:05 +0000

This has been discovered when investigating a failure in CSIT. Not sure whether this is the primary reason for the failure, but such exceptions should not hapen anyway.

The test isolates and rejoins a cluster member while ExaBGP is connected, the failure [0] happens when Robo reads bgp-rib for the peer just after the rejoin:

{"errors":{"error":[

{"error-type":"application","error-tag":"data-missing","error-message":"Request could not be completed because the relevant data model content does not exist "}

]}}

The affected member-1 contains the exception is its karaf.log [1]:

2017-09-13 07:34:20,705 | WARN | ult-dispatcher-4 | ClusterSingletonServiceGroupImpl | 288 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0 | Service group bgp-state-provider-service-group service org.opendaylight.protocol.bgp.state.StateProviderImpl@2d0e1d16 failed to stop, attempting to continue
java.util.ConcurrentModificationException
at java.util.HashMap$KeySet.forEach(HashMap.java:935)[:1.8.0_141]
at org.opendaylight.protocol.bgp.state.StateProviderImpl.closeServiceInstance(StateProviderImpl.java:177)[190:org.opendaylight.bgpcep.bgp-openconfig-state:0.8.0]
at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.stopServices(ClusterSingletonServiceGroupImpl.java:515)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lostOwnership(ClusterSingletonServiceGroupImpl.java:487)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.serviceOwnershipChanged(ClusterSingletonServiceGroupImpl.java:410)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lockedOwnershipChanged(ClusterSingletonServiceGroupImpl.java:354)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ownershipChanged(ClusterSingletonServiceGroupImpl.java:337)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.mdsal.singleton.dom.impl.AbstractClusterSingletonServiceProviderImpl.ownershipChanged(AbstractClusterSingletonServiceProviderImpl.java:234)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl.ownershipChanged(DOMClusterSingletonServiceProviderImpl.java:23)[288:org.opendaylight.mdsal.singleton-dom-impl:2.3.0]
at org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.onEntityOwnershipChanged(EntityOwnershipListenerActor.java:44)[237:org.opendaylight.controller.sal-distributed-datastore:1.6.0]
at org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.handleReceive(EntityOwnershipListenerActor.java:33)[237:org.opendaylight.controller.sal-distributed-datastore:1.6.0]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38)[230:org.opendaylight.controller.sal-clustering-commons:1.6.0]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)[35:com.typesafe.akka.actor:2.4.18]
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)[35:com.typesafe.akka.actor:2.4.18]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)[35:com.typesafe.akka.actor:2.4.18]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[35:com.typesafe.akka.actor:2.4.18]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[35:com.typesafe.akka.actor:2.4.18]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[35:com.typesafe.akka.actor:2.4.18]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[35:com.typesafe.akka.actor:2.4.18]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[35:com.typesafe.akka.actor:2.4.18]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[323:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[323:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[323:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[323:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]

[0] https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-ha-only-nitrogen/152/log.html.gz#s1-s2-t10-k2-k2
[1] https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-ha-only-nitrogen/152/odl1_karaf.log.gz

OpenDaylight JIRA

[BGPCEP-689] ConcurrentModificationException when closing bgp-state-provider-service-group service instance