[NETCONF-327] Incorrect handling of device transactions in clustered setting Created: 05/Dec/16  Updated: 15/Mar/19  Resolved: 17/Feb/17

Status: Resolved
Project: netconf
Component/s: netconf
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Jakub Morvay Assignee: Jakub Morvay
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 7295

 Description   

In clustered setting mount point's transactions are delegated to one node in cluster and that node actually configures the device.

The implementation however depends only on one write transaction at a time. So the delegated calls can interleave together, for example one node can submit transaction of the other node, etc.

 Comments   
Comment by Vladimir Lavor [ 05/Dec/16 ]

The log error looks like re-usage of closed transaction.

Example:
2016-11-30 16:26:42,824 | ERROR | on-dispatcher-44 | DataTreeChangeListenerActor | 205 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingClusteredDOMDataTreeChangeListenerAdapter@1bc842c8
java.lang.IllegalStateException: RemoteDevice

{overcloud-controller-0.opnfvapex.com}

: Transaction org.opendaylight.netconf.sal.connect.netconf.sal.tx.WriteCandidateTx@395c290c already finished

Cannot reliably reproduce, appears randomly when transaction.put/read/delete/submit etc. us used.

Possible workaround is to close existing transaction and create a new one when this error is caught - this helps in most cases. Also ensure that only one netconf transaction is opened at a time.

Tested with netconf-clustered-topology

Comment by Jakub Morvay [ 05/Dec/16 ]

I think, the one reliable workaround is to have just one transaction open at once.
You shouldn't see any problem with reused transactions than.

Comment by Robert Varga [ 06/Dec/16 ]

Please report full exception stack.

Comment by Robert Varga [ 06/Dec/16 ]

Without the stack trace, it is quite hard to track down what exactly is happening and what triggered the ISE.

Comment by Vladimir Lavor [ 06/Dec/16 ]

Full exception stack:

2016-11-30 16:35:16,896 | ERROR | on-dispatcher-78 | DataTreeChangeListenerActor | 205 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingClusteredDOMDataTreeChangeListenerAdapter@8aec786
java.lang.IllegalStateException: RemoteDevice

{overcloud-controller-0.opnfvapex.com}

: Transaction org.opendaylight.netconf.sal.connect.netconf.sal.tx.WriteCandidateTx@4dac5057 already finished
at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[65:com.google.guava:18.0.0]
at org.opendaylight.netconf.sal.connect.netconf.sal.tx.AbstractWriteTx.checkNotFinished(AbstractWriteTx.java:66)[310:org.opendaylight.netconf.sal-netconf-connector:1.5.0.SNAPSHOT]
at org.opendaylight.netconf.sal.connect.netconf.sal.tx.AbstractWriteTx.commit(AbstractWriteTx.java:140)[310:org.opendaylight.netconf.sal-netconf-connector:1.5.0.SNAPSHOT]
at org.opendaylight.netconf.sal.connect.netconf.sal.tx.WriteCandidateTx.submit(WriteCandidateTx.java:97)[310:org.opendaylight.netconf.sal-netconf-connector:1.5.0.SNAPSHOT]
at org.opendaylight.netconf.topology.singleton.impl.tx.NetconfMasterDOMTransaction.submit(NetconfMasterDOMTransaction.java:154)[313:org.opendaylight.netconf.topology-singleton:1.2.0.SNAPSHOT]
at org.opendaylight.netconf.topology.singleton.impl.tx.NetconfWriteOnlyTransaction.submit(NetconfWriteOnlyTransaction.java:84)[313:org.opendaylight.netconf.topology-singleton:1.2.0.SNAPSHOT]
at org.opendaylight.netconf.sal.connect.netconf.sal.tx.ReadWriteTx.submit(ReadWriteTx.java:59)[310:org.opendaylight.netconf.sal-netconf-connector:1.5.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:134)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.submit(BindingDOMWriteTransactionAdapter.java:83)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.0.SNAPSHOT]
at org.opendaylight.vbd.impl.VppModifier.createVirtualInterfaceOnVpp(VppModifier.java:432)[317:org.opendaylight.honeycomb.vbd.impl:1.1.0.SNAPSHOT]
at org.opendaylight.vbd.impl.VbdBridgeDomain.addVxlanTunnel(VbdBridgeDomain.java:700)[317:org.opendaylight.honeycomb.vbd.impl:1.1.0.SNAPSHOT]
at org.opendaylight.vbd.impl.VbdBridgeDomain.modifyNode(VbdBridgeDomain.java:511)[317:org.opendaylight.honeycomb.vbd.impl:1.1.0.SNAPSHOT]
at org.opendaylight.vbd.impl.VbdBridgeDomain.handleModifiedChildren(VbdBridgeDomain.java:459)[317:org.opendaylight.honeycomb.vbd.impl:1.1.0.SNAPSHOT]
at org.opendaylight.vbd.impl.VbdBridgeDomain.onDataTreeChanged(VbdBridgeDomain.java:428)[317:org.opendaylight.honeycomb.vbd.impl:1.1.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)[173:org.opendaylight.controller.sal-binding-broker-impl:1.5.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:54)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:35)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:28)[205:org.opendaylight.controller.sal-clustering-commons:1.5.0.SNAPSHOT]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)[186:com.typesafe.akka.actor:2.4.11]
at akka.actor.Actor$class.aroundReceive(Actor.scala:484)[186:com.typesafe.akka.actor:2.4.11]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)[186:com.typesafe.akka.actor:2.4.11]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[186:com.typesafe.akka.actor:2.4.11]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[186:com.typesafe.akka.actor:2.4.11]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[186:com.typesafe.akka.actor:2.4.11]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[186:com.typesafe.akka.actor:2.4.11]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[186:com.typesafe.akka.actor:2.4.11]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[181:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[181:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[181:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[181:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]

Comment by Jakub Morvay [ 14/Feb/17 ]

NETCONF-327 - Incorrect handling of device transactions in clustered setting

Comment by Jakub Morvay [ 14/Feb/17 ]

(In reply to Jakub Morvay from comment #6)
> NETCONF-327 - Incorrect handling of device transactions in clustered setting

Sorry for invalid comment..

https://git.opendaylight.org/gerrit/51875

Generated at Wed Feb 07 20:14:44 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.