[NETCONF-479] OptimisticLockFailedException on controller-config when both odl-netconf-connector-ssh and odl-netconf-clustered-topology are installed Created: 17/Oct/17  Updated: 15/Mar/19  Resolved: 01/Oct/18

Status: Resolved
Project: netconf
Component/s: netconf
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Vratko Polak Assignee: Jakub Morvay
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 9307

 Description   

This happens only rarely, but this can make autorelease build fail, by failing SingleFeatureTest on some Distribution feature.

Distribution knows the two features are incompatible when running on 3 node cluster, but this is failing in 1 node tests. Nevertheless, both features are pulled into odl-integration-compatible-with-all in current Oxygen builds:
odl-netconf-connector-ssh is pulled in by odl-unimgr,
odl-netconf-clustered-topology is pulled in by odl-vbd.

Looking at the log [0] from the SFT failure, we see this (after removing lines from unrelated features):

2017-10-17 05:36:00,548 | INFO | on-dispatcher-54 | NetconfTopologyUtils | 448 - org.opendaylight.netconf.topology-singleton - 1.4.0 | RemoteDevice

{controller-config} : using the default directory cache/schema
2017-10-17 05:36:00,564 | INFO | lt-dispatcher-22 | NetconfTopologyContext | 448 - org.opendaylight.netconf.topology-singleton - 1.4.0 | Master was selected: IpAddress [_ipv4Address=Ipv4Address [_value=127.0.0.1]]
2017-10-17 05:36:00,583 | INFO | on-dispatcher-50 | AbstractNetconfTopology | 194 - netconf-topology-config - 1.4.0 | Connecting RemoteDevice{Uri [_value=controller-config]} , with config Node{getNodeId=Uri [_value=controller-config], augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.netconf.node.topology.rev150114.NetconfNode=NetconfNode{getActorResponseWaitTime=5, getBetweenAttemptsTimeoutMillis=2000, getConcurrentRpcLimit=0, getConnectionTimeoutMillis=20000, getCredentials=LoginPassword{getPassword=admin, getUsername=admin, augmentations={}}, getDefaultRequestTimeoutMillis=60000, getHost=Host [_ipAddress=IpAddress [_ipv4Address=Ipv4Address [_value=127.0.0.1]]], getKeepaliveDelay=120, getMaxConnectionAttempts=0, getPort=PortNumber [_value=1830], getSchemaCacheDirectory=schema, getSleepFactor=1.5, isReconnectOnChangedSchema=true, isSchemaless=false, isTcpOnly=false}}}
2017-10-17 05:36:00,654 | INFO | lt-dispatcher-22 | RemoteDeviceConnectorImpl | 448 - org.opendaylight.netconf.topology-singleton - 1.4.0 | RemoteDevice{controller-config}

: Adding keepalive facade.
2017-10-17 05:36:00,697 | WARN | on-dispatcher-50 | AbstractNetconfTopology | 194 - netconf-topology-config - 1.4.0 | Adding keepalive facade, for device Uri [_value=controller-config]
2017-10-17 05:36:00,701 | WARN | rd-dispatcher-41 | ShardDataTree | 354 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0 | member-1-shard-topology-operational: Store Tx member-1-datastore-operational-fe-0-chn-11-txn-0-0: Conflicting modification for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=topology-netconf}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=controller-config}

].

which ultimately leads to:
2017-10-17 05:36:00,792 | ERROR | ime-limited test | SingleFeatureTest | 75 - PAXEXAM-PROBE-2f9f5f86-a820-45e5-b6ef-75e8a38bd5d0 - 0.0.0 | installFeature() failed
java.lang.IllegalStateException: RemoteDevice

{controller-config}

Transaction(init) not committed correctly
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceTopologyAdapter$1.onFailure(NetconfDeviceTopologyAdapter.java:244)[443:org.opendaylight.netconf.sal-netconf-connector:1.7.0]
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1228)[77:com.google.guava:22.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_141]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_141]

Can Netconf make the two features compatible, or should Distribution split their test features and fail if 1-node Netconf feature is found in 3-node Distribution feature tree, and vice versa?

[0] https://logs.opendaylight.org/releng/jenkins092/autorelease-release-oxygen/19/integration/distribution/features/singles/odl-integration-compatible-with-all/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz



 Comments   
Comment by Tomas Cere [ 17/Oct/17 ]

There is no reason to pull in odl-netconf-connector-ssh unless you are using config subsystem still. unimgr should move towards odl-netconf-topology.
odl-netconf-clustered-topology and odl-netconf-topology are still considered incompatible but unless you are spawning a netconf connector which shouldnt happen in SFT it wont fail it.

Comment by Michael Vorburger [ 26/Oct/17 ]

distribution jobs fail A LOT due to this, and IMHO that's a PITA that is blocking productivity for everyone, and untenable.

unimgr-dev has not reacted to https://lists.opendaylight.org/pipermail/unimgr-dev/2017-October/000470.html in 24 hours, and I would therefore like to propose https://git.opendaylight.org/gerrit/#/c/64761/ to work around this. Objections, anyone?

Comment by Vratko Polak [ 26/Oct/17 ]

Currently I think Change-Id I9c80c4dd5c43365d4d46f2b5bf89dd1f0f22f57f [3] is stable enough (and better overall).

[3] https://git.opendaylight.org/gerrit/#/q/I9c80c4dd5c43365d4d46f2b5bf89dd1f0f22f57f

Comment by Michael Vorburger [ 27/Oct/17 ]

https://git.opendaylight.org/gerrit/#/c/64797/ ?

Comment by Roshni kk [ 16/Mar/18 ]

While installing the feature odl-netconf-clustered-topology (on a single node topology), the issue is still seen in Nitrogen SR2 also.

 

Please find the karaf.log

====================

Exception in thread "CommitFutures-2" java.lang.IllegalStateException: RemoteDevice{controller-config}  Transaction(init) not committed correctly
        at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceTopologyAdapter$1.onFailure(NetconfDeviceTopologyAdapter.java:243)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1228)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
Exception in thread "CommitFutures-0"   at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:731)
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:769)
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:716)
        at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:799)
        at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:90)
        at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97)
        at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236)
        at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200)
        at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:729)
        at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333)
        at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270)
        at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44)
        at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)
        at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)
        at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
        at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)
        at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727)
        at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)
        at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:174)
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128)
        at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)

 

 

 

 

Comment by Jakub Morvay [ 01/Oct/18 ]

This should be fixed with https://git.opendaylight.org/gerrit/#/c/64797/

Nitrogen is no longer supported, closing this issue.

Generated at Wed Feb 07 20:15:08 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.