https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1324 netvirt <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:53688268439021}]/AugmentationIdentifier{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-group, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)port-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)table-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=20}]/flow/flow[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=L3.53688268439021.20.0L3.}] </pre> </div></div> NETVIRT-1324

OptimisticLockFailedException.../flow/flow...id=L3.53688268439021.20.0L3

Bug Lowest Resolved Done Michael Vorburger Sam Hague csit:3node Tue, 19 Jun 2018 21:16:02 +0000 Mon, 17 Sep 2018 19:13:57 +0000 Mon, 17 Sep 2018 19:13:57 +0000 Fluorine Neon 0 3 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> is this just an exception from CSIT that we turned in to a jira? No real context to go on<br/> right? Will someone be able to address it with just the exception line?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> I could probably help with this this one (by adding retry), but the log above is too minimalistic to be actionable - we would need a full stack trace, to see where in the code that OptimisticLockFailedException comes from. If that's not available, I propose that you close this issue, and a new one be opened with details, when available.</p> <p>I normally add the job and the karaf logs that this came from. I will try to dig it up.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> adding a retry - does that mean an exception would not come out until the last retry failed? Similar to other exceptions you fixed a while back with the retry framework?</p> <p>Other question: that is the right behavior, right? OLFE is not a problem in itself since you could have multiple writers, so you retry and hope it passes. Or is this a design issue that should be fixed not to have multiple writers to the same tree data?</p> <p>&gt; does that mean an exception would not come out until the last retry failed? </p> <p>yup; that's what the <a href="https://github.com/opendaylight/genius/blob/master/mdsalutil/mdsalutil-api/src/main/java/org/opendaylight/genius/infra/RetryingManagedNewTransactionRunner.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/genius/blob/master/mdsalutil/mdsalutil-api/src/main/java/org/opendaylight/genius/infra/RetryingManagedNewTransactionRunner.java</a> does (see <a href="https://github.com/opendaylight/genius/blob/master/mdsalutil/mdsalutil-api/src/main/java/org/opendaylight/genius/infra/RetryingManagedNewTransactionRunnerImpl.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/genius/blob/master/mdsalutil/mdsalutil-api/src/main/java/org/opendaylight/genius/infra/RetryingManagedNewTransactionRunnerImpl.java</a> for details)</p> <p>&gt;  that is the right behavior, right? OLFE is not a problem in itself since you could have multiple writers, so you retry and hope it passes.</p> <p>It will fix the problem, is very easy to do (you should try doing it yourself!), and is what we have done elsewhere.</p> <p>&gt; Or is this a design issue that should be fixed not to have multiple writers to the same tree data?</p> <p>You can also fix this issue that way, but that is a much bigger design change (if it's even possible).</p> <p>this is still happening, but it's possible that it's not affecting functionality. Without more context, it will be hard for someone like <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> to look at it.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> can you dig up a karaf.log that has this?</p> <p>still happening, will add further triage: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/31/odl_1/odl1_err_warn_exception.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/31/odl_1/odl1_err_warn_exception.log.gz</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-08-13T05:08:13,967 | WARN | opendaylight-cluster-data-akka.actor.default-dispatcher-5 | LocalThreePhaseCommitCohort | 235 - org.opendaylight.controller.sal-distributed-datastore - 1.7.4 | Failed to prepare transaction member-1-datastore-config-fe-0-txn-112-0 on backend org.opendaylight.controller.md.sal.common.api.data.OptimisticLockFailedException: Optimistic lock failed for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:119324593392072}]/AugmentationIdentifier{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-group, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)port-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)table-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=20}]/flow/flow[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=L3.119324593392072.20.0L3.}] Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction. 2018-08-13T05:08:32,925 | WARN | pool-89-thread-1 | OutboundQueueEntry | 389 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.6.4 | Can't commit a completed message. 2018-08-13T05:08:32,925 | WARN | pool-89-thread-1 | HashedWheelTimer | 62 - io.netty.common - 4.1.22.Final | An exception was thrown by TimerTask. java.lang.NullPointerException: null 2018-08-13T05:09:00,106 | WARN | epollEventLoopGroup-9-2 | HandshakeManagerImpl | 380 - org.opendaylight.openflowplugin.impl - 0.6.4 | issuing disconnect during handshake [/10.30.170.201:56960] </pre> </div></div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> et al, so as discussed during the Kernel Projects call today:</p> <p>In <a href="https://jira.opendaylight.org/browse/CONTROLLER-1855" title="Controller should not error log any OptimisticLockFailedException which applications actually handle" class="issue-link" data-issue-key="CONTROLLER-1855"><del>CONTROLLER-1855</del></a>, it was decided that WARN from LocalThreePhaseCommitCohort was "benign" and could be ignored - if applications handled it; in this case, whatever writes to opendaylight:flow:inventory needs to retry the failed future; see below. (I had argued in the discussion of that bug that it should therefore just be an INFO or even DEBUG intead of a WARN; but it was decided that it's safer to keep it as a WARN, but not ERROR.)</p> <p>So the (first; others are totally unrelated to this issue!) WARN shown above should be ignored. What still does need to be looked into in this issue is these two ERROR logs; this first one is directly related to that WARN on opendaylight:flow:inventory:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-08-13T05:42:03,918 | ERROR | CommitFutures-30 | JobCoordinatorImpl | 281 - org.opendaylight.infrautils.jobcoordinator-impl - 1.3.4 | Job still failed on final retry: JobEntry{key='MAC-FA:16:3E:D2:50:A7 ELAN_TAG-5006DPN_ID-119324593392072', mainWorker=org.opendaylight.netvirt.elan.internal.ElanPacketInHandler$$Lambda$1757/325670119@164ed935, rollbackWorker=null, retryCount=3/3, futures=[org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$AsyncNotifyingSettableFuture@534c5598[status=FAILURE, cause=[OptimisticLockFailedException{message=Optimistic lock failed for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:119324593392072}]/AugmentationIdentifier{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-group, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)port-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)table-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=50}]/flow/flow[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=50500611932459339207227FA:16:3E:D2:50:A7}], errorList=[RpcError [message=Optimistic lock failed for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:119324593392072}]/AugmentationIdentifier{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-group, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)port-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)table-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=50}]/flow/flow[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=50500611932459339207227FA:16:3E:D2:50:A7}], severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was replaced by other transaction.]]}]]]}</pre> </div></div> <p>whereas this other one is separate:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-08-13T07:29:52,155 | ERROR | opendaylight-cluster-data-notification-dispatcher-76 | UpgradeUtils | 266 - org.opendaylight.genius.mdsalutil-impl - 0.4.4 | Unable to update UpgradeState java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed for path /(urn:opendaylight:serviceutils:upgrade?revision=2018-07-02)upgrade-config, errorList=[RpcError [message=Optimistic lock failed for path /(urn:opendaylight:serviceutils:upgrade?revision=2018-07-02)upgrade-config, severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was replaced by other transaction.]]} Caused by: org.opendaylight.controller.md.sal.common.api.data.OptimisticLockFailedException: Optimistic lock failed for path /(urn:opendaylight:serviceutils:upgrade?revision=2018-07-02)upgrade-config Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was replaced by other transaction.</pre> </div></div> <p><a href="https://jira.opendaylight.org/browse/GENIUS-207" title="UpgradeUtils OptimisticLockFailedException" class="issue-link" data-issue-key="GENIUS-207"><del>GENIUS-207</del></a> will track the solution to the OLFE in genius UpgradeState.</p> <p>This bug will track netvirt.elan.internal.ElanPacketInHandler flow inventory job OLFE.</p> Relates CONTROLLER-1855 NETVIRT-1015 Development Epic Link NETVIRT-996 Rank 0|i03frj: