[OPNFLWPLUG-887] After karaf restart OF writes fail due to "No queue present, failing request" in OutboundQueueProviderImpl Created: 09/May/17  Updated: 27/Sep/21  Resolved: 17/May/17

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Josh Hershberg Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 8401
Priority: Highest

 Description   

Scenario:
1. Bring up openstack+netvirt
2. restart karaf
3. restart ovs
4. note that flows are not reinstalled

From debugging it seems that the outbound queue is null, "No queue present, failing request" in the log. The queue is initially allocated but then removed due to the following stack trace (I added this, no exception was actually thrown), note the second to last line, "onNotAbleToStartMastership":
3903 at org.opendaylight.openflowplugin.impl.connection.OutboundQueueProviderImpl.onConnectionQueueChanged(OutboundQueueProviderImpl.java:45)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
3904 at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractOutboundQueueManager.close(AbstractOutboundQueueManager.java:111)[295:org.opendaylight.openflowjava.openflow-protocol-impl:0.10.0.SNAPSHOT]
3905 at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl$2.removeRegistration(ConnectionAdapterImpl.java:210)[295:org.opendaylight.openflowjava.openflow-protocol-impl:0.10.0.SNAPSHOT]
3906 at org.opendaylight.yangtools.concepts.AbstractRegistration.close(AbstractRegistration.java:43)[67:org.opendaylight.yangtools.concepts:1.2.0.SNAPSHOT]
3907 at org.opendaylight.openflowplugin.impl.connection.ConnectionContextImpl.unregisterOutboundQueue(ConnectionContextImpl.java:201)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
3908 at org.opendaylight.openflowplugin.impl.connection.ConnectionContextImpl.closeConnection(ConnectionContextImpl.java:117)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
3909 at org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl.close(ContextChainImpl.java:107)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
3910 at org.opendaylight.openflowplugin.impl.lifecycle.ContextChainHolderImpl.destroyContextChain(ContextChainHolderImpl.java:157)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
3911 at org.opendaylight.openflowplugin.impl.lifecycle.ContextChainHolderImpl.onNotAbleToStartMastership(ContextChainHolderImpl.java:203)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
3912 at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$6.onSuccess(StatisticsContextImpl.java:546)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]

 Comments   
Comment by Josh Hershberg [ 10/May/17 ]

The transaction that fails that causes onNotAbleToStartMastership to be called is:

java.util.concurrent.ExecutionException: TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalArgumentException: Metadata not available for modification NodeModification [identifier=(urn:opendaylight:flow:inventory?revision=2013-08-19)group[

{(urn:opendaylight:flow:inventory?revision=2013-08-19)group-id=5000}

], modificationType=TOUCH, childModification={AugmentationIdentifier

{childNames=[(urn:opendaylight:group:statistics?revision=2013-11-11)group-statistics]}

=NodeModification [identifier=AugmentationIdentifier

{childNames=[(urn:opendaylight:group:statistics?revision=2013-11-11)group-statistics]}

, modificationType=TOUCH, childModification={(urn:opendaylight:group:statistics?revision=2013-11-11)group-statistics=NodeModification [identifier=(urn:opendaylight:group:statistics?revision=2013-11-11)group-statistics, modificationType=WRITE, childModification={}]}]}]]]}
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)[65:com.google.guava:18.0.0]
at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.submitWriteTransaction(TransactionChainManager.java:181)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.initialSubmitWriteTransaction(TransactionChainManager.java:95)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.initialSubmitTransaction(DeviceContextImpl.java:228)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$6.onSuccess(StatisticsContextImpl.java:537)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$6.onSuccess(StatisticsContextImpl.java:529)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)[65:com.google.guava:18.0.0]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl.statChainFuture(StatisticsContextImpl.java:307)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl.access$200(StatisticsContextImpl.java:57)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$3.onSuccess(StatisticsContextImpl.java:296)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$3.onSuccess(StatisticsContextImpl.java:293)[307:org.opendaylight.openflowplugin.impl:0.5.0.SNAPSHOT]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture$1.run(Futures.java:918)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:914)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)[65:com.google.guava:18.0.0]
at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)[65:com.google.guava:18.0.0]

Comment by Tomas Slusny [ 11/May/17 ]

It is possible that this is happening because of device initialization future, I created patch that should solve this (also with explanation) here: https://git.opendaylight.org/gerrit/#/c/56838/

Can you try it with that patch and tell if it helps?

Comment by Tomas Slusny [ 15/May/17 ]

As https://bugs.opendaylight.org/show_bug.cgi?id=8401 was merged on master and Joshua confirmed that with that patch scenario here seems to work fine, this can be set to waiting until it will be merged also on stable/carbon here: https://git.opendaylight.org/gerrit/#/c/56927/

Comment by A H [ 15/May/17 ]

Patch https://git.opendaylight.org/gerrit/#/c/56927/ was successfully merged. Can we mark this bug as fixed and resolved?

Comment by A H [ 16/May/17 ]

Another patch was submitted in relation to this bug: https://git.opendaylight.org/gerrit/#/c/57096/

Generated at Wed Feb 07 20:33:39 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.