[OVSDB-468] Extracting commands was interrupted.java.lang.InterruptedException: null Created: 17/Sep/18  Updated: 24/Jan/24  Resolved: 24/Jan/24

Status: Resolved
Project: ovsdb
Component/s: Clustering
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Sam Hague Assignee: Robert Varga
Resolution: Done Votes: 0
Labels: csit:3node, csit:exception
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File odl1_karaf.log.tar.xz    
Issue Links:
Blocks
is blocked by AAA-268 Add Provide-Capability to aaa-cert Resolved
Epic Link: Clustering Stability

 Description   

During 3node testing using graceful start and stop the following exception is seen. Graceful start and stop means using the bin/stop and start commands to stop and start ODL rather than using kill -9. This means there is an orderly stop to the bundles where each bundle is destroyed. Some bundles in ovsdb are destroyed and the exceptions start.

The flow is all three ODLs are up. Then shutdown ODL1 via bin/stop. The exception comes out.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/408/shague-haproxy-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/8/odl_1/odl1_karaf.log.gz

2018-09-17T16:51:16,985 | WARN  | transaction-invoker-impl-0 | TransactionInvokerImpl           | 393 - org.opendaylight.ovsdb.southbound-impl - 1.8.0.SNAPSHOT | Extracting commands was interrupted.java.lang.InterruptedException: null
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) [?:?]
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) [?:?]
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) [?:?]
	at org.opendaylight.ovsdb.southbound.transactions.md.TransactionInvokerImpl.extractCommandsFromQueue(TransactionInvokerImpl.java:174) [393:org.opendaylight.ovsdb.southbound-impl:1.8.0.SNAPSHOT]
	at org.opendaylight.ovsdb.southbound.transactions.md.TransactionInvokerImpl.extractCommands(TransactionInvokerImpl.java:168) [393:org.opendaylight.ovsdb.southbound-impl:1.8.0.SNAPSHOT]
	at org.opendaylight.ovsdb.southbound.transactions.md.TransactionInvokerImpl.run(TransactionInvokerImpl.java:87) [393:org.opendaylight.ovsdb.southbound-impl:1.8.0.SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
	at java.lang.Thread.run(Thread.java:748) [?:?]

2018-09-17T16:51:17,005 | WARN  | nioEventLoopGroup-5-3 | DefaultPromise                   | 55 - io.netty.common - 4.1.22.Final | An exception was thrown by org.opendaylight.ovsdb.lib.impl.ChannelConnectionHandler.operationComplete()org.osgi.service.blueprint.container.ServiceUnavailableException: The Blueprint container is being or has been destroyed: (&(type=default)(objectClass=org.opendaylight.controller.md.sal.binding.api.DataBroker))
	at org.apache.aries.blueprint.container.ReferenceRecipe.getService(ReferenceRecipe.java:241) [75:org.apache.aries.blueprint.core:1.8.3]
	at org.apache.aries.blueprint.container.ReferenceRecipe.access$000(ReferenceRecipe.java:56) [75:org.apache.aries.blueprint.core:1.8.3]
	at org.apache.aries.blueprint.container.ReferenceRecipe$ServiceDispatcher.call(ReferenceRecipe.java:306) [75:org.apache.aries.blueprint.core:1.8.3]
	at Proxy4966181c_caff_4391_9c63_f6d49059a6f2.newReadOnlyTransaction(Unknown Source) [?:?]
	at org.opendaylight.ovsdb.southbound.OvsdbConnectionManager.retryConnection(OvsdbConnectionManager.java:617) [393:org.opendaylight.ovsdb.southbound-impl:1.8.0.SNAPSHOT]
	at org.opendaylight.ovsdb.southbound.OvsdbConnectionManager.disconnected(OvsdbConnectionManager.java:186) [393:org.opendaylight.ovsdb.southbound-impl:1.8.0.SNAPSHOT]
	at org.opendaylight.ovsdb.lib.impl.OvsdbConnectionService.channelClosed(OvsdbConnectionService.java:492) [389:org.opendaylight.ovsdb.library:1.8.0.SNAPSHOT]
	at org.opendaylight.ovsdb.lib.impl.ChannelConnectionHandler.operationComplete(ChannelConnectionHandler.java:25) [389:org.opendaylight.ovsdb.library:1.8.0.SNAPSHOT]
	at org.opendaylight.ovsdb.lib.impl.ChannelConnectionHandler.operationComplete(ChannelConnectionHandler.java:15) [389:org.opendaylight.ovsdb.library:1.8.0.SNAPSHOT]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) [55:io.netty.common:4.1.22.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) [55:io.netty.common:4.1.22.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) [55:io.netty.common:4.1.22.Final]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) [55:io.netty.common:4.1.22.Final]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1356) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38) [58:io.netty.transport:4.1.22.Final]
	at io.netty.channel.AbstractChannelHandlerContext$12.run(AbstractChannelHandlerContext.java:576) [58:io.netty.transport:4.1.22.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [55:io.netty.common:4.1.22.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [55:io.netty.common:4.1.22.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [58:io.netty.transport:4.1.22.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) [55:io.netty.common:4.1.22.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [55:io.netty.common:4.1.22.Final]
	at java.lang.Thread.run(Thread.java:748) [?:?]


 Comments   
Comment by Robert Varga [ 01/Feb/23 ]

Right, so this is activation and shutdown paths are just up to the snuff. This should be alleviated by a simple migration to OSGi DS, but for that we need AAA-206.

Comment by Robert Varga [ 24/Jan/24 ]

This will be resolved in 2024.03 Calcium.

Generated at Wed Feb 07 20:36:29 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.