[OPNFLWPLUG-686] Li-Plugin: Transaction failure exception in 3-node cluster while processing port statistics Created: 11/May/16  Updated: 27/Sep/21  Resolved: 11/Jul/16

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Anil Vishnoi Assignee: Anil Vishnoi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File log-20160524-101534.tar.gz    
Issue Links:
Duplicate
is duplicated by OPNFLWPLUG-715 Occasionally is on statistics gatheri... Resolved
External issue ID: 5873

 Description   

I am testing net-virt with openflowplugin lithium plugin in 3-node cluster environment. Whenever OVSDB connects to net-virt, it creates the bridge and set the controller of the bridge to all the three controllers in the 3-node cluster. Whichever instance of openflowplugin become the owner of the openflow bridge, it starts throwing following exception while processing the port statisitics

2016-05-11 07:25:47,207 | INFO | ntLoopGroup-11-4 | LifecycleConductorImpl | 289 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Starting/Stopping services for node Uri [_value=openflow:82098239239489] was successful
2016-05-11 07:25:48,676 | WARN | ntLoopGroup-11-4 | StatisticsGatheringUtils | 289 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | stats processing of type OFPMPPORTSTATS for node Uri [_value=openflow:82098239239489] failed during write-to-tx step
java.lang.IllegalStateException: Transaction DOM-CHAIN-2-3 is no longer running
at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[65:com.google.guava:18.0.0]
at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerWriteTransaction.checkRunning(AbstractDOMBrokerWriteTransaction.java:145)[181:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerWriteTransaction.put(AbstractDOMBrokerWriteTransaction.java:74)[181:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.spi.ForwardingDOMDataReadWriteTransaction.put(ForwardingDOMDataReadWriteTransaction.java:49)[150:org.opendaylight.controller.sal-core-spi:1.4.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.put(AbstractWriteTransaction.java:46)[153:org.opendaylight.controller.sal-binding-broker-impl:1.4.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.put(BindingDOMWriteTransactionAdapter.java:40)[153:org.opendaylight.controller.sal-binding-broker-impl:1.4.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.writeToTransaction(TransactionChainManager.java:208)[289:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.writeToTransaction(DeviceContextImpl.java:328)[289:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsGatheringUtils.processNodeConnectorStatistics(StatisticsGatheringUtils.java:380)[289:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT

Full log file for one of the controller from 3-node cluster is attached with the bug.

 Comments   
Comment by Miroslav Macko [ 30/May/16 ]

We were trying to reproduce the error. I am appending logs from test in 3-node cluster without netvirt. There is problem with TxChain, when we are in state of closing.

2016-05-24 10:12:08,504 | DEBUG | CommitFutures-17 | StatisticsGatheringUtils | 277 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Stats reply FAILED for node Uri [_value=openflow:1] of type OFPMPTABLE: [RpcError [message=Connection closed., severity=ERROR, errorType=APPLICATION, tag=, applicationTag=null, info=null, cause=null]]
2016-05-24 10:12:08,504 | DEBUG | CommitFutures-17 | StatisticsContextImpl | 277 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Device connection is closed for Node : Uri [_value=openflow:1].
2016-05-24 10:12:08,504 | DEBUG | CommitFutures-17 | TransactionChainManager | 277 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | WriteTx is null for node KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]}

. Write data for InstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.snapshot.gathering.status.grouping.SnapshotGatheringStatusEnd, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]], org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableStatisticsGatheringStatus, org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.snapshot.gathering.status.grouping.SnapshotGatheringStatusEnd]}

was not realized.
2016-05-24 10:12:08,504 | WARN | CommitFutures-17 | StatisticsGatheringUtils | 277 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Can't write to transaction: {}
java.lang.Exception: Cannot write into transaction.
at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.writeToTransaction(TransactionChainManager.java:227)[277:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.writeToTransaction(DeviceContextImpl.java:336)[277:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsGatheringUtils.markDeviceStateSnapshotEnd(StatisticsGatheringUtils.java:515)[277:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT]
at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$1.onFailure(StatisticsContextImpl.java:137)[277:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[54:com.google.guava:18.0.0]...

We will follow up on that.

Thanks,
Miro

Comment by Miroslav Macko [ 30/May/16 ]

Attachment log-20160524-101534.tar.gz has been added with description: 3-node cluster log

Comment by Jozef Bacigal [ 21/Jun/16 ]

https://git.opendaylight.org/gerrit/#/c/40536/

Comment by Jozef Bacigal [ 11/Jul/16 ]

Anil can we close this bug?

Generated at Wed Feb 07 20:33:07 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.