karaf.log
[OVSDB-324] operational datastore is not getting updated after the previous transaction failed. Created: 18/Mar/16 Updated: 30/Oct/17 Resolved: 31/Aug/16 |
|
| Status: | Resolved |
| Project: | ovsdb |
| Component/s: | Southbound.hw_vtep |
| Affects Version/s: | unspecified |
| Fix Version/s: | None |
| Type: | Bug | ||
| Reporter: | suneel verma | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: All |
||
| Attachments: |
karaf.log
|
| External issue ID: | 5555 |
| Description |
|
The following message keeps coming in the plugin logs. 2016-03-18 11:39:06,279 | WARN | n-invoker-impl-0 | TransactionInvokerImpl | 228 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.2.2.SNAPSHOT | Exception invoking Transaction: |
| Comments |
| Comment by Anil Vishnoi [ 17/May/16 ] |
|
Hi Vishal, Any update on this bug ? Thanks |
| Comment by Vishal Thapar [ 17/May/16 ] |
|
I have a workaround for it already merged in hwvtep plugin. This issue happens when there is a bug in hwvtep plugin code [or could be in OVSDB code too], resulting in exception. TransactionInvoker retries failed transactions but if there is bug it will fail everytime. My workaround catches exceptions. A proper fix could be to catch non-DataStore exceptions in TransactionInvoker and not not retry in that case. Not sure if it is good idea or not. |
| Comment by Hideyuki Tai [ 24/Aug/16 ] |
|
We've faced a similar issue twice yesterday. It seems to me that the issue halted the thread of the run() method of the TransactionInvokerImpl class of org.opendaylight.ovsdb.southbound.transactions.md package. After the issue happened, the OVSDB Southbound plugin didn't process any update notifications. That's really critical issue. So I've changed the "Importance" of the bug into "blocker". We've seen this issue when we tested the legacy NetVirt with 3-ODL-node cluster on the stable/beryllium branch. We've faced this issue twice in three test runs. Therefore, I think it is not a rare event. It would happen often. Just before the thread was halted, ODL output the following WARN messages. I don't think it shows the direct cause of the issue. However, I can guess something bad happens around ShardDataTreeTransactionChain. 2016-08-23 22:10:57,909 | WARN | lt-dispatcher-15 | ConcurrentDOMDataBroker | 166 - org.opendaylight.controller.sal-distributed-datastore - 1.3.4.SNAPSHOT | Tx: DOM-CHAIN-0-58 Error during phase CAN_COMMIT, starting Abort 2016-08-23 22:10:57,932 | WARN | n-invoker-impl-0 | SouthboundUtil | 253 - org.opendaylight.ovsdb.southbound-impl - 1.2.5.SNAPSHOT | Read Operational/DS for Node failed! KeyedInstanceIdentifier {targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/69d5898c-a35e-4639-9df7-7fbda31d5c48]]]]}ReadFailedException{message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[ {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[ {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/69d5898c-a35e-4639-9df7-7fbda31d5c48}], errorList=[RpcError [message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[ {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[ {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/69d5898c-a35e-4639-9df7-7fbda31d5c48}], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.Exception: Error creating READ_WRITE transaction on shard topology]]} |
| Comment by Hideyuki Tai [ 24/Aug/16 ] |
|
Attachment karaf.log has been added with description: Log messages when we faced the issue that the run() thread was halted |
| Comment by Hideyuki Tai [ 24/Aug/16 ] |
|
We faced the issue again. And, we've observed that IllegalStateException happens at calling the ReadWriteTransaction.submit() in the run method of the TransactionInvokerImpl. The reason for the IllegalStateException was that the submit() method was called when the state of the DOMBrokerTransactionChain was FAILED. The reason that the chain was at FAILED state was that a previous calling to the submit() on the chain failed. As of now, I'm not sure why a previous calling to the submit() was failed. What I know is that just before the IllegalStateException happened, ReadWriteTransaction.read() failed during the phase CAN_COMMIT. I'm not sure even why the read() method needs to enter "CAN_COMMIT", that's a little bit weird to me, since I think reading data itself doesn't require any commit. For more details, please see the attached karaf.log. To know what kind of Exception happens in the run method of the TransactionInvoker, in our local environment, we've changed the condition of the catch block in the line 98 of the TransactionInvokerImpl.java as follows to catch any Exception. [southbound/southbound-impl/src/main/java/org/opendaylight/ovsdb/southbound/transactions/md/TransactionInvokerImpl.java] Current: } catch (InterruptedException e) { Changed: }catch (Exception e) { The attached karaf.log was the log file when we observed that the run() thread was halted with the above code change. |
| Comment by Anil Vishnoi [ 30/Aug/16 ] |
|
stable/boron : https://git.opendaylight.org/gerrit/#/c/44806/ |
| Comment by Hideyuki Tai [ 31/Aug/16 ] |
|
I think we can close this bug. I mean I think it's ok to change the status to RESOLVED. The current status is we've already merged the change for this bug into the master and stable/boron branches. And, we've not seen this issue after that. |