[CONTROLLER-1307] Clustering: Data Broker allows put with Inconsistent Instance Identifier and Data Created: 12/May/15  Updated: 04/Jun/15  Resolved: 04/Jun/15

Status: Resolved
Project: controller
Component/s: mdsal
Affects Version/s: Post-Helium
Fix Version/s: None

Type: Bug
Reporter: Moiz Raja Assignee: Tony Tkacik
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 3189
Priority: Normal

 Description   

Steps to reproduce,

1. Build controller stable/lithium
2. Build openflowplugin stable/lithium or master
3. Run the openflowplugin karaf distribution
4. feature:install odl-openflowplugin-all-li
5. tail the log and make sure the datastore starts up properly
Look for log message : shard-manager-config: All Shards are ready - data store config is ready
6. Using postman or such PUT the following flow,

PUT : http://localhost:8181/restconf/config/opendaylight-inventory:nodes/node/openflow:1/table/2/flow/FooXf7

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<flow xmlns="urn:opendaylight:flow:inventory">
<strict>false</strict>
<instructions>
<instruction>
<order>0</order>
<apply-actions>
<action>
<order>0</order>
<dec-nw-ttl/>
</action>
</apply-actions>
</instruction>
</instructions>
<table_id>2</table_id>
<id>130</id>
<cookie_mask>255</cookie_mask>
<match>
<ethernet-match>
<ethernet-type>
<type>2048</type>
</ethernet-type>
<ethernet-destination>
<address>ff:ff:ff:ff:ff:aa</address>
</ethernet-destination>
<ethernet-source>
<address>00:00:00:11:23:ae</address>
</ethernet-source>
</ethernet-match>
<ipv4-source>10.1.2.0/24</ipv4-source>
<ipv4-destination>20.4.0.0/16</ipv4-destination>
<ip-match>
<ip-protocol>56</ip-protocol>
<ip-dscp>15</ip-dscp>
<ip-ecn>1</ip-ecn>
</ip-match>
<in-port>0</in-port>
</match>
<cookie>7</cookie>
<flow-name>FooXf7</flow-name>
<priority>7</priority>
<barrier>false</barrier>
</flow>

8. PUT the same flow again and you will see the following exception in the log and the PUT will fail.

My expectation is that this should work as it is the only way to modify a flow. Also, note that PUTting a different flow into the same table also fails.

2015-05-12 11:25:47,673 | ERROR | lt-dispatcher-30 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config An exception occurred while preCommitting transaction member-1-txn-3
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value
at com.google.common.base.Absent.get(Absent.java:47)[64:com.google.guava:18.0.0]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:78)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.create(DataTreeCandidatePayload.java:99)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.continueCommit(Shard.java:322)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:339)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:296)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:254)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:139)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:230)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[174:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[167:com.typesafe.akka.actor:2.3.10]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[167:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[167:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[167:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[167:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[167:com.typesafe.akka.actor:2.3.10]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
2015-05-12 11:25:47,675 | WARN | lt-dispatcher-23 | ConcurrentDOMDataBroker | 182 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx: DOM-3 Error during phase CAN_COMMIT, starting Abort
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value
at com.google.common.base.Absent.get(Absent.java:47)[64:com.google.guava:18.0.0]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:78)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.create(DataTreeCandidatePayload.java:99)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.continueCommit(Shard.java:322)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:339)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:296)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:254)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:139)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:230)[182:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[174:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[167:com.typesafe.akka.actor:2.3.10]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[172:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[167:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[167:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[167:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[167:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[167:com.typesafe.akka.actor:2.3.10]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[164:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]

 Comments   
Comment by Tony Tkacik [ 13/May/15 ]

Did bit analysis and exception was caused by two different factors:

XML and RESTCONF URI for PUT were inconsistent:

URI specifies flow id as FooXf7, but xml body of flow specifies id 130.
This bug is tracked in restconf as CONTROLLER-1249.

Restconf created instance identifier with FooXf7, but data body contained id 130.

Data Broker and Data Tree, failed to verify
that last argument of instance identifier is different from Node Identifier
in data, which resulted in inconsistencies in internal structures and failed.

So to fix this is to do check on input that last path argument of II and
node identifier of data matches.

Decreasing to normal, since that bug is observed only with incorrect II and data.
For correct data and II clustered datastore behaves normally and operation works.

Comment by Tony Tkacik [ 13/May/15 ]

In-Memory Data Tree and In-Memory Data Store (transitevely)
https://git.opendaylight.org/gerrit/20225

Clustering Data Store:

https://git.opendaylight.org/gerrit/20226

Generated at Wed Feb 07 19:55:12 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.