Operating System: All
Platform: All

I should point out that we try to write the modified node with a write merge.

I should point out that we try to commit the modified node with a write merge, not a overwrite.

Hi Moiz,

can you take a look at this? It is really important and pressing.

thanks,

Reinaldo,

Couple of questions,

1. Is this happening in code that has been merged into the SFC project?
2. If it has been, what are the steps to reproduce this problem?

I need to create a unit test to reproduce this problem.

Steps to reproduce.

1 - Check out SFC distribution or get integration build.

2 - Start an OVS switch with the following configuration:

Manager "tcp:192.168.1.14:6640"
Bridge br-tun
Port br-tun
Interface br-tun
type:internal

The IP address is your ODL's IP address

3 - (Run SFC karaf) or (integration and load all SFC features)

4 - After OVS connects state will be created in SFC. Then use the following command
to update state on OVS:

sudo ovs-vsctl add-port br-tun vx1 – set interface type=vxlan options:local_ip=10.0.0.1

This will create a notification in the OVS->OVSDB->SFC direction. When SFC tries to commit the modified
OVS node configuration to the datastore you will see all those errors and the system will stop operating. Not only this
transaction but any further transactions will not happen

This appears to be a serialization issue. When Reinaldo disabled persistence things worked fine.

This issue affects persistence and replication so still high priority.

Yes - DataTreeCandidatePayload is only created for persistence to the journal. I don't know why DataTreeCandidatePayload fails fast is there's a nested unmodified node (would have to ask Robert) but have you tried removing IllegalArgumentException and writing the UNMODIFIED byte? Maybe it's benign or is valid. The read code seems to handle UNMODIFIED.

(In reply to Moiz Raja from comment #6)
> This appears to be a serialization issue. When Reinaldo disabled persistence
> things worked fine.
>
> This issue affects persistence and replication so still high priority.

As I told Moiz, if I use a write-put (overwrite) instead of write-merge things work better. This is just a data point that I hope helps but I can not use write overwrite.

So the top-level node is defensive in order to deal with empty transactions. An UNMODIFIED node nested below root is an anomaly, as such nodes should normally be pruned during DataTreeModification.ready().

Reinaldo, can you explain what 'work better' means?

Slight correction... I believe that in the reproduction instructions:

sudo ovs-vsctl add-port br-tun vx1 – set interface type=vxlan options:local_ip=10.0.0.1

should be replaced with:

sudo ovs-vsctl add-port br-tun vx1 – set interface br-tun options:type=vxlan,local_ip=10.0.0.1

Works better as in I do not see barf messages with simple OVS transactions but I have not done serious testing because I can not use it.

Robert, this is not a top level node where this exception happens so as Tom recommends maybe it's ok to just write a UNMODIFIED byte here.

Reinaldo please try this fix https://git.opendaylight.org/gerrit/20310. Make sure you turn persistence on again.

I think that this is happening for a specific case. This is the node which appeared as UNMODIFIED

mod
—

NodeModification [identifier=AugmentationIdentifier

, modificationType=MERGE, childModification={(urn:cisco:params:xml:ns:yang:sfc-sff-ovs?revision=2014-07-01)ovs-options=NodeModification [identifier=(urn:cisco:params:xml:ns:yang:sfc-sff-ovs?revision=2014-07-01)ovs-options, modificationType=MERGE, childModification={}]}]

newMeta.data
------------

ImmutableAugmentationNode{nodeIdentifier=AugmentationIdentifier

, value=[ImmutableContainerNode{nodeIdentifier=(urn:cisco:params:xml:ns:yang:sfc-sff-ovs?revision=2014-07-01)ovs-options, value=[], attributes={}}]}

oldMeta.data
------------

ImmutableAugmentationNode{nodeIdentifier=AugmentationIdentifier

, value=[ImmutableContainerNode{nodeIdentifier=(urn:cisco:params:xml:ns:yang:sfc-sff-ovs?revision=2014-07-01)ovs-options, value=[], attributes={}}]}

I tried and the fix worked. I do not see clustering errors and all OVS information appears in SFC.

I tried to repro in a unit test by writing or merging the same node twice. But it succeeded.

It appears to be a specific case.

As Robert pointed out, InMemoryDataTreeModification.ready() is supposed to prune the tree of UNMODIFIED children. However, in the code path from the ex trace, I don't see where InMemoryDataTreeModification.ready() is called. There are other code paths in CDS (e.g. on recovery) where it is called. It seems like that might be the problem. However I'm unclear at this point exactly where ready() should/would be called.

Regardless, I think your patch is good - we really don't need the code to be that paranoid about UNMODIFIED children. There may be cases where something may not be pruned.

(In reply to Moiz Raja from comment #14)
> I think that this is happening for a specific case. This is the node which
> appeared as UNMODIFIED
>
> mod
> —
>
> NodeModification
> [identifier=AugmentationIdentifier

, modificationType=MERGE,
> childModification={(urn:cisco:params:xml:ns:yang:sfc-sff-ovs?revision=2014-
> 07-01)ovs-options=NodeModification
> [identifier=(urn:cisco:params:xml:ns:yang:sfc-sff-ovs?revision=2014-07-
> 01)ovs-options, modificationType=MERGE, childModification={}]}]
>
> newMeta.data
> ------------
>
> ImmutableAugmentationNode{nodeIdentifier=AugmentationIdentifier

,
> value=[ImmutableContainerNode{nodeIdentifier=(urn:cisco:params:xml:ns:yang:
> sfc-sff-ovs?revision=2014-07-01)ovs-options, value=[], attributes={}}]}
>
> oldMeta.data
> ------------
>
> ImmutableAugmentationNode{nodeIdentifier=AugmentationIdentifier

,
> value=[ImmutableContainerNode{nodeIdentifier=(urn:cisco:params:xml:ns:yang:
> sfc-sff-ovs?revision=2014-07-01)ovs-options, value=[], attributes={}}]}

Did you try restarting the controller? Would also be good to verify the data recovers correctly from persistence.

(In reply to Reinaldo Penno from comment #15)
> I tried and the fix worked. I do not see clustering errors and all OVS
> information appears in SFC.

Hi Tom,

just tested that as you requested. I restarted the controller and all SFC-OVS info is there.

{
"service-function-forwarders": {
"service-function-forwarder": [
{
"name": "ovsdb://192.168.1.27:39707/bridge/br-tun",
"sff-data-plane-locator": [
{
"name": "vx1",
"service-function-forwarder-ovs:ovs-options":

,
"data-plane-locator":

,
"service-function-forwarder-ovs:ovs-bridge":

},
{
"name": "br-tun",
"service-function-forwarder-ovs:ovs-options": {},
"service-function-forwarder-ovs:ovs-bridge":

,
"data-plane-locator":

}
],
"service-function-forwarder-ovs:ovs-bridge":

,
"service-function-forwarder-ovs:ovs-node":

},
{
"name": "ovsdb://192.168.1.27:39717/bridge/br-tun",
"sff-data-plane-locator": [
{
"name": "br-tun",
"service-function-forwarder-ovs:ovs-bridge":

,
"data-plane-locator":

,
"service-function-forwarder-ovs:ovs-options": {}
},
{
"name": "vx1",
"service-function-forwarder-ovs:ovs-bridge":

,
"data-plane-locator":

,
"service-function-forwarder-ovs:ovs-options":

}
],
"service-function-forwarder-ovs:ovs-bridge":

,
"service-function-forwarder-ovs:ovs-node":

}
]
}
}

What do you do with ovs-options? That is the child which appears unmodified.

Cool. I'm good with merging the patch. We can continue looking into why there were UNMODIFIED children but this will unblock you.

(In reply to Reinaldo Penno from comment #18)
> Hi Tom,
>
> just tested that as you requested. I restarted the controller and all
> SFC-OVS info is there.
>
> {
> "service-function-forwarders": {
> "service-function-forwarder": [
> {
> "name": "ovsdb://192.168.1.27:39707/bridge/br-tun",
> "sff-data-plane-locator": [
> {
> "name": "vx1",
> "service-function-forwarder-ovs:ovs-options":

,
> "data-plane-locator":

,
> "service-function-forwarder-ovs:ovs-bridge":

> },
> {
> "name": "br-tun",
> "service-function-forwarder-ovs:ovs-options": {},
> "service-function-forwarder-ovs:ovs-bridge":

,
> "data-plane-locator":

> }
> ],
> "service-function-forwarder-ovs:ovs-bridge":

,
> "service-function-forwarder-ovs:ovs-node":

> },
> {
> "name": "ovsdb://192.168.1.27:39717/bridge/br-tun",
> "sff-data-plane-locator": [
> {
> "name": "br-tun",
> "service-function-forwarder-ovs:ovs-bridge":

,
> "data-plane-locator":

,
> "service-function-forwarder-ovs:ovs-options": {}
> },
> {
> "name": "vx1",
> "service-function-forwarder-ovs:ovs-bridge":

,
> "data-plane-locator":

,
> "service-function-forwarder-ovs:ovs-options":

> }
> ],
> "service-function-forwarder-ovs:ovs-bridge":

,
> "service-function-forwarder-ovs:ovs-node":

> }
> ]
> }
> }

DataTreeModification.ready() is called via ShardDataTree.finishTransaction(), which is invoked from CohortEntry.ready().

Yup - missed that.

(In reply to Robert Varga from comment #21)
> DataTreeModification.ready() is called via
> ShardDataTree.finishTransaction(), which is invoked from CohortEntry.ready().