Operating System: All
Platform: All

Attachment bug2353.tar.xz has been added with description: Archive with curl output and karaf.log from 3 instances (third one did not even had ipv4 topology).

Looks like a clustering problem. Since neither replication nor persistence is enabled, this should be working locally.

I do not think we can recover from this failure in the app – we have no way of affectivng the datastore, nor do we have visibility into how the DS heals.

Vratko,

Could you please try this with a controller built from master. There were a couple of issues with the transaction chaining implementation in the clustered datastore that may possibly affect you.

Also I will need to check the configuration to see how persistence/replication were disabled.

> Could you please try this with a controller built from master.

Will do that tomorrow.

> Also I will need to check the configuration to see how
> persistence/replication were disabled.

Persistence: The old way, using
system:property shard.persistent false
and I just realized it does not work that way anymore.
Is the new way documented somewhere on a wiki page?

But snapshot and journal directories were not present, so hopefully there was not much difference in behavior from truly no-persistence setup.

Replication:
Attached archive with clustering configuration, but also BGP and PCEP ODL configuration and XRVR configuration.

Attachment conf.tar.xz has been added with description: Archive containing various configuration, as present when logs were gathered.

I forgot to mention, that the same ODL and XRVR configuration was manually tested against non-clustered configuration (basically everything the same, just odl-mdsal-clustering feature not installed) and linkstate topology was there indeed.

I added some notes on how to disable persistence. This is applicable to post stable/helium code only.

https://wiki.opendaylight.org/view/Running_and_testing_an_OpenDaylight_Cluster#How_do_we_disable_persistence.3F

Persistence is enabled by default in stable/helium so I wonder why snapshots and journal directory were not created for you.

I looked at the attached configuration and it seems to be correct. I can only suspect that you are being hit by the following bugs,

2318
2319
2337
2339
2340

All of these are resolved on master but need to be merged to stable/helium.

> Could you please try this with a controller built from master.

I tried the build from
https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/2719/artifact/distributions/extra/karaf/target/distribution-karaf-0.3.0-SNAPSHOT.tar.gz

What I got is the same bug as described in
https://bugs.opendaylight.org/show_bug.cgi?id=1918#c2
so clustering did not even start on two of the three nodes.
It would take me some more time to work around that.

>> snapshot and journal directories were not present,
>
> I wonder why snapshots and journal directory were not created for you.

Pardon my weak English. I meant to say that those directories were not created before starting karaf. Of course they were created during karaf run.

> https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/2719/artifact/distributions/extra/karaf/target/distribution-karaf-0.3.0-SNAPSHOT.tar.gz

The bug is also present on master, also errors look similar.
First dangerously looking error is actually this one

2014-11-12 11:39:23,329 | ERROR | lt-dispatcher-17 | OneForOneStrategy | 234 - com.typesafe.akka.slf4j - 2.3.4 | Node identifier contains different value: (urn:opendaylight:params:xml:ns:yang:bgp-linkstate?revision=2013-11-25)isis-area-id[[B@7e1ac7e4] than value itself: [B@10398a6a

which was also present in Helium branch logs, I just did not recognize it then.
In master branch I spotted it because it comes directly before more familiar clustering error:

2014-11-12 11:39:28,362 | ERROR | CommitFutures-1 | RIBImpl | 263 - org.opendaylight.bgpcep.bgp-rib-impl - 0.4.0.SNAPSHOT | Broken chain in RIB KeyedInstanceIdentifier

transaction DOM-CHAIN-3-11
TransactionCommitFailedException

Attachment karaf_20141112.log.xz has been added with description: master branch compressed full log from .10 node

>
> 2014-11-12 11:39:23,329 | ERROR | lt-dispatcher-17 | OneForOneStrategy
> | 234 - com.typesafe.akka.slf4j - 2.3.4 | Node identifier contains different
> value:
> (urn:opendaylight:params:xml:ns:yang:bgp-linkstate?revision=2013-11-25)isis-
> area-id[[B@7e1ac7e4] than value itself: [B@10398a6a
>

This error emanates from the ImmutableLeafSetEntryNodeBuilder:

ImmutableLeafSetEntryNode(final YangInstanceIdentifier.NodeWithValue nodeIdentifier, final T value, final Map<QName, String> attributes) {
super(nodeIdentifier, value, attributes);
Preconditions.checkArgument(nodeIdentifier.getValue().equals(value),
"Node identifier contains different value: %s than value itself: %s",
nodeIdentifier, value);
}

The value is a byte [] (as evidenced by toString output "[B@...") so I can see why the values don't match as byte[]#equals only checks reference equality. The code needs to check element equality in case of array.

So this appears to be a bug in ImmutableLeafSetEntryNodeBuilder but why doesn't this manifest with the in-memory data store (IMDS)? I suspect with the IMDS the 2 byte[] vars happen to be the same instance/reference. However in the CDS, the data is serialized/de-serialized so this results in different instances and equality fails.

Tom, this is related to the byte[] serialization defect. That was fixed however you can have a NodeIdentifier also with a byte[] as the value (leaflists) that is where this problem occurs.

This we can fix in the ValueSerializer but I suspect there is more to this bug than just this. I will be investigating this further...

https://git.opendaylight.org/gerrit/#/c/12820/ - yangtools
https://git.opendaylight.org/gerrit/#/c/12794/ - controller
https://git.opendaylight.org/gerrit/#/c/12806/ - bgpcep

https://git.opendaylight.org/gerrit/#/c/12827/ - contoller - stable/helium

Yangtools patch merged

https://git.opendaylight.org/gerrit/#/c/12820/ - yangtools:master
https://git.opendaylight.org/gerrit/#/c/12884/ - yangtools:stable/helium