Operating System: All
Platform: All

Attachment opendaylight.tar.xz has been added with description: ODL artifact, logs and config

yourkitsnapshot available at

https://drive.google.com/file/d/0ByXiyf4iY7RYSHJWTXhNOU0zZ2c/view?usp=sharing

Looks like https://bugs.opendaylight.org/show_bug.cgi?id=2255

yeah looks a lot like it. Will retest with ping pong DB.

Further testing revealed that this bug is newly introduced but indeed very similar to 2255.

A new queue was added into the transaction path but the fix as implemented on the CONTROLLER-957 was omitted for it. I have no idea where that queue is configured and what is its name so I cannot test increase of the capacity of this new queue but I do know it is a new queue because when I increased the capacity of the 2 queues I know about (“inmemory-config-datastore-provider” and “inmemory-operational-datastore-provider”) to 65000, the log messages produced by this bug indicated that the capacity of this overflowing queue is still 5000.

(In reply to Jozef Behran from comment #4)
> Further testing revealed that this bug is newly introduced but indeed very
> similar to 2255.
>
> A new queue was added into the transaction path but the fix as implemented
> on the CONTROLLER-957 was omitted for it. I have no idea where that queue is
> configured and what is its name so I cannot test increase of the capacity of
> this new queue but I do know it is a new queue because when I increased the
> capacity of the 2 queues I know about (“inmemory-config-datastore-provider”
> and “inmemory-operational-datastore-provider”) to 65000, the log messages
> produced by this bug indicated that the capacity of this overflowing queue
> is still 5000.

No, there are no new config parameters for in-memory-data-store
https://github.com/opendaylight/controller/blob/stable/helium/opendaylight/md-sal/sal-inmemory-datastore/src/main/yang/opendaylight-inmemory-datastore-provider.yang

You need to increase value of "max-data-store-executor-queue-size" (by default 5000).

That's exactly what I did. I did that for both, "inmemory-config-datastore-provider" and "inmemory-operational-datastore-provider". However the exception reported suggests it had no effect, I still can see "limit=5000" instead of "limit=65000" in its message.

Here is the relevant section of the configuration file:

— snip —
<module>
<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider">prefix:inmemory-config-datastore-provider</type>
<name>config-store-service</name>
<inmemory-config-datastore-provider xmlns="urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider">
<schema-service>
<type xmlns:dom="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">dom:schema-service</type>
<name>yang-schema-service</name>
</schema-service>
<max-data-store-executor-queue-size>65000</max-data-store-executor-queue-size>
</inmemory-config-datastore-provider>
</module>
<module>
<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider">prefix:inmemory-operational-datastore-provider</type>
<name>operational-store-service</name>
<inmemory-operational-datastore-provider xmlns="urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider">
<schema-service>
<type xmlns:dom="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">dom:schema-service</type>
<name>yang-schema-service</name>
</schema-service>
<max-data-store-executor-queue-size>65000</max-data-store-executor-queue-size>
</inmemory-operational-datastore-provider>
</module>
— snip —

The .yang file suggests these configuration elements are placed correctly here. Anybody has any idea what is/might be wrong here (or elsewhere)?

There is another "queue size" configuration parameter in "dom-in-memory-data-broker" named "max-data-broker-commit-queue-size"
https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/sal-dom-broker/src/main/yang/opendaylight-dom-broker-impl.yang;h=fa6d4961939b6f5bcbf4f5eb50d042ea9aa86556;hb=refs/heads/stable/helium

Snippet example (01-mdsal.xml):

<module>
<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl">prefix:dom-inmemory-data-broker</type>
<name>inmemory-data-broker</name>

<schema-service>
<type xmlns:dom="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">dom:schema-service</type>
<name>yang-schema-service</name>
</schema-service>

<config-data-store>
<type xmlns:config-dom-store-spi="urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store">config-dom-store-spi:config-dom-datastore</type>
<name>config-store-service</name>
</config-data-store>

<operational-data-store>
<type xmlns:operational-dom-store-spi="urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store">operational-dom-store-spi:operational-dom-datastore</type>
<name>operational-store-service</name>
</operational-data-store>

<max-data-broker-commit-queue-size xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl">50000</max-data-broker-commit-queue-size>
</module>

New tests with configuration edited as described show that Helium scales to above 1M prefixes when only RIB is updated and only 5k prefixes when RIB and Topology are both updated.

Test Setup

Cluster size 1
RAM 16G
Heap 6553M
permGen 0.5G
CPUs 4
Garbage Collection Default
Karaf Features odl-bgpcep-all, odl-restconf-noauth
ODL build Helium SR4

Java version java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Results

Total Failed Passed Pass %
Critical tests 5 0 5 100%
All tests 6 0 6 100%

Fixed

1M routes on IMDS, RIB only, 16G RAM, 6553M HEAP: 3m 58s

5k routes on IMDS, RIB+Topology, 16G RAM, 6553M HEAP: 1m 24s + 5m 26s

We can process 1m routes RIB only but 5k routes when using RIB and topology. To process more than 5k we expect we need to use the ping-pong data-broker.

This is no longer available in helium because of incompatibilities.