https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1669 controller <p>The job is here:<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/19/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/19/</a></p> <p>OutOfMemoryError on node 2 can be found here:<br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/19/archives/odl2_karaf_console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/19/archives/odl2_karaf_console.log.gz</a></p> <p>Heap dump file created <span class="error">&#91;1584803866 bytes in 11.824 secs&#93;</span><br/> SLF4J: Failed toString() invocation on an object of type <span class="error">&#91;org.opendaylight.controller.cluster.access.client.ConnectedClientConnection&#93;</span><br/> java.lang.OutOfMemoryError: Java heap space</p> <p>The test is very simple. On all 3 nodes a write-trancactions rpc is started.<br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/19/archives/log.html.gz#s1-t1-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/19/archives/log.html.gz#s1-t1-k4</a></p> <p>After that the robot just waits for rpc to finish. Then after 65s the rest is timed out from robot side, none of 3 responses were delivered within this timeframe.</p> <p>Similar bugs:<br/> OutOfMemoryError - <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8419" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8419</a><br/> not returned rpc - <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8372" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8372</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1669

C: write-transactions from all nodes cause OutOfMemoryError

Bug Resolved Done Tom Pantelis Peter Gubka Thu, 11 May 2017 14:16:05 +0000 Tue, 25 Jul 2023 08:24:36 +0000 Fri, 12 May 2017 10:34:24 +0000 clustering 0 3 <p>It choked trying log the toString of ConnectedClientConnection:</p> <p>SLF4J: Failed toString() invocation on an object of type <span class="error">&#91;org.opendaylight.controller.cluster.access.client.ConnectedClientConnection&#93;</span><br/> java.lang.OutOfMemoryError: Java heap space<br/> at java.util.Arrays.copyOf(Arrays.java:3332)<br/> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)<br/> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)<br/> at java.lang.StringBuilder.append(StringBuilder.java:202)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:362)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.DataTreeState.toString(DataTreeState.java:63)<br/> at java.lang.String.valueOf(String.java:2994)<br/> at java.lang.StringBuilder.append(StringBuilder.java:131)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.toString(InMemoryDataTree.java:153)<br/> at java.lang.String.valueOf(String.java:2994)<br/> at java.lang.StringBuilder.append(StringBuilder.java:131)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)<br/> at org.opendaylight.controller.cluster.access.client.BackendInfo.toString(BackendInfo.java:69)<br/> at java.lang.String.valueOf(String.java:2994)<br/> at java.lang.StringBuilder.append(StringBuilder.java:131)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)<br/> at org.opendaylight.controller.cluster.access.client.AbstractClientConnection.toString(AbstractClientConnection.java:306)<br/> at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:297)<br/> at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:269)<br/> at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:227)<br/> at org.ops4j.pax.logging.slf4j.Slf4jLogger.info(Slf4jLogger.java:533)<br/> at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.reconnectConnection(ClientActorBehavior.java:302)<br/> at org.opendaylight.controller.cluster.access.client.ConnectedClientConnection.lockedReconnect(ConnectedClientConnection.java:24)<br/> at org.opendaylight.controller.cluster.access.client.AbstractClientConnection.runTimer(AbstractClientConnection.java:218)<br/> at org.opendaylight.controller.cluster.access.client.AbstractClientConnection$$Lambda$113/242180988.execute(Unknown Source)<br/> at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:118)<br/> at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:42)<br/> at org.opendaylight.controller.cluster.access.client.AbstractClientActor.onReceiveCommand(AbstractClientActor.java:59)<br/> at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)</p> <p>We don't want to print out the entire InMemoryDataTree.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/56539/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56539/</a> added ShardBackendInfo.dataTree to the toString output in stable/carbon. This patch was not cherry pick to master however. We obviously can't print the dataTree contents - I think the intent of that patch was to output the presence of the dataTree which means it's local.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/56539/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56539/</a> added ShardBackendInfo.dataTree to the toString output in stable/carbon. This patch was not cherry pick to master however. We obviously can't print the dataTree contents - I think the intent of that patch was to output the presence of the dataTree which means it's local.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/56539/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56539/</a> added ShardBackendInfo.dataTree to the toString output in stable/carbon. This patch was not cherry pick to master however. We obviously can't print the dataTree contents - I think the intent of that patch was to output the presence of the dataTree which means it's local.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/56880/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56880/</a></p> Development External issue ID 8424 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02rzz: