[CONTROLLER-1669] C: write-transactions from all nodes cause OutOfMemoryError Created: 11/May/17  Updated: 25/Jul/23  Resolved: 12/May/17

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Peter Gubka Assignee: Tom Pantelis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 8424

 Description   

The job is here:
https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/19/

OutOfMemoryError on node 2 can be found here:
https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/19/archives/odl2_karaf_console.log.gz

Heap dump file created [1584803866 bytes in 11.824 secs]
SLF4J: Failed toString() invocation on an object of type [org.opendaylight.controller.cluster.access.client.ConnectedClientConnection]
java.lang.OutOfMemoryError: Java heap space

The test is very simple. On all 3 nodes a write-trancactions rpc is started.
https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/19/archives/log.html.gz#s1-t1-k4

After that the robot just waits for rpc to finish. Then after 65s the rest is timed out from robot side, none of 3 responses were delivered within this timeframe.

Similar bugs:
OutOfMemoryError - https://bugs.opendaylight.org/show_bug.cgi?id=8419
not returned rpc - https://bugs.opendaylight.org/show_bug.cgi?id=8372

 Comments   
Comment by Tom Pantelis [ 11/May/17 ]

It choked trying log the toString of ConnectedClientConnection:

SLF4J: Failed toString() invocation on an object of type [org.opendaylight.controller.cluster.access.client.ConnectedClientConnection]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:362)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.DataTreeState.toString(DataTreeState.java:63)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.toString(InMemoryDataTree.java:153)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
at org.opendaylight.controller.cluster.access.client.BackendInfo.toString(BackendInfo.java:69)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
at org.opendaylight.controller.cluster.access.client.AbstractClientConnection.toString(AbstractClientConnection.java:306)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:297)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:269)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:227)
at org.ops4j.pax.logging.slf4j.Slf4jLogger.info(Slf4jLogger.java:533)
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.reconnectConnection(ClientActorBehavior.java:302)
at org.opendaylight.controller.cluster.access.client.ConnectedClientConnection.lockedReconnect(ConnectedClientConnection.java:24)
at org.opendaylight.controller.cluster.access.client.AbstractClientConnection.runTimer(AbstractClientConnection.java:218)
at org.opendaylight.controller.cluster.access.client.AbstractClientConnection$$Lambda$113/242180988.execute(Unknown Source)
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:118)
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:42)
at org.opendaylight.controller.cluster.access.client.AbstractClientActor.onReceiveCommand(AbstractClientActor.java:59)
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)

We don't want to print out the entire InMemoryDataTree.

Comment by Tom Pantelis [ 11/May/17 ]

https://git.opendaylight.org/gerrit/#/c/56539/ added ShardBackendInfo.dataTree to the toString output in stable/carbon. This patch was not cherry pick to master however. We obviously can't print the dataTree contents - I think the intent of that patch was to output the presence of the dataTree which means it's local.

Comment by Tom Pantelis [ 11/May/17 ]

https://git.opendaylight.org/gerrit/#/c/56539/ added ShardBackendInfo.dataTree to the toString output in stable/carbon. This patch was not cherry pick to master however. We obviously can't print the dataTree contents - I think the intent of that patch was to output the presence of the dataTree which means it's local.

Comment by Tom Pantelis [ 11/May/17 ]

https://git.opendaylight.org/gerrit/#/c/56539/ added ShardBackendInfo.dataTree to the toString output in stable/carbon. This patch was not cherry pick to master however. We obviously can't print the dataTree contents - I think the intent of that patch was to output the presence of the dataTree which means it's local.

Comment by Tom Pantelis [ 11/May/17 ]

Submitted https://git.opendaylight.org/gerrit/#/c/56880/

Generated at Wed Feb 07 19:56:08 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.