[CONTROLLER-2028] Thousands of guava Finalizer threads in waiting

Fri, 4 Feb 2022 08:23:48 +0000

Distribution: onap-karaf-0.15.1
Features installed: odl-netconf-topology, odl-restconf-nb-rfc8040, odl-mdsal-apidocs, jolokia

Situation:

Mounting of 10 000 netconf-testtool devices (sending of PUT mount requests one after the other) is causing 10 000 threads in "waiting".

Thread dump (jstack <pid>) was captured during mounting 10k devices

before-mount-dump10k.txt - thread dump after ODL starts, before sending PUT requests
2000kdump10k.txt - thread dump after 2k PUT mount requests
5000kdump10k.txt - thread dump after 5k PUT mount requests
7000kdump10k.txt - thread dump after 7k PUT mount requests
10000kdump10k.txt - thread dump after 10k PUT mount requests

Snippet of a thread in waiting:

"com.google.common.base.internal.Finalizer" #4549 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=76.30s tid=0x00007f4e8c1f9800 nid=0x7a78 in Object.wait()  [0x00007f4d50f50000]"com.google.common.base.internal.Finalizer" #4549 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=76.30s tid=0x00007f4e8c1f9800 nid=0x7a78 in Object.wait()  [0x00007f4d50f50000]   java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.12/Native Method) - waiting on <no object reference available> at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007272f7b60> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(java.base@11.0.12/ReferenceQueue.java:176) at com.google.common.base.internal.Finalizer.run(Finalizer.java:145) at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

I found only this reference which uses Finalizer internally:
https://github.com/opendaylight/controller/blob/37e9a32e285a174443c312ef257fc1df359b50d9/opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/TransactionContextCleanup.java#L33

based on:
https://guava.dev/releases/7.0/api/docs/com/google/common/base/internal/Finalizer.html

OpenDaylight JIRA

[CONTROLLER-2028] Thousands of guava Finalizer threads in waiting