[NETCONF-222] Netty LEAK error leads to RemoteNetconfCommand unable to to establish internal connection to netconf server for client: 127.0.0.1

Fri, 15 Mar 2019 22:22:26 +0000

OS: 64-bit Linux 2.6.32 guested in VirtualBox on a 64-bit Windows Server 2008R2

ODL version: Lithium-SR3

Problem description:

During the loading of 00-nettty.xml in the booting of ODL controller, Netty's ResourceLeakDetector reported a LEAK error as the following:

2016-06-23 17:02:59,433 | ERROR | config-pusher | ResourceLeakDetector | 95 - io.netty.common - 4.0.26.Final | | LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetectionLevel=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

However 00-netty.xml was still loaded successfully. But later in the connection from ODL's internal netconf client to internal netconf server, ODL's RemoteNetconfCommand reported a failure as the following:

After our hacking of netty, the failure is because of a map naming LocalChannelRegistry.boundChannels in netty had no element of "local:netconf" as normal. It seems that netty internal data structure was destroyed or not correctly maintained because of the LEAK error reported above this failure. But who caused the LEAK？We don't know.

Full karaf.log:

2016-06-23 17:06:08,258 | DEBUG | oupCloseable-5-1 | ProtocolSessionPromise 2016-06-23 17:06:08,265 | DEBUG | oupCloseable-5-1 | AbstractDispatcher 2016-06-23 17:06:08,360 | INFO | o-group-thread-8 | ServerSession 2016-06-23 17:06:08,363 | INFO | NioProcessor-3 | ClientSessionImpl 2016-06-23 17:06:08,370 | INFO | NioProcessor-3 | ClientSessionImpl 2016-06-23 17:06:08,371 | INFO | NioProcessor-3 | ClientSessionImpl 2016-06-23 17:06:10,510 | WARN | NioProcessor-3 | AcceptAllServerKeyVerifier 2016-06-23 17:06:10,581 | INFO | NioProcessor-3 | ClientSessionImpl 2016-06-23 17:06:10,590 | INFO | NioProcessor-3 | ClientUserAuthServiceNew 2016-06-23 17:06:10,596 | INFO | NioProcessor-3 | UserAuthKeyboardInteractive 2016-06-23 17:06:10,686 | INFO | o-group-thread-5 | ServerUserAuthService 2016-06-23 17:06:10,687 | INFO | NioProcessor-3 | ClientUserAuthServiceNew 2016-06-23 17:06:10,692 | DEBUG | oupCloseable-5-2 | ProtocolSessionPromise 2016-06-23 17:06:10,693 | DEBUG | oupCloseable-5-2 | ProtocolSessionPromise 2016-06-23 17:06:10,694 | DEBUG | oupCloseable-5-2 | AbstractSessionNegotiator 2016-06-23 17:06:10,698 | WARN | entLoopGroup-4-2 | RemoteNetconfCommand 2016-06-23 17:06:10,713 | DEBUG | oupCloseable-5-2 | ReconnectPromise 2016-06-23 17:06:10,713 | DEBUG | oupCloseable-5-2 | ReconnectPromise 2016-06-23 17:06:18,400 | DEBUG | pool-15-thread-1 | AbstractSessionNegotiator java.lang.IllegalStateException: Session was not established after at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSessionNe at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTi at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(Hashed at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)[:1.7.0_5 | 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.ProtocolSessionPromise.connect(ProtocolSessionPromise.java:55) | Promise ProtocolSessionPromise@712253c6(incomplete) attempting connect /127.0.0.1:1830 for 2000ms
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.AbstractDispatcher.createClient(AbstractDispatcher.java:201) | Client created.
| 118 - org.apache.sshd.core - 0.14.0 | | Server session created from /127.0.0.1:41459
| 118 - org.apache.sshd.core - 0.14.0 | | Client session created
| 118 - org.apache.sshd.core - 0.14.0 | | Start flagging packets as pending until key exchange is done
| 118 - org.apache.sshd.core - 0.14.0 | | Server version string: SSH-2.0-SSHD-CORE-0.14.0
| 118 - org.apache.sshd.core - 0.14.0 | | Server at /127.0.0.1:1830 presented unverified RSA key: a0:7c:73:f0:ae:bd:6a:42:ff:85:5d:c9:a9:cf:34:5f
| 118 - org.apache.sshd.core - 0.14.0 | | Dequeing pending packets
| 118 - org.apache.sshd.core - 0.14.0 | | Received SSH_MSG_USERAUTH_FAILURE
| 118 - org.apache.sshd.core - 0.14.0 | | Received Password authentication en-US
| 118 - org.apache.sshd.core - 0.14.0 | | Session admin@/127.0.0.1:41459 authenticated
| 118 - org.apache.sshd.core - 0.14.0 | | Received SSH_MSG_USERAUTH_SUCCESS
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.ProtocolSessionPromise$BootstrapConnectListener.operationComplete(ProtocolSessionPromise.java:95) | Promise ProtocolSessionPromise@712253c6(incomplete) connection resolved
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.ProtocolSessionPromise$BootstrapConnectListener.operationComplete(ProtocolSessionPromise.java:118) | Promise ProtocolSessionPromise@712253c6(incomplete) connection successful
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.AbstractSessionNegotiator.channelActive(AbstractSessionNegotiator.java:79) | Starting session negotiation on channel [id: 0x37ccf37e]
| 297 - org.opendaylight.controller.netconf-ssh - 0.3.3.Lithium-SR3 | | Unable to establish internal connection to netconf server for client: 127.0.0.1
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.ReconnectPromise$ClosedChannelHandler.channelInactive(ReconnectPromise.java:118) | Connection to /127.0.0.1:1830 was dropped during negotiation, reattempting
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.ReconnectPromise$ClosedChannelHandler.channelInactive(ReconnectPromise.java:121) | Reconnecting after connection to /127.0.0.1:1830 was dropped
| 97 - org.opendaylight.controller.protocol-framework - 0.6.3.Lithium-SR3 | org.opendaylight.protocol.framework.AbstractSessionNegotiator.negotiationFailed(AbstractSessionNegotiator.java:51) | Negotiation on channel [id: 0x94a5f38f] failed
HashedWheelTimer$HashedWheelTimeout(deadline: 101358660 ns ago, task: org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSessionNegotiator$2@65695f20)
gotiator$2.run(AbstractNetconfSessionNegotiator.java:129)[115:org.opendaylight.controller.netconf-netty-util:0.3.3.Lithium-SR3]
mer.java:581)[95:io.netty.common:4.0.26.Final]
WheelTimer.java:655)[95:io.netty.common:4.0.26.Final]
class="error">[95:io.netty.common:4.0.26.Final]
1]

OpenDaylight JIRA

[NETCONF-222] Netty LEAK error leads to RemoteNetconfCommand unable to to establish internal connection to netconf server for client: 127.0.0.1