https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-422 netconf <p>I'm running clustered 3-node ODL with 5 active netconf sessions with Honeycomb netconf server after some time generates<br/> 2017-05-15 13:04:04.423 UTC <span class="error">&#91;netconf-netty-1&#93;</span> WARN o.o.n.i.u.DeserializerExceptionHandler - An exception occurred during message handling<br/> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: GC overhead limit exceeded <br/> in honeycomb and periodic messages in ODL:<br/> Caused by: NetconfDocumentedException{message=RemoteDevice</p> {overcloud-controller-2.opnfv.org} <p>:Master is down.</p> <p>Please contact me (mperina@cisco.com) about logs since thier size is over 1GB.</p> <p>Operating System: All<br/> Platform: All</p> NETCONF-422

java.lang.OutOfMemoryError: GC overhead limit exceeded generates "Master is down" exception

Bug Resolved Cannot Reproduce Unassigned Matej Perina Mon, 15 May 2017 14:21:47 +0000 Fri, 15 Mar 2019 22:22:41 +0000 Tue, 16 May 2017 13:29:11 +0000 netconf 0 4 <p>Is this a blocker bug for Carbon? If so, is there an ETA for when a fix can be completed? If not, could someone from the NETCONF team please retarget the for Nitrogen or Carbon SR1?</p> <p>I've found this in master node logs:</p> <p>2017-05-15 12:29:51,897 | TRACE | lt-dispatcher-31 | NetconfDeviceCommunicator | 299 - org.opendaylight.netconf.sal-netconf-connector - 1.5.0.Carbon | RemoteDevice</p> {overcloud-controller-0.opnfv.org}: Sending message &lt;rpc message-id="m-3327" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;<br/> 2017-05-15 12:29:51,898 | TRACE | oupCloseable-3-1 | NetconfDeviceCommunicator | 299 - org.opendaylight.netconf.sal-netconf-connector - 1.5.0.Carbon | Finished sending request &lt;rpc message-id="m-3327" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;<br/> 2017-05-15 12:30:04,211 | DEBUG | oupCloseable-3-1 | NetconfDeviceCommunicator | 299 - org.opendaylight.netconf.sal-netconf-connector - 1.5.0.Carbon | RemoteDevice{overcloud-controller-0.opnfv.org} <p>: Message received &lt;rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-3327"&gt;<br/> 2017-05-15 12:30:04,213 | TRACE | oupCloseable-3-1 | NetconfDeviceCommunicator | 299 - org.opendaylight.netconf.sal-netconf-connector - 1.5.0.Carbon | RemoteDevice</p> {overcloud-controller-0.opnfv.org} <p>: Matched request: &lt;rpc message-id="m-3327" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;</p> <p>Master node sends rpc to the device. Probably due to "GC overhead limit exceeded" on the device, rpc-reply is sent from device to ODL 11s later. Default ask timeout in odl-netconf-clustered-topology is 5s, hence "Master is down." message.</p> <p>Ask timeout can be set when mount point is created by parameter "actor-response-wait-time", see netconf-node-topology.yang.<br/> In my opinion, you need to somehow throttle reads/writes via netconf, so honeycomb manages to process messages in time.<br/> Maybe "concurrent-rpc-limit" parameter from netconf-node-topology.yang could be used. It limits how many rpcs can be sent, before reply to first arrives. If limit is exceeded, it throws an exception.</p> <p>This is a problem on the SB device, which is running Boron-SR3 code and is probably not sized properly. Lowering priority and targetting Boron-SR4.</p> Development External issue ID 8454 External issue URL ODL SR Target Milestone Rank 0|i01yfz: