https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/BGPCEP-111 bgpcep <p>When doing performance test, sometimes the situation described by the following log fragments happens, during add-lsp:</p> <p>2014-07-18 03:40:04.005 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-4&#93;</span> <br/> TRACE o.o.p.pcep.impl.PCEPSessionImpl - Message sent to socket: Pcinitiate [_pcinitiateMessage=PcinitiateMessage [_requests=[Requests [_srp=Srp [_operationId=SrpIdNumber <span class="error">&#91;_value=1&#93;</span>, ...</p> <p>2014-07-18 03:40:04.007 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-4&#93;</span> DEBUG o.o.p.pcep.impl.PCEPSessionImpl - PCEP message Pcrpt [_pcrptMessage=PcrptMessage [_reports=[Reports [ ... _srp=Srp [_operationId=SrpIdNumber <span class="error">&#91;_value=1&#93;</span>, ...</p> <p>2014-07-18 03:40:04.007 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-4&#93;</span> WARN o.o.b.p.t.p.Stateful07TopologySessionListener - Request ID SrpIdNumber <span class="error">&#91;_value=1&#93;</span> not found in outstanding DB</p> <p>This results in add-lsp never returning any output.</p> <p>Operating System: Linux<br/> Platform: PC</p> BGPCEP-111

PCEP request occasionally not found in outstanding DB

Bug Verified Done Dana Kutenicsova Vratko Polak Fri, 18 Jul 2014 14:24:49 +0000 Sun, 3 Mar 2019 11:50:16 +0000 Tue, 29 Jul 2014 12:31:05 +0000 Bugzilla Migration Bugzilla Migration 1.0 General Fri, 25 Jul 2014 00:00:00 +0000 0 3 <p>Hypothesis: Apparently, ODL was assuming that it can add request to waitingRequests upon sending Pcinitiate sooner than Pcrpt handling will look for it. As log evidences, this is not always so.</p> <p>My idea on how the fix could look like:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/9160/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/9160/</a></p> <p>So the logs provided point actually to the fact that the request should have been on the queue: the listener writing the 'sent to socket' message is attached to the same future the listener which moves the message. The processing happens on the same thread, so the other listener should have been invoked, too.</p> <p><a href="https://git.opendaylight.org/gerrit/9162" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/9162</a> adds trace messages to track lifecycle of the list, please retry with that.</p> <p>&gt; please retry with that</p> <p>Basically the same. Excerpts:</p> <p>2014-07-21 09:18:24.295 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7&#93;</span> <br/> TRACE o.o.p.pcep.impl.PCEPSessionImpl - Message sent to socket: Pcinitiate ...</p> <p>2014-07-21 09:18:24.296 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7&#93;</span> TRACE o.o.p.p.i.PCEPByteToMessageDecoder - Received to decode: ...</p> <p>2014-07-21 09:18:24.300 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7&#93;</span> TRACE o.o.b.p.t.p.AbstractTopologySessionListener - Removed request SrpIdNumber <span class="error">&#91;_value=3&#93;</span> object null</p> <p>2014-07-21 09:18:24.300 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7&#93;</span> WARN o.o.b.p.t.p.Stateful07TopologySessionListener - Request ID SrpIdNumber <span class="error">&#91;_value=3&#93;</span> not found in outstanding DB</p> <p>2014-07-21 09:18:24.320 PDT <span class="error">&#91;nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7&#93;</span> TRACE o.o.b.p.t.p.AbstractTopologySessionListener - Request SrpIdNumber <span class="error">&#91;_value=3&#93;</span> sent to peer (object org.opendaylight.bgpcep.pcep.topology.provider.PCEPRequest@20deec50)</p> <p>&gt; the logs provided point actually to the fact that the request should have been on the queue<br/> &gt; The processing happens on the same thread, so the other listener should have been invoked, too.</p> <p>It was invoked, just 20 milliseconds too late.</p> <p>My testing reached statistical significance for me to say: It is suspicious that I have never seen this bug when testing against single PCC (others present but do nothing, maybe apart keepalives).<br/> But, logs do not show anything interesting regarding communication on other Pcep sessions (usually handled by different threads and with long enough time differences).</p> <p>Training to get Change-Id right (and without syntax errors).<br/> <a href="https://git.opendaylight.org/gerrit/#/c/9160/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/9160/</a></p> <p>So looking at the traces, it seems that netty does not deliver the notification of the request leaving the socket first, but rather continues to process the message from the router.</p> <p>Your fixes are just papering over the problem, not fixing it, simply because membership on a particular list hold the logical state of the request.</p> <p>In particular, even with your fix, if the PCC closes the connection after we have pushed it, but have not received the future notification, we will inform the client of an Unsent error, where in fact it is Noack.</p> <p>So what we really need to do is get netty to order the events as they occur.</p> <p>&gt; So what we really need to do is get netty to order the events as they occur.</p> <p>Agreed.</p> <p>&gt; In particular, even with your fix, if the PCC closes the connection after we have pushed it, but have not received the future notification, we will inform the client of an Unsent error, where in fact it is Noack.</p> <p>I would have to think more about interaction between onSessionDown and !isSuccess.<br/> For commit b2c0fb9b66d35776f0bf4e6cea614c7c1e419d35 line 216, are we sure PCC has definitely not received the message?</p> <p>&gt; Your fixes are just papering over the problem</p> <p>So it is not worth to do a quick and dirty fix for "RPC hangs" bug if that would introduce "false unsent" bug?</p> <p>This bug is verified to not appear in ODL based on this build: <a href="https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/1741" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/1741</a></p> Development External issue ID 1394 External issue URL ODL SR Target Milestone Rank 0|i02a9r: