https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/BGPCEP-494 bgpcep <p>Issue happens in real network environment. If a PCRpt message is received with bandwidth reoptimization object (ref: <a href="https://tools.ietf.org/html/rfc5440#section-7.7" class="external-link" target="_blank" rel="nofollow noopener">https://tools.ietf.org/html/rfc5440#section-7.7</a>) it causes the controller to loop and ultimately results in heap OutOfMemory error. Crafted packet used to repro the issue is attached. Is is not clear at this point why PCC is including bandwidth reoptimization object in PCRpt</p> <p>Operating System: All<br/> Platform: All</p> BGPCEP-494

PCRpt received with bandwidth reoptimization object leads to loop causing OOM

Bug Resolved Done Ajay L Ajay L Wed, 20 Jul 2016 18:20:13 +0000 Sun, 3 Mar 2019 11:49:48 +0000 Wed, 10 Aug 2016 09:20:14 +0000 Bugzilla Migration Bugzilla Migration PCEP 0 2 <p>Attachment pcrpt-oom-repro.pcap has been added with description: PCRpt used to reproduce the issue</p> <p>Cisco DE is investigating if/how a 0x5 / 0x2 BW object could be sent from the router, but current XR code is only expected to send 0x5 / 0x5 BW Sample.</p> <p>Based on OOM heap dump analysis, controller is receiving PCRpt with objects in below sequence:<br/> LSP<br/> ERO<br/> LSPA<br/> Bandwidth<br/> Bandwidth Reoptimization</p> <p>RFC 5440 (<a href="https://tools.ietf.org/html/rfc5440#section-7.7" class="external-link" target="_blank" rel="nofollow noopener">https://tools.ietf.org/html/rfc5440#section-7.7</a>) describes the 2 types of bandwidth objects and specifies under what scenario the reoptimization bandwidth object is used:</p> <p>" o R (Reoptimization - 1 bit): when set, the requesting PCC specifies<br/> that the PCReq message relates to the reoptimization of an<br/> existing TE LSP. For all TE LSPs except zero-bandwidth LSPs, when<br/> the R bit is set, an RRO (see Section 7.10) MUST be included in<br/> the PCReq message to show the path of the existing TE LSP. Also,<br/> for all TE LSPs except zero-bandwidth LSPs, when the R bit is set,<br/> the existing bandwidth of the TE LSP to be reoptimized MUST be<br/> supplied in a BANDWIDTH object (see Section 7.7). This BANDWIDTH<br/> object is in addition to the instance of that object used to<br/> describe the desired bandwidth of the reoptimized LSP. For zero-<br/> bandwidth LSPs, the RRO and BANDWIDTH objects that report the<br/> characteristics of the existing TE LSP are optional."</p> <p> &lt;request&gt;::= &lt;RP&gt;<br/> &lt;END-POINTS&gt;<br/> <span class="error">&#91;&lt;LSPA&gt;&#93;</span><br/> <span class="error">&#91;&lt;BANDWIDTH&gt;&#93;</span><br/> <span class="error">&#91;&lt;metric-list&gt;&#93;</span><br/> [&lt;RRO&gt;<span class="error">&#91;&lt;BANDWIDTH&gt;&#93;</span>]<br/> <span class="error">&#91;&lt;IRO&gt;&#93;</span><br/> <span class="error">&#91;&lt;LOAD-BALANCING&gt;&#93;</span></p> <p>Stateful PCEP draft (ref: <a href="https://tools.ietf.org/html/draft-ietf-pce-stateful-pce-15" class="external-link" target="_blank" rel="nofollow noopener">https://tools.ietf.org/html/draft-ietf-pce-stateful-pce-15</a>) introduces PCRpt message and it is defined as below:</p> <p> &lt;PCRpt Message&gt; ::= &lt;Common Header&gt;<br/> &lt;state-report-list&gt;<br/> Where:</p> <p> &lt;state-report-list&gt; ::= &lt;state-report&gt;<span class="error">&#91;&lt;state-report-list&gt;&#93;</span></p> <p> &lt;state-report&gt; ::= <span class="error">&#91;&lt;SRP&gt;&#93;</span><br/> &lt;LSP&gt;<br/> &lt;path&gt;<br/> Where:<br/> &lt;path&gt;::= &lt;intended_path&gt;&lt;attribute-list&gt;<span class="error">&#91;&lt;actual_path&gt;&#93;</span></p> <p> Where:<br/> &lt;intended_path&gt; is represented by the ERO object defined in<br/> section 7.9 of <span class="error">&#91;RFC5440&#93;</span>.<br/> &lt;attribute-list&gt; is defined in <span class="error">&#91;RFC5440&#93;</span> and extended by<br/> PCEP extensions.<br/> &lt;actual_path&gt; is represented by the RRO object defined in<br/> section 7.10 of <span class="error">&#91;RFC5440&#93;</span>.</p> <p>Now attribute-list in RFC 5440 includes the bandwidth object:</p> <p> &lt;attribute-list&gt;::=<span class="error">&#91;&lt;LSPA&gt;&#93;</span><br/> <span class="error">&#91;&lt;BANDWIDTH&gt;&#93;</span><br/> <span class="error">&#91;&lt;metric-list&gt;&#93;</span><br/> <span class="error">&#91;&lt;IRO&gt;&#93;</span></p> <p> &lt;metric-list&gt;::=&lt;METRIC&gt;<span class="error">&#91;&lt;metric-list&gt;&#93;</span></p> <p>So technically the bandwidth reoptimization object can be expected within PCRpt message</p> <p>Proposed fix will make 2 changes:<br/> 1. Add bandwidth reoptimization object to the list of possible objects in PCRpt message<br/> 2. Gracefully handle condition where unexpected object is present in received message</p> <p>(In reply to Al Goddard from comment #1)<br/> &gt; Cisco DE is investigating if/how a 0x5 / 0x2 BW object could be sent from<br/> &gt; the router, but current XR code is only expected to send 0x5 / 0x5 BW Sample.</p> <p>Thx Al for the update. FYI - in the PCRpt sent by CRS which causes the issue, the bandwidth object has bandwidth value as zero whereas the bandwidth reoptimization object has non-zero bandwidth value</p> <p>master: <a href="https://git.opendaylight.org/gerrit/42434" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/42434</a><br/> stable/beryllium: <a href="https://git.opendaylight.org/gerrit/42435" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/42435</a></p> <p>(In reply to Ajay L from comment #3)<br/> &gt; (In reply to Al Goddard from comment #1)<br/> &gt; &gt; Cisco DE is investigating if/how a 0x5 / 0x2 BW object could be sent from<br/> &gt; &gt; the router, but current XR code is only expected to send 0x5 / 0x5 BW Sample.<br/> &gt; <br/> &gt; Thx Al for the update. FYI - in the PCRpt sent by CRS which causes the<br/> &gt; issue, the bandwidth object has bandwidth value as zero whereas the<br/> &gt; bandwidth reoptimization object has non-zero bandwidth value</p> <hr /> <p>From Cisco PCEP DE:</p> <p>Hi Al,</p> <p>I checked the code, and do not see a way for the PCE report message to ever contain the type 2 BW object.<br/> This is based on the code provided in 5.3.3, and the code in the SMU/ engineering images.</p> <p>For now, until such a packet is actually seen, I would be disinclined to think that was the trigger.<br/> As discussed, I will cook up a test XRv image for you to actually send this object from the router though.</p> <p>Thanks,<br/> Jon</p> <p>(In reply to Al Goddard from comment #5)<br/> &gt; (In reply to Ajay L from comment #3)<br/> &gt; &gt; (In reply to Al Goddard from comment #1)<br/> &gt; &gt; &gt; Cisco DE is investigating if/how a 0x5 / 0x2 BW object could be sent from<br/> &gt; &gt; &gt; the router, but current XR code is only expected to send 0x5 / 0x5 BW Sample.<br/> &gt; &gt; <br/> &gt; &gt; Thx Al for the update. FYI - in the PCRpt sent by CRS which causes the<br/> &gt; &gt; issue, the bandwidth object has bandwidth value as zero whereas the<br/> &gt; &gt; bandwidth reoptimization object has non-zero bandwidth value<br/> &gt; <br/> &gt; ----<br/> &gt; From Cisco PCEP DE:<br/> &gt; <br/> &gt; Hi Al,<br/> &gt; <br/> &gt; I checked the code, and do not see a way for the PCE report message to ever<br/> &gt; contain the type 2 BW object.<br/> &gt; This is based on the code provided in 5.3.3, and the code in the SMU/<br/> &gt; engineering images.<br/> &gt; <br/> &gt; For now, until such a packet is actually seen, I would be disinclined to<br/> &gt; think that was the trigger.<br/> &gt; As discussed, I will cook up a test XRv image for you to actually send this<br/> &gt; object from the router though.<br/> &gt; <br/> &gt; Thanks,<br/> &gt; Jon</p> <p>Attaching couple of screenshots from heap dump analysis which show the various objects, including bandwidth and reoptimization bandwidth objects, received from PCRpt (after parsing by ODL code)</p> <p>Attachment Processed-pcrpt-objects.png has been added with description: Processed PCRpt objects</p> <p>Attachment Pending-pcrpt-objects.png has been added with description: Pending PCRpt objects</p> <p>Additional info/request from Cisco DE:</p> <p>Is this is from the heap dump after the heap went OOM?<br/> There are a number of steps between a message being sent and it arriving parsed and settled as internal objects in the controller heap.</p> <p>Am I correct that the 0x5/0x5 BW object is not visible here?</p> <p>If it is possible to correlate this back to an actual message, that would help. <br/> If that is not possible, correlating back to an LSP would help also. (assuming router state is available)<br/> Ideally, a procedure to replicate this would be ideal, or was this a one-off?</p> <p>(In reply to Al Goddard from comment #9)<br/> &gt; Additional info/request from Cisco DE:<br/> &gt; <br/> &gt; <br/> &gt; Is this is from the heap dump after the heap went OOM?<br/> &gt; There are a number of steps between a message being sent and it arriving<br/> &gt; parsed and settled as internal objects in the controller heap.</p> <p>Agree. But analysis so far does not show any issue in ODL parsing logic</p> <p>&gt; <br/> &gt; Am I correct that the 0x5/0x5 BW object is not visible here?</p> <p>0x5/0x5 BW object? Did u mean 0x5/0x1 or 0x5/0x2? I see both of those objects in the heap dump</p> <p>&gt; <br/> &gt; If it is possible to correlate this back to an actual message, that would<br/> &gt; help. <br/> &gt; If that is not possible, correlating back to an LSP would help also. <br/> &gt; (assuming router state is available)</p> <p>Attaching a screenshot showing LSP symbolic name which is "DCCRS2_t7"</p> <p>&gt; Ideally, a procedure to replicate this would be ideal, or was this a one-off?</p> <p>Agree. But I think this has been seen only once so far</p> <p>Attachment lsp-symbolic-path-name.png has been added with description: LSP symbolic path-name</p> <p>Can you answer the two questions from Cisco:</p> <p>The image with the knob to change the bandwidth value is building now and should be available in a few hours.<br/> I also wanted to address a couple of the outstanding points/questions (see below, responses in bold).</p> <p>Thanks,<br/> Jon</p> <p>1. Regarding this:<br/> &gt; Is this is from the heap dump after the heap went OOM?<br/> &gt; There are a number of steps between a message being sent and it arriving<br/> &gt; parsed and settled as internal objects in the controller heap.</p> <p>Agree. But analysis so far does not show any issue in ODL parsing logic</p> <p>My understanding is that it was the parsing of the object that resulted in the eventual looping until OOM state. Was it something else?<br/> So far, the analysis of messages captured before and since does not point to a problem here.</p> <p>Can you answer: _______________</p> <p>2. Do you send an RP object R-bit=1 in any messages? Spec shows this object as part of at PCRep message, not a PCRpt message.</p> <p>The router implements RFC5440 as well as stateful PCEP drafts, so it can originate and process PCRep messages which do contain the RP object.<br/> My understanding (from the attached) was that the memory was being interpreted as a PCRpt message. (Unless this screenshot is a result of the “crafted” message that was used.)'</p> <p>Can you answer: _______________</p> <p>3. Let me know if there are any specific debugs to validate the LSP 7 (shown below) would be sending this object.</p> <p>The ‘dump-messages’ debug can provide such low-level debugging of all messages originating from or arriving on the router, if this can be recreated..</p> <p>(In reply to Al Goddard from comment #12)<br/> &gt; Can you answer the two questions from Cisco:<br/> &gt; <br/> &gt; <br/> &gt; <br/> &gt; The image with the knob to change the bandwidth value is building now and<br/> &gt; should be available in a few hours.<br/> &gt; I also wanted to address a couple of the outstanding points/questions (see<br/> &gt; below, responses in bold).<br/> &gt; <br/> &gt; Thanks,<br/> &gt; Jon<br/> &gt; <br/> &gt; 1. Regarding this:<br/> &gt; &gt; Is this is from the heap dump after the heap went OOM?<br/> &gt; &gt; There are a number of steps between a message being sent and it arriving<br/> &gt; &gt; parsed and settled as internal objects in the controller heap.<br/> &gt; <br/> &gt; Agree. But analysis so far does not show any issue in ODL parsing logic<br/> &gt; <br/> &gt; My understanding is that it was the parsing of the object that resulted in<br/> &gt; the eventual looping until OOM state. Was it something else?<br/> &gt; So far, the analysis of messages captured before and since does not point to<br/> &gt; a problem here.<br/> &gt; <br/> &gt; Can you answer: _______________</p> <p>Processing of objects in PCRpt message caused the loop. I was referring to the fact that issue was in the processing of objects and not in de-serializing or parsing the data received from wire into objects. So we still believe that somehow type=2 bandwidth object was received</p> <p>&gt; <br/> &gt; 2. Do you send an RP object R-bit=1 in any messages? Spec shows this<br/> &gt; object as part of at PCRep message, not a PCRpt message.<br/> &gt; <br/> &gt; The router implements RFC5440 as well as stateful PCEP drafts, so it can<br/> &gt; originate and process PCRep messages which do contain the RP object.<br/> &gt; My understanding (from the attached) was that the memory was being<br/> &gt; interpreted as a PCRpt message. (Unless this screenshot is a result of the<br/> &gt; “crafted” message that was used.)'<br/> &gt; <br/> &gt; Can you answer: _______________<br/> &gt; </p> <p>The screenshot is from the OOM heap dump seen in ATT setup, not the crafted one. We do not believe PCRep getting interpreted as PCRpt is happening here. Per RFC 5440: "PCRep is a PCEP message sent by a PCE to a requesting PCC in response to a previously received PCReq message.". So PCRep is supposed to be received by the PCC (router in this case, not the controller)</p> <p>&gt; <br/> &gt; 3. Let me know if there are any specific debugs to validate the LSP 7 (shown<br/> &gt; below) would be sending this object.<br/> &gt; <br/> &gt; The ‘dump-messages’ debug can provide such low-level debugging of all<br/> &gt; messages originating from or arriving on the router, if this can be<br/> &gt; recreated..</p> <p>master: <a href="https://git.opendaylight.org/gerrit/#/c/42434/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/42434/</a></p> Development External issue ID 6242 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02cmv: