https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/BGPCEP-493 bgpcep <p>Possibly a Bug in distributed datastore or other offset-0 component.</p> <p>This is a Boron regression in Singlepeer Changecount suite. Symptoms are slightly different in "mixed" job <span class="error">&#91;0&#93;</span> (topology practically freezes) and in "non-mixed" job <span class="error">&#91;1&#93;</span> (topology only slows down).</p> <p>The TimeoutException occures in karaf.log, followed by several other ERRORs. This does not happen in Prefix Count suite (or anywhere in Beryllium tests).</p> <p>Segments of logs will be attached, as the whole logs are very big and even the segments take too many lines to paste here. Both segments end with two occurences of "raced with transacion PingPongTransaction" error, in the real logs there were many more of such occurrences.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-mixed-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-mixed-only-boron/</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-only-boron/</a></p> <p>Operating System: All<br/> Platform: All</p> BGPCEP-493

Topology freezes or slows down due to java.util.concurrent.TimeoutException

Bug Resolved Done Claudio David Gasparini Vratko Polak Tue, 19 Jul 2016 13:58:26 +0000 Sun, 3 Mar 2019 11:49:48 +0000 Wed, 10 Aug 2016 09:21:12 +0000 Bugzilla Migration Bugzilla Migration BGP 0 5 <p>Attachment logs_20160719.tar.xz has been added with description: Archive with the promised karaf.log segments</p> <p>Routes injection (applciation peer) use-case:<br/> 1. Testing latest snapshots - Boron (2016/07/28)<br/> 2. using ODL Java BGP benchmak application - 500k routes (plain IPv4)</p> <p>Error reproduced with Boron:<br/> java.util.concurrent.TimeoutException: Futures timed out after <span class="error">&#91;5 seconds&#93;</span><br/> The testing app RPC hangs for a long time (implies, that routes are not written to app-rib tree).</p> <p>With unplugged BGP topology provider, test sometimes pass (no errors/RPC finish). </p> <p>The same test runs with Beryllium (latest snapshot) without any problems.</p> <p>More investigation needed.</p> <p>Memory footprint analysis and comparison (Beryllium vs Boron) shows Boron's memory consumuption significant rise which results in observed timeout erros, casused by insane GC activity (application inactivity).<br/> The regression was introduced by add-path related changes - namely OffsetMap - where a new instance is created by each route entry (instead of using shared one).</p> <p>Attachment karaf1.log.zip has been added with description: NPE karaf log</p> <p>I forgot to add larger comment.</p> <p>The attached log is a part of the whole log, it has NPE as the first exaception. The lof is taken from <a href="https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-1node-periodic-bgp-ingest-mixed-only-boron/5/archives/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-1node-periodic-bgp-ingest-mixed-only-boron/5/archives/</a>. Unfortunately, the full kig has 11M zipped (1.4G text) and will be deleted during the weekend.</p> <p>What happened in that suite: tool started to advertised routes and after around 500k routes tho tool side output looked like:<br/> 2016-08-03 18:22:54,502 INFO BGP-Dummy-1: Iteration: 432000 - total remaining prefixes: 567991<br/> 2016-08-03 18:22:55,651 INFO BGP-Dummy-1: Iteration: 433000 - total remaining prefixes: 566991<br/> 2016-08-03 18:22:56,045 INFO BGP-Dummy-1: Iteration: 434000 - total remaining prefixes: 565991<br/> 2016-08-03 18:22:56,305 INFO BGP-Dummy-1: Iteration: 435000 - total remaining prefixes: 564991<br/> 2016-08-03 18:22:56,936 INFO BGP-Dummy-1: Iteration: 436000 - total remaining prefixes: 563991<br/> 2016-08-03 18:22:58,002 WARNING BGP-Dummy-1: Input and output both blocked for 1.0 seconds.<br/> 2016-08-03 18:22:59,414 WARNING BGP-Dummy-1: Input and output both blocked for 1.0 seconds.<br/> 2016-08-03 18:23:00,662 WARNING BGP-Dummy-1: Input and output both blocked for 1.0 seconds.<br/> 2016-08-03 18:23:01,730 INFO BGP-Dummy-1: Iteration: 437000 - total remaining prefixes: 562991<br/> 2016-08-03 18:23:01,951 INFO BGP-Dummy-1: Iteration: 438000 - total remaining prefixes: 561991<br/> 2016-08-03 18:23:01,994 ERROR BGP-Dummy-1: Peer has overstepped the hold timer.<br/> Unhandled exception in thread started by &lt;function job at 0x7fb17575c2a8&gt;<br/> Traceback (most recent call last):<br/> File "play.py", line 1780, in job<br/> state.perform_one_loop_iteration()<br/> File "play.py", line 1676, in perform_one_loop_iteration<br/> self.timer.check_peer_hold_time(self.timer.snapshot_time)<br/> File "play.py", line 1227, in check_peer_hold_time<br/> raise RuntimeError("Peer has overstepped the hold timer.")<br/> RuntimeError: Peer has overstepped the hold timer.<br/> Traceback (most recent call last):<br/> File "play.py", line 1829, in &lt;module&gt;<br/> threaded_job(arguments)<br/> File "play.py", line 1823, in threaded_job<br/> time.sleep(5)</p> <p>Hopefully attached log will show what problems occurred in odl.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/43110/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/43110/</a></p> <p>Boron<br/> &lt;output xmlns="urn:opendaylight:params:xml:ns:yang:odl-bgp-app-peer-benchmark"&gt;<br/> &lt;result&gt;<br/> &lt;duration&gt;61184&lt;/duration&gt;<br/> &lt;rate&gt;8196&lt;/rate&gt;<br/> &lt;count&gt;500000&lt;/count&gt;<br/> &lt;/result&gt;<br/> &lt;/output&gt;</p> <p>Be<br/> &lt;output xmlns="urn:opendaylight:params:xml:ns:yang:odl-bgp-app-peer-benchmark"&gt;<br/> &lt;result&gt;<br/> &lt;rate&gt;9615&lt;/rate&gt;<br/> &lt;count&gt;500000&lt;/count&gt;<br/> &lt;duration&gt;52295&lt;/duration&gt;<br/> &lt;/result&gt;<br/> &lt;/output&gt;</p> <p>stable/boron: <a href="https://git.opendaylight.org/gerrit/#/c/43590/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/43590/</a></p> Blocks BGPCEP-486 Development External References bgpcep_bgp_ingest_mixed_txt_singlepeer_changecount_check_talking_ipv4_topology_count External issue ID 6237 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02cmn: