https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/BGPCEP-385 bgpcep <p>10 iBGP rrc scenario failed for 1M prefixes in ipv4 topology target (Lithium)</p> <p>please have a look at karaf.log</p> <p>BGP Synchronization finished for table TablesKey ... and <br/> Failed to acquire operation permit for transaction member ...</p> <p>Note: prefixes successfully introduced to topology and announced to iBGP rrc peers for low numbers like 10k-100k prefixes</p> <p>peers<br/> =====</p> <p>peers logs contain</p> <p>10x: Target total number of prefixes to be introduced: 100000<br/> 10x: All update messages generated<br/> many times: Input and output both blocked for ... seconds<br/> none of: total_received_update_message_counter: 900000</p> <p>console<br/> =======</p> <p>bgpcep-bgp-ingest.txt.Manypeers Peercount :: BGP performance of ingesting f...<br/> ==============================================================================<br/> Check_That_Ipv4_Topology_Is_Empty :: Wait and check that the examp... waiting 120s<br/> 2016-02-01 17:27:27: actual (expected) prefix count is 0 (0)</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> PASS </td> </tr> </tbody></table> </div> <p>------------------------------------------------------------------------------<br/> Reconfigure_ODL_To_Accept_Connections :: Configure BGP peer module... | PASS |<br/> ------------------------------------------------------------------------------<br/> Start_Talking_BGP_Manager :: Start Python manager to connect speak... | PASS |<br/> ------------------------------------------------------------------------------<br/> Wait_For_Ipv4_Topology :: Wait until example-ipv4-topology reaches... waiting 3220.0<br/> 2016-02-01 17:27:33: actual (expected) prefix count is 0 (1000000)<br/> 2016-02-01 17:27:44: actual (expected) prefix count is 6490 (1000000)<br/> 2016-02-01 17:27:54: actual (expected) prefix count is 23318 (1000000)<br/> 2016-02-01 17:28:05: actual (expected) prefix count is 23318 (1000000)<br/> 2016-02-01 17:28:15: actual (expected) prefix count is 23318 (1000000)<br/> 2016-02-01 17:28:25: actual (expected) prefix count is 23318 (1000000)<br/> 2016-02-01 17:28:35: actual (expected) prefix count is 23960 (1000000)<br/> 2016-02-01 17:28:45: actual (expected) prefix count is 30177 (1000000)<br/> 2016-02-01 17:28:56: actual (expected) prefix count is 36561 (1000000)<br/> 2016-02-01 17:29:06: actual (expected) prefix count is 43822 (1000000)<br/> 2016-02-01 17:29:16: actual (expected) prefix count is 50556 (1000000)<br/> 2016-02-01 17:29:27: actual (expected) prefix count is 57659 (1000000)<br/> 2016-02-01 17:29:37: actual (expected) prefix count is 62079 (1000000)<br/> 2016-02-01 17:29:47: actual (expected) prefix count is 75352 (1000000)<br/> 2016-02-01 17:29:58: actual (expected) prefix count is 82623 (1000000)<br/> 2016-02-01 17:30:12: actual (expected) prefix count is 94136 (1000000)<br/> 2016-02-01 17:30:24: actual (expected) prefix count is 107114 (1000000)<br/> 2016-02-01 17:30:39: actual (expected) prefix count is 116712 (1000000)<br/> 2016-02-01 17:30:55: actual (expected) prefix count is 118980 (1000000)<br/> 2016-02-01 17:31:06: actual (expected) prefix count is 118980 (1000000)<br/> 2016-02-01 17:31:17: actual (expected) prefix count is 118980 (1000000)<br/> 2016-02-01 17:31:28: actual (expected) prefix count is 122804 (1000000)<br/> 2016-02-01 17:31:39: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:31:49: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:32:00: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:32:11: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:32:22: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:32:33: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:32:44: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:32:55: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:33:06: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:33:16: actual (expected) prefix count is 126627 (1000000)<br/> 2016-02-01 17:33:27: actual (expected) prefix count is 131606 (1000000)<br/> 2016-02-01 17:33:38: actual (expected) prefix count is 137537 (1000000)<br/> 2016-02-01 17:33:49: actual (expected) prefix count is 144321 (1000000)<br/> 2016-02-01 17:34:00: actual (expected) prefix count is 149499 (1000000)<br/> 2016-02-01 17:34:12: actual (expected) prefix count is 154614 (1000000)<br/> 2016-02-01 17:34:23: actual (expected) prefix count is 160054 (1000000)<br/> 2016-02-01 17:34:34: actual (expected) prefix count is 167039 (1000000)<br/> 2016-02-01 17:34:51: actual (expected) prefix count is 185780 (1000000)<br/> 2016-02-01 17:35:10: actual (expected) prefix count is 196039 (1000000)<br/> 2016-02-01 17:35:22: actual (expected) prefix count is 200480 (1000000)<br/> 2016-02-01 17:35:33: actual (expected) prefix count is 209110 (1000000)<br/> 2016-02-01 17:35:45: actual (expected) prefix count is 214540 (1000000)<br/> 2016-02-01 17:36:04: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:36:16: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:36:27: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:36:38: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:36:50: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:37:01: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:37:13: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:37:24: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:37:35: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:37:47: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:37:58: actual (expected) prefix count is 216054 (1000000)<br/> 2016-02-01 17:38:10: actual (expected) prefix count is 216197 (1000000)<br/> 2016-02-01 17:38:21: actual (expected) prefix count is 221796 (1000000)<br/> 2016-02-01 17:38:33: actual (expected) prefix count is 227779 (1000000)<br/> 2016-02-01 17:38:44: actual (expected) prefix count is 233735 (1000000)<br/> 2016-02-01 17:38:56: actual (expected) prefix count is 240232 (1000000)<br/> 2016-02-01 17:39:08: actual (expected) prefix count is 245768 (1000000)<br/> 2016-02-01 17:39:20: actual (expected) prefix count is 252202 (1000000)<br/> 2016-02-01 17:39:37: actual (expected) prefix count is 264713 (1000000)<br/> 2016-02-01 17:39:49: actual (expected) prefix count is 271351 (1000000)<br/> 2016-02-01 17:40:09: actual (expected) prefix count is 290581 (1000000)<br/> 2016-02-01 17:40:21: actual (expected) prefix count is 292937 (1000000)<br/> 2016-02-01 17:40:35: actual (expected) prefix count is 305671 (1000000)<br/> 2016-02-01 17:40:51: actual (expected) prefix count is 318981 (1000000)<br/> 2016-02-01 17:41:18: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:41:31: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:41:43: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:42:02: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:42:14: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:42:26: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:42:38: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:42:50: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:43:02: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:43:15: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:43:27: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:43:39: actual (expected) prefix count is 326555 (1000000)<br/> 2016-02-01 17:43:51: actual (expected) prefix count is 327944 (1000000)<br/> 2016-02-01 17:44:03: actual (expected) prefix count is 328264 (1000000)<br/> 2016-02-01 17:44:15: actual (expected) prefix count is 334875 (1000000)<br/> 2016-02-01 17:44:27: actual (expected) prefix count is 341499 (1000000)<br/> 2016-02-01 17:44:40: actual (expected) prefix count is 348160 (1000000)<br/> 2016-02-01 17:44:52: actual (expected) prefix count is 355702 (1000000)<br/> 2016-02-01 17:45:05: actual (expected) prefix count is 362327 (1000000)<br/> 2016-02-01 17:45:17: actual (expected) prefix count is 370097 (1000000)<br/> 2016-02-01 17:45:30: actual (expected) prefix count is 380722 (1000000)<br/> 2016-02-01 17:45:50: actual (expected) prefix count is 397260 (1000000)<br/> 2016-02-01 17:46:12: actual (expected) prefix count is 417419 (1000000)<br/> 2016-02-01 17:46:42: actual (expected) prefix count is 423856 (1000000)<br/> 2016-02-01 17:46:55: actual (expected) prefix count is 423856 (1000000)<br/> 2016-02-01 17:47:07: actual (expected) prefix count is 425929 (1000000)<br/> 2016-02-01 17:47:21: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:47:33: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:47:46: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:47:59: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:48:12: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:48:25: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:48:38: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:48:51: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:49:03: actual (expected) prefix count is 432227 (1000000)<br/> 2016-02-01 17:49:16: actual (expected) prefix count is 436931 (1000000)<br/> 2016-02-01 17:49:29: actual (expected) prefix count is 443489 (1000000)<br/> 2016-02-01 17:49:42: actual (expected) prefix count is 450601 (1000000)<br/> 2016-02-01 17:49:55: actual (expected) prefix count is 458153 (1000000)<br/> 2016-02-01 17:50:09: actual (expected) prefix count is 465288 (1000000)<br/> 2016-02-01 17:50:22: actual (expected) prefix count is 474002 (1000000)<br/> 2016-02-01 17:50:36: actual (expected) prefix count is 481159 (1000000)<br/> 2016-02-01 17:50:55: actual (expected) prefix count is 492945 (1000000)<br/> 2016-02-01 17:51:09: actual (expected) prefix count is 505630 (1000000)<br/> 2016-02-01 17:51:50: actual (expected) prefix count is 521927 (1000000)<br/> 2016-02-01 17:52:10: actual (expected) prefix count is 521927 (1000000)<br/> 2016-02-01 17:52:24: actual (expected) prefix count is 521927 (1000000)<br/> 2016-02-01 17:52:37: actual (expected) prefix count is 521927 (1000000)<br/> 2016-02-01 17:52:50: actual (expected) prefix count is 521927 (1000000)<br/> 2016-02-01 17:53:04: actual (expected) prefix count is 521927 (1000000)<br/> 2016-02-01 17:53:17: actual (expected) prefix count is 527548 (1000000)<br/> 2016-02-01 17:53:30: actual (expected) prefix count is 535550 (1000000)<br/> 2016-02-01 17:53:44: actual (expected) prefix count is 543942 (1000000)<br/> 2016-02-01 17:53:58: actual (expected) prefix count is 550884 (1000000)<br/> 2016-02-01 17:54:19: actual (expected) prefix count is 570927 (1000000)<br/> 2016-02-01 17:54:33: actual (expected) prefix count is 575054 (1000000)<br/> 2016-02-01 17:55:23: actual (expected) prefix count is 583938 (1000000)<br/> 2016-02-01 17:57:43: actual (expected) prefix count is 590214 (1000000)<br/> 2016-02-01 18:04:32: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:04:46: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:05:00: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:05:14: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:05:28: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:05:41: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:05:55: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:06:09: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:06:22: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:06:36: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:06:50: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:07:03: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:07:17: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:07:31: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:07:44: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:07:58: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:08:12: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:08:26: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:08:39: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:08:53: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:09:07: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:09:20: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:09:34: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:09:48: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:10:01: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:10:15: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:10:29: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:10:43: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:10:57: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:11:11: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:11:24: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:11:38: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:11:52: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:12:05: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:12:19: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:12:33: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:12:46: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:13:00: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:13:14: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:13:28: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:13:41: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:13:55: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:14:09: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:14:23: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:14:36: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:14:50: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:15:04: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:15:18: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:15:31: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:15:45: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:15:59: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:16:13: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:16:26: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:16:40: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:16:54: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:17:08: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:17:22: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:17:36: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:17:49: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:18:03: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:18:17: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:18:31: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:18:45: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:18:58: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:19:12: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:19:26: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:19:40: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:19:54: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:20:08: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:20:21: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:20:35: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:20:49: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:21:03: actual (expected) prefix count is 591817 (1000000)<br/> 2016-02-01 18:21:17: actual (expected) prefix count is 591817 (1000000)</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> FAIL </td> </tr> </tbody></table> </div> <p>Operating System: All<br/> Platform: All</p> BGPCEP-385

10 iBGP rrc scenario failed for 1M prefixes in ipv4 topology target

Bug Resolved Won't Do Milos Fabian Radovan Sajben Tue, 2 Feb 2016 09:14:43 +0000 Sun, 3 Mar 2019 11:49:38 +0000 Tue, 18 Oct 2016 09:02:25 +0000 Bugzilla Migration Bugzilla Migration BGP 0 3 <p>Attachment karaf.txt has been added with description: karaf log</p> <p>please see the beryllium result for 10x100 prefixes: <a href="https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-1node-periodic-bgp-ingest-only-beryllium/2/artifact/karaf.log/*view*/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-1node-periodic-bgp-ingest-only-beryllium/2/artifact/karaf.log/*view*/</a></p> <p>2016-02-02 11:30:37,947 | WARN | lt-dispatcher-20 | SimpleShardDataTreeCohort | 143 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.SNAPSHOT | Store Tx member-1-chn-39-txn-53: Data validation failed for path /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)bgp-rib/rib/rib[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)id=example-bgp-rib} <p>]/peer/peer[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)peer-id=bgp://127.0.0.4} <p>].<br/> org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)bgp-rib/rib/rib[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)id=example-bgp-rib} <p>]/peer/peer[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)peer-id=bgp://127.0.0.4} <p>] does not exist. Cannot apply modification to its children.</p> <p>2016-02-02 11:30:38,105 | WARN | oupCloseable-6-6 | DefaultChannelPipeline | 102 - io.netty.common - 4.0.33.Final | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.<br/> java.lang.IllegalStateException: Transaction chain has failed<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:173)<span class="error">&#91;39:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerTransactionChain.checkNotFailed(DOMBrokerTransactionChain.java:73)<span class="error">&#91;143:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerTransactionChain.submit(DOMBrokerTransactionChain.java:84)<span class="error">&#91;143:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerWriteTransaction.submit(AbstractDOMBrokerWriteTransaction.java:137)<span class="error">&#91;143:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT&#93;</span></p> <p>Test suite console:<br/> ===================</p> <p>bgpcep-bgp-ingest.txt.Manypeers Peercount :: BGP performance of ingesting f...<br/> ==============================================================================<br/> Check_For_Empty_Ipv4_Topology_Before_Talking :: Wait for example-i... | PASS |<br/> ------------------------------------------------------------------------------<br/> Reconfigure_ODL_To_Accept_Connections :: Configure BGP peer module... | PASS |<br/> ------------------------------------------------------------------------------<br/> Start_Talking_BGP_Manager :: Start Python manager to connect speak... | PASS |<br/> ------------------------------------------------------------------------------<br/> Wait_For_Ipv4_Topology :: Wait until example-ipv4-topology reaches... 226.0s<br/> 2016-02-02 11:30:36: actual (expected) prefix count is 0 (1000)<br/> 2016-02-02 11:30:46: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:30:56: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:31:06: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:31:16: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:31:26: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:31:36: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:31:46: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:31:56: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:32:06: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:32:16: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:32:27: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:32:37: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:32:47: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:32:57: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:33:07: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:33:17: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:33:27: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:33:37: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:33:47: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:33:57: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:34:07: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:34:17: actual (expected) prefix count is 14 (1000)<br/> 2016-02-02 11:34:27: actual (expected) prefix count is 14 (1000)</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> FAIL </td> </tr> </tbody></table> </div> <p>This test fails due to <a href="https://bugs.opendaylight.org/show_bug.cgi?id=5185" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=5185</a></p> <p>Keyword 'Check_Ipv4_Topology' failed after retrying for 3 minutes 46 seconds. The last error was: 14 != 1000</p> <p>What is the maximum Java heap size?<br/> What garbage collector is used?</p> <p>+ source /w/workspace/bgpcep-csit-1node-periodic-bgp-ingest-only-beryllium/test/csit/scripts/controllermem_6g.sh<br/> ++ CONTROLLERMEM=6144m</p> <p>garbage collector - no info in logs anyway this suite is just a part of the whole BGP performance suite = the same as for the rest ...</p> <p>... current sandbox result passed the topology check: <a href="https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-1node-periodic-bgp-ingest-only-beryllium/1/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-1node-periodic-bgp-ingest-only-beryllium/1/console</a></p> <p>Wait_For_Ipv4_Topology :: Wait until example-ipv4-topology reaches... 49620.0s<br/> 2016-02-08 08:29:08: actual (expected) prefix count is 0 (1000000)<br/> 2016-02-08 08:29:18: actual (expected) prefix count is 4210 (1000000)<br/> ...<br/> 2016-02-08 09:20:05: actual (expected) prefix count is 1000000 (1000000)</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> PASS </td> </tr> </tbody></table> </div> <p>------------------------------------------------------------------------------<br/> Check_Logs_For_Updates :: Check BGP peer logs for received updates. | FAIL |<br/> Keyword 'Check_File_For_Occurence' failed after retrying for 20 seconds. The last error was: '0 &gt;= 2' should be true.</p> <p>and failed on one of bgp peer update count ...</p> <p>On top of possible (still not verified) memory pressure on ODL_SYSTEM, there seem to be problems on robot slave too <span class="error">&#91;0&#93;</span>:<br/> Output: /w/workspace/bgpcep-csit-1node-periodic-bgp-ingest-only-stable-lithium/output.xml<br/> Traceback (most recent call last):<br/> File "/tmp/tmp.UyJmSZoTFJrobot_venv/bin/pybot", line 6, in &lt;module&gt;<br/> run_cli(sys.argv<span class="error">&#91;1:&#93;</span>)<br/> File "/tmp/tmp.UyJmSZoTFJrobot_venv/lib/python2.7/site-packages/robot/run.py", line 466, in run_cli<br/> RobotFramework().execute_cli(arguments)<br/> File "/tmp/tmp.UyJmSZoTFJrobot_venv/lib/python2.7/site-packages/robot/utils/application.py", line 45, in execute_cli<br/> rc = self._execute(arguments, options)<br/> File "/tmp/tmp.UyJmSZoTFJrobot_venv/lib/python2.7/site-packages/robot/utils/application.py", line 87, in _execute<br/> error, details = get_error_details(exclude_robot_traces=False)<br/> File "/tmp/tmp.UyJmSZoTFJrobot_venv/lib/python2.7/site-packages/robot/utils/error.py", line 46, in get_error_details<br/> details = ErrorDetails(exclude_robot_traces=exclude_robot_traces)<br/> File "/tmp/tmp.UyJmSZoTFJrobot_venv/lib/python2.7/site-packages/robot/utils/error.py", line 59, in ErrorDetails<br/> raise exc_value<br/> MemoryError</p> <p>Robot VM now uses only 2GB RAM, so scaling suite should be reviewed for memory inefficiencies.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-only-stable-lithium/116/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-1node-periodic-bgp-ingest-only-stable-lithium/116/console</a></p> <p>Testing locally with Beryllium SR2 and latest - 10 rr-clients, each advertise 10k prefixes.<br/> The log contain a lot of "Failed to acquire operation permit for transaction" and " Generation of DataTreeCandidateNode events took longer than expected. Elapsed time"</p> <p>After some time, ODL runs out of memory:</p> <p>2016-04-29 11:26:20,980 | WARN | oupCloseable-2-6 | BGPSessionImpl | 185 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.2.SNAPSHOT | Failed to send message Update [_attributes=Attributes [_asPath=AsPath{getSegments=[], augmentations={}}, _clusterId=ClusterId{getCluster=[Ipv4Address <span class="error">&#91;_value=127.0.0.1&#93;</span>], augmentations={}}, _localPref=LocalPref{getPref=100, augmentations={}}, _origin=Origin{getValue=Igp, augmentations={}}, _originatorId=OriginatorId{getOriginator=Ipv4Address <span class="error">&#91;_value=127.0.0.1&#93;</span>, augmentations={}}, _unrecognizedAttributes=[], augmentation=[Attributes1 [_mpReachNlri=MpReachNlri [_advertizedRoutes=AdvertizedRoutes [_destinationType=DestinationIpv4Case [_destinationIpv4=DestinationIpv4 [_ipv4Prefixes=[Ipv4Prefixes [_prefix=Ipv4Prefix <span class="error">&#91;_value=2.0.210.48/28&#93;</span>, augmentation=[]]], augmentation=[]], augmentation=[]], augmentation=[]], _afi=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.Ipv4AddressFamily, _cNextHop=Ipv4NextHopCase{getIpv4NextHop=Ipv4NextHop{getGlobal=Ipv4Address <span class="error">&#91;_value=192.0.2.1&#93;</span>, augmentations={}}, augmentations={}}, _safi=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.UnicastSubsequentAddressFamily, augmentation=[]]]]], augmentation=[]] to socket io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Uncaught error from thread <span class="error">&#91;opendaylight-cluster-data-notification-dispatcher-56&#93;</span> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem<span class="error">&#91;opendaylight-cluster-data&#93;</span><br/> 2016-04-29 11:26:29,548 | WARN | on-dispatcher-52 | OperationLimiter | 175 - org.opendaylight.controller.sal-distributed-datastore - 1.3.2.SNAPSHOT | Failed to acquire operation permit for transaction member-1-chn-5-txn-3699-1461921917380<br/> java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Uncaught error from thread <span class="error">&#91;opendaylight-cluster-data-akka.actor.default-dispatcher-17&#93;</span> shutting down JVM since 'akka.jvm-exit-o2016-04-29 11:26:37,565 | WARN | on-dispatcher-52 | OperationLimiter | 175 - org.opendaylight.controller.sal-distributed-datastore - 1.3.2.SNAPSHOT | Failed to acquire operation permit for transaction member-1-chn-5-txn-3699-1461921917380<br/> n-fatal-error' is enabled for ActorSystem<span class="error">&#91;opendaylight-cluster-data&#93;</span><br/> at org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.multiprotocol.rev130919.Attributes1Builder.build(Attributes1Builder.java:38)<br/> at org.opendaylight.protocol.bgp.rib.spi.AbstractRIBSupport.buildUpdate(AbstractRIBSupport.java:241)<br/> 2016-04-29 11:26:45,944 | WARN | on-dispatcher-52 | OperationLimiter | 175 - org.opendaylight.controller.sal-distributed-datastore - 1.3.2.SNAPSHOT | Failed to acquire operation permit for transaction member-1-chn-5-txn-3699-1461921917380<br/> at org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.advertise(AdjRibOutListener.java:143)<br/> at org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.processRouteChange(AdjRibOutListener.java:114)<br/> at org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.processSupportedFamilyRoutes(AdjRibOutListener.java:95)<br/> at org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.onDataTreeChanged(AdjRibOutListener.java:87)<br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)<br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)<br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)<br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<br/> java.lang.OutOfMemoryError: GC overhead limit exceeded</p> <p>Initial try-outs shows that pressure can be lowered by narrowing LocRibWriter DCN listener scope to Effective-RIB-In tables only.</p> <p>stable/beryllium: <a href="https://git.opendaylight.org/gerrit/#/c/38286" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/38286</a></p> Development External issue ID 5185 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02byn: