BGP Scale tests with >= 10k prefixes fail following Transaction chain failure

During BGP scale testing of the RC4 Helium release candidate we were unable to use the play.py tool to push more than 10k prefixes into the test artifact and obtain a result.

1. 1. The test set up

1. 1. 1. Testsetup
         CPU 4 vCPUs for Linux, no restriction
         RAM: 16G

java -version java version "1.7.0_67"; Java(TM) SE Runtime Environment (build 1.7.0_67-b01); Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
Garbage collection default

Xmx 2G
permGen 0.5G
Xms 128MB
Setup: 1 node
Clustering: No
Replication: No
Persistence: No
DS: (IMDS) Installed features: cdl’s default

1. 1. 1. Test description
         10k BGP paths were pushed into ODL using the play.py tool, the RESTCONF URL RIB was then polled until the 10k prefixes appeared.

Detailed description of the tests

There are 8 tests in the suite:

TEST CASE 1: Waiting for RIB (no idle periods)
TEST CASE 2: Waiting for RIB (with idle periods)
TEST CONTROLLER-1: Count of prefixes in RIB matches
TEST CONTROLLER-2: Waiting for TOPOLOGY(no idle periods)
TEST CONTROLLER-3: Waiting for TOPOLOGY (with idle periods)
TEST CONTROLLER-4: Count of prefixes in RIB matches

TEST CONTROLLER-5: Connection Alive
TEST CONTROLLER-6: Connection Would Stay Alive

1. Test suite starts with the Test Case “Waiting for RIB (no idle periods)”. This test case watches the CPU load and the “uptodate” attribute. When the CPU load drops, it ends its waiting, looks at the “uptodate” and fails if it is still “false”. The meaning is that there is a bottleneck. This test case is marked as “auxiliary” because its failure does not mean the functionality is broken, it may mean that there is just some lock contention or other bottleneck in the code.

2. The test suite then continues with Test Case “Waiting for RIB (with idle periods)”. This one watches only the “uptodate” attribute. It passes when it sees “uptodate:true” or fails on timeout. This test case is marked “critical” because if it fails, it most likely means either the performance is unacceptable or the RIB failed somehow. If the previous test case passed, this test case shall complete in less than 1 second. Otherwise the time this test case took shall be added to the time the previous test case took to determine the processing time.

3. The test then continues with Test Case “Count of prefixes in RIB matches”. This one downloads the entire RIB and counts the prefixes returned. The count must match what was pushed, otherwise you will get a FAIL. As a bonus when the test fails, it will emit the count of prefixes that were actually found in the RIB. This test case is done now to test that the connection can survive even when someone tries to download the RIB.

These tests are repackaged to inspect the RIB +TOPOLOGY.

The failure occurs during RIB tests

015-07-22 11:39:50,322 | ERROR | oupCloseable-4-3 | DOMDataCommitCoordinatorImpl | 165 - org.opendaylight.controller.sal-broker-impl - 1.1.4.Helium-SR4_0 | The commit executor's queue is full - submit task was rejected.
DeadlockDetectingListeningExecutorService{delegate=FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=4999, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=0, Completed Task Count=1367, Total Task Count=6366}}
java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@51d27c8d rejected from FastThreadPoolExecutor

{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=1366, Total Task Count=6367}