Operating System: All
Platform: All

Additional information:

• Topology provider was turned ON during all tests.
• It is triggred even without topology (RIB only is enough).

After long investigation I realized that only about 6000 routes will go through before these timeouts show up. And it does not matter whether I run the topology provider or not. At 7500 I start to get this AKKA timeout error.

To hit this bug configure EXABGP with 10000 routes and then point it to an ODL instance configured with clustering enabled. I advise to switch topology off as RIB is just enough to hit the bug. The attachment is an EXABGP configuration file with 15000 routes which is fairly large but not too large.

Attachment 15k.cfg.xz has been added with description: EXABGP configuration with 15000 routes

How do we unzip this xz file ? tar -xzf does not work with it

I had to download Ez7z on mac to unzip the xz attachment.

(In reply to Moiz Raja from comment #4)
> How do we unzip this xz file ? tar -xzf does not work with it

The 'z' from -xzf is for .gz only. The letter for .xz is 'J'.
Every linux-based tar program I have seen works well with just "tar -xf", no mater if targeted to .tar, .tar.gz (.tgz) or .tar.xz (.txz).

(In reply to Vratko Polák from comment #6)
> (In reply to Moiz Raja from comment #4)
> > How do we unzip this xz file ? tar -xzf does not work with it

Oh, there was no .tar before .xz
In that case, "unxz" command is available in Linux distributions, usually in a package named xz-utils.

https://git.opendaylight.org/gerrit/#/c/14155/ - master

The change increased resiliency against large batches of data but after trying to push 1 million prefixes I still get akka.TimeOutException. Or a "dead letter encountered" log record.

After reviewing the fix I found a suspicious line (in TransactionProxy.java):

this.operationLimiter = new Semaphore(actorContext.getTransactionOutstandingOperationLimit());

I suspect the limit here is too large. It occurs to me that it does not give AKKA enough "breathing space" so when e.g. disk swapping occurs at a time when there is a bunch of huge transactions in the queue and another bunch waiting at the limiter to be transmitted, then AKKA starts getting timeouts.

However consultation with other developers revealed that there are multiple timeouts and other buffering limits involved so I am not sure at all what is wrong here and/or how to fix it.

Increasing the akka timeout to 900 seconds makes the problem go away.

Jozef,

Couple of questions,

1. Which akka timeout did you increase to 900 seconds to fix this issue? Was it the operation timeout or was it the transaction timeout?

2. How do we reproduce this problem with 1 Million prefixes?

3. How much time does it take to ingest 1 Million prefixes?

After more testing I discovered that this exception occurs reliably when I try to push 2 million of prefixes into BGP. Changing the AKKA timeout does not affect anything in this case. The exception will occur after roughly 5 minutes even when I set the timeout to 15 minutes. Additionally I realized that the exception can occur roughly once in 5 tries even when running on 1 million.

Answers:

1. I increased "operational-timeout-in-seconds" in module "distributed-operational-store-module" and "distributed-config-store-module". I have no idea how to set "transaction timeout" to anything because the config files here are different from what I could find by searching "akka" in Google. However I found some config file which mentions "akka" and "timeout" in one section, so I am going to take look on that.

2. I need to build a testcase and it will take a while (the test I use right now uses a repackaged build with custom configuration, I need to extract the customizations etc). Once done I will attach it here.

3. About 4 minutes with IMDS. When CDS does not fail, then it also takes about 4 minutes. When it fails, it may take up to 1 hour (while it generates multi-GB log file).

Steps to reproduce with 1M of routes:

1. Extract ODL tarball into your home directory.
2. Enter the directory that was made by step 2 and run bin/karaf.
3. Enter "feature:install odl-bgpcep-bgp-all".
4. Enter "feature:install odl-restconf-noauth".
5. Install clustering according to your wishes (persistence, replication, etc).
6. Enter "logout" and wait until karaf exits.
7. Copy the file "41-bgp-example.xml" from the attached package into etc/opendaylight/karaf (overwrite the file with the same name that is there).
8. Run bin/karaf again and wait about 5 minutes for ODL to boot (you can use "top" to shorten the wait, watch for the CPU usage of the Java process to drop below about 10% and stay there).
9. In another terminal extract the play.py from the attached package and then run "python play.py --gencount=1000000".

Notes:

• The "41-bgp-example.xml" file from the package sets up topology updating. That makes it much more likely to hit the bug.
• If you intent to change the content of "41-bgp-example.xml", do a DIFF betwen the original file and the one in the package.
• In the last step you can specify any count you want in the --gencount argument, up to 180 million.

Attachment test.tar.gz has been added with description: Package with tools for testing ODL with up to 180 million prefixes

According to Vratko, here is a faster path to hit the bug:

1. Extract ODL tarball into your home directory.
2. Enter the directory that was made by step 2.
3. Copy the file "41-bgp-example.xml" from the attached package into directory etc/opendaylight/karaf (create the directory if it does not exist).
4. Run bin/karaf
5. Enter "feature:install odl-bgpcep-bgp-all".
6. Enter "feature:install odl-restconf-noauth".
7. Install clustering according to your wishes (persistence, replication, etc).
8. In another terminal extract the play.py from the attached package and then run "python play.py --gencount=1000000".

Additionally, try 2 million routes if you have difficulty hitting the bug.

Hi josef,

I could not make your instructions work. The play.py scripts fails with following error, when i try to use it

Traceback (most recent call last):
File "play.py", line 366, in <module>
Main()
File "play.py", line 319, in Main
FromODL,ToODL = ConnectToODL(args.myip, args.myport, args.peerip, args.peerport, CtlLog)
File "play.py", line 192, in ConnectToODL
ODL.connect((peerip, int(peerport)))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 60] Operation timed out

Can you give me pointers what i need to do to reproduce it? I followed your comments written above.

(sorry for very late reply, I was overbooked during this month)

1. Did you try to run the "play.py" command multiple times?
2. Did you try to restart ODL and try again? I sometimes experienced ODL to hang on installation of some feature, maybe in your case it got stuck around the code that handles the connection.
3. Do you have a link to the ODL build you are trying to test?

Basically, the problem is that I could not reproduce this "Operation timed out" error on my setup.

Jozef, in the meantime we did get over the problem of executing play.py. It works

Will track this issue as part of 3340 as it's newer and has more relevant info