[NETCONF-32] Feature install of odl-netconf-connector-ssh is very slow Created: 23/Jun/15  Updated: 15/Mar/19  Resolved: 27/Aug/18

Status: Verified
Project: netconf
Component/s: netconf
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Shaleen Saxena Assignee: Shaleen Saxena
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Text File jstack0.txt     Text File jstack1.txt     Text File karaf.log    
Issue Links:
Blocks
blocks CONTROLLER-1412 Clustering : Often odl-cluster-data &... Resolved
External issue ID: 3842

 Description   

This issue is seen with Lithium (RC built on June 23). I haven't tried previous builds of Lithium; but I am sure it will happen on them as well.

The steps are:

  • Download and unzip distribution-karaf-0.3.0-Lithium.zip
  • Start the controller (bin/karaf)
  • Wait about 5 minute for controller start to finish
  • Install netconf (feature:install odl-netconf-connector-ssh)
  • Took over 10 minutes for install to finish.

I have attached the karaf.log. See the timestamps in the following snippet from it:

2015-06-23 09:12:27,661 | INFO | pool-8-thread-1 | DeferredResourceConfig | 219 - com.sun.jersey.jersey-server - 1.17.0 | Instantiated the Application class org.opendaylight.aaa.idm.IdmLightApplication
2015-06-23 09:22:18,449 | INFO | config-pusher | HydrogenDataBrokerAdapter | 194 - org.opendaylight.controller.sal-binding-broker-impl - 1.2.0.Lithium | ForwardedBackwardsCompatibleBroker started.



 Comments   
Comment by Shaleen Saxena [ 23/Jun/15 ]

Attachment karaf.log has been added with description: Karaf log of install

Comment by Shaleen Saxena [ 23/Jun/15 ]

jstack while odl-netconf-connector-ssh install is in progress.

Comment by Shaleen Saxena [ 23/Jun/15 ]

Attachment jstack0.txt has been added with description: First jstack

Comment by Shaleen Saxena [ 23/Jun/15 ]

jstack while odl-netconf-connector-ssh install is in progress.

Comment by Shaleen Saxena [ 23/Jun/15 ]

Attachment jstack1.txt has been added with description: Second jstack

Comment by Tom Pantelis [ 23/Jun/15 ]

Specifically, this thread reveals the culprit:

java.lang.Thread.State: BLOCKED (on object monitor)
at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:218)
waiting to lock <0x00000006c2e081a8> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access$300(NativePRNG.java:125)
at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:119)
at java.security.SecureRandom.generateSeed(SecureRandom.java:517)
at org.apache.sshd.common.random.BouncyCastleRandom.<init>(BouncyCastleRandom.java:57)
at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:48)
at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:41)
at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:37)
at org.apache.sshd.SshBuilder$BaseBuilder.fillWithDefaultValues(SshBuilder.java:117)
at org.apache.sshd.SshBuilder$ServerBuilder.fillWithDefaultValues(SshBuilder.java:365)
at org.apache.sshd.SshBuilder$ServerBuilder.fillWithDefaultValues(SshBuilder.java:361)
at org.apache.sshd.SshBuilder$BaseBuilder.build(SshBuilder.java:234)
at org.apache.sshd.SshBuilder$BaseBuilder.build(SshBuilder.java:254)
at org.apache.sshd.SshServer.setUpDefaultServer(SshServer.java:366)
at org.opendaylight.controller.netconf.ssh.SshProxyServer.<init>(SshProxyServer.java:58)
at org.opendaylight.controller.netconf.ssh.osgi.NetconfSSHActivator.startSSHServer(NetconfSSHActivator.java:104)
at org.opendaylight.controller.netconf.ssh.osgi.NetconfSSHActivator.start(NetconfSSHActivator.java:58)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711)

NetconfSSHActivator is the activator for the netconf-ssh bundle. It is blocked trying to seed the native random number generator.

This can cause a timeout for the 01-netconf config in the ConfigPusher waiting for the odl-netconfig-client-cfg capability b/c activation of the netconf-config-dispatcher bundle is delayed.

Comment by Maros Marsalek [ 24/Jun/15 ]

We have experienced this only with the netconf testtool in a VM (it uses the same ssh proxy server). We have found a workaround, by disabling the Bouncycastle security provider (the default JDK one does not cause this issue) :

https://wiki.opendaylight.org/view/OpenDaylight_Controller:Netconf:Testtool#Slow_creation_of_devices_on_virtual_machines

Could you try disabling BC provider according to the link above and also removing lib/ext/bcprov-jdk15on-1.52.jar in your distribution ? Then just try again.

Can you provide the more detailed information for your environment ? Java version, OS, VM or Native etc.

Comment by Shaleen Saxena [ 24/Jun/15 ]

(In reply to Maros Marsalek from comment #4)

The setup is a VM on OpenStack running Ubuntu 14.4.

> uname -a
Linux ssaxena-dev 3.13.0-46-generic #79-Ubuntu SMP Tue Mar 10 20:06:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

> java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

After following the procedure you outlined, the install went much faster.

Comment by Maros Marsalek [ 25/Jun/15 ]

Glad to hear that the workaround helped.

Now we need to find out whats wrong with BouncyCastle, whether it affects only OpenJDK and if its linked somehow to using Ubuntu/VM.

Comment by Robert Varga [ 13/Nov/15 ]

Move to NETCONFI project.

Comment by Robert Varga [ 27/Aug/18 ]

ShaleenS is this still relevant?

Comment by Shaleen Saxena [ 27/Aug/18 ]

Probably not. We can close this issue.

Comment by Shaleen Saxena [ 27/Aug/18 ]

This is a very old issue. The release is no longer supported.

Generated at Wed Feb 07 20:13:59 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.