[CONTROLLER-1043] Too many file handles opened Created: 24/Nov/14  Updated: 01/Dec/14  Resolved: 01/Dec/14

Status: Resolved
Project: controller
Component/s: netconf
Affects Version/s: Post-Helium
Fix Version/s: None

Type: Bug
Reporter: Wojciech Dec Assignee: Maros Marsalek
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Text File lsof.txt    
External issue ID: 2430

 Description   

Starting karaf on OSX Maverics with the following features integration project stable/helium build of 24 Nov 2014 results in a catastrophic run-away icorrect TCP socket creation/closure for for IPv6 connections. The suspect is netconf. The jvm runs out of file-handles and crashes.

feature:install odl-netconf-connector-all odl-restconf-noauth odl-toaster-ui

Error (although unclear if this is an effect or the root issue)

2014-11-24 20:58:13,456 | INFO | ventExecutor-1-1 | ProtocolSessionPromise | 128 - org.opendaylight.controller.protocol-framework - 0.5.2.SNAPSHOT | Failed to connect to ts2-host4:2022
io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:447)[125:io.netty.transport:4.0.23.Final]
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:298)[125:io.netty.transport:4.0.23.Final]
at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:133)[125:io.netty.transport:4.0.23.Final]
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:115)[125:io.netty.transport:4.0.23.Final]
at org.opendaylight.protocol.framework.ProtocolSessionPromise.connect(ProtocolSessionPromise.java:53)[128:org.opendaylight.controller.protocol-framework:0.5.2.SNAPSHOT]
at org.opendaylight.protocol.framework.ProtocolSessionPromise$BootstrapConnectListener$ReconnectingStrategyListener.operationComplete(ProtocolSessionPromise.java:143)[128:org.opendaylight.controller.protocol-framework:0.5.2.SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)[126:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)[126:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:397)[126:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.PromiseTask.setSuccessInternal(PromiseTask.java:106)[126:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:124)[126:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:338)[126:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[126:io.netty.common:4.0.23.Final]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_71]
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)[125:io.netty.transport:4.0.23.Final]
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:70)[125:io.netty.transport:4.0.23.Final]
at sun.reflect.GeneratedConstructorAccessor95.newInstance(Unknown Source)[:1.7.0_71]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.7.0_71]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)[:1.7.0_71]
at java.lang.Class.newInstance(Class.java:379)[:1.7.0_71]
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:445)[125:io.netty.transport:4.0.23.Final]
... 13 more
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)[:1.7.0_71]
at sun.nio.ch.Net.socket(Net.java:423)[:1.7.0_71]
at sun.nio.ch.Net.socket(Net.java:416)[:1.7.0_71]
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:104)[:1.7.0_71]
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)[:1.7.0_71]
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:58)[125:io.netty.transport:4.0.23.Final]
... 19 more

lsof output gives an extensive list of non deallocated file handles:



 Comments   
Comment by Wojciech Dec [ 24/Nov/14 ]

Attachment lsof.txt has been added with description: lsof output

Comment by Maros Marsalek [ 25/Nov/14 ]

Hi Wojciech,

Could you give me more detailed information about:

  • What features were installed
  • How did you configure the netconf connector for your device (Restconf, initial conifg)
  • Was your device up and running
  • Was ODL able to connect to your device, or it was attempting in a loop(failing) increasing the open files count

Also if you have the log from that run, it might be useful.

Comment by Maros Marsalek [ 01/Dec/14 ]

https://git.opendaylight.org/gerrit/#/c/13198/

https://git.opendaylight.org/gerrit/#/c/13201/

Generated at Wed Feb 07 19:54:32 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.