[NETVIRT-547] Too many open files while booting VMs in 200 nodes scale setup

Fri, 6 Apr 2018 00:02:55 +0000

Running on Carbon code from 16/3/2017, I was performing scale testing on a 3-node cluster. I had an openstack with 200 connected compute nodes (200 OVSs) and I was booting VMs. I tried to reach 200 VMs. Somewhere in the process of booting the VMs, the ODLs started suffering from "Too many open files" Exceptions. These can be seen best in ODL1 logs in the karaf.log.1.

From 20:38:37 - 22:01:10, the only thing this ODL is doing is snapshots of the datastore, as a result of actions the other ODLs are performing. Because the snapshot mechanism was just recently replaced as a result of this bug: https://bugs.opendaylight.org/show_bug.cgi?id=7521, I suspect this is related.

At 22:06:14 it starts suffering from:
2017-03-16 22:06:14,840 | WARN | entLoopGroup-4-1 | DefaultChannelPipeline | 141 - io.netty.common - 4.1.8.Final | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept() failed: Too many open files
at io.netty.channel.unix.Errors.newIOException(Errors.java:117)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.unix.Socket.accept(Socket.java:263)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.epoll.AbstractEpollServerChannel$EpollServerSocketUnsafe.epollInReady(AbstractEpollServerChannel.java:129)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)[141:io.netty.common:4.1.8.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)[141:io.netty.common:4.1.8.Final]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

OpenDaylight JIRA

[NETVIRT-547] Too many open files while booting VMs in 200 nodes scale setup