Running on Carbon code from 16/3/2017, I was performing scale testing on a 3-node cluster. I had an openstack with 200 connected compute nodes (200 OVSs) and I was booting VMs. I tried to reach 200 VMs. Somewhere in the process of booting the VMs, the ODLs started suffering from "Too many open files" Exceptions. These can be seen best in ODL1 logs in the karaf.log.1.

From 20:38:37 - 22:01:10, the only thing this ODL is doing is snapshots of the datastore, as a result of actions the other ODLs are performing. Because the snapshot mechanism was just recently replaced as a result of this bug: https://bugs.opendaylight.org/show_bug.cgi?id=7521, I suspect this is related.

At 22:06:14 it starts suffering from:
2017-03-16 22:06:14,840 | WARN | entLoopGroup-4-1 | DefaultChannelPipeline | 141 - io.netty.common - 4.1.8.Final | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept() failed: Too many open files
at io.netty.channel.unix.Errors.newIOException(Errors.java:117)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.unix.Socket.accept(Socket.java:263)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.epoll.AbstractEpollServerChannel$EpollServerSocketUnsafe.epollInReady(AbstractEpollServerChannel.java:129)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)[147:io.netty.transport-native-epoll:4.1.8.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)[141:io.netty.common:4.1.8.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)[141:io.netty.common:4.1.8.Final]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]