Uploaded image for project: 'netvirt'
  1. netvirt
  2. NETVIRT-547

Too many open files while booting VMs in 200 nodes scale setup

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Medium
    • Resolution: Cannot Reproduce
    • Carbon
    • None
    • General
    • None
    • Operating System: All
      Platform: All

    • 8017

    Description

      Running on Carbon code from 16/3/2017, I was performing scale testing on a 3-node cluster. I had an openstack with 200 connected compute nodes (200 OVSs) and I was booting VMs. I tried to reach 200 VMs. Somewhere in the process of booting the VMs, the ODLs started suffering from "Too many open files" Exceptions. These can be seen best in ODL1 logs in the karaf.log.1.

      From 20:38:37 - 22:01:10, the only thing this ODL is doing is snapshots of the datastore, as a result of actions the other ODLs are performing. Because the snapshot mechanism was just recently replaced as a result of this bug: https://bugs.opendaylight.org/show_bug.cgi?id=7521, I suspect this is related.

      At 22:06:14 it starts suffering from:
      2017-03-16 22:06:14,840 | WARN | entLoopGroup-4-1 | DefaultChannelPipeline | 141 - io.netty.common - 4.1.8.Final | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
      io.netty.channel.unix.Errors$NativeIoException: accept() failed: Too many open files
      at io.netty.channel.unix.Errors.newIOException(Errors.java:117)[147:io.netty.transport-native-epoll:4.1.8.Final]
      at io.netty.channel.unix.Socket.accept(Socket.java:263)[147:io.netty.transport-native-epoll:4.1.8.Final]
      at io.netty.channel.epoll.AbstractEpollServerChannel$EpollServerSocketUnsafe.epollInReady(AbstractEpollServerChannel.java:129)[147:io.netty.transport-native-epoll:4.1.8.Final]
      at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)[147:io.netty.transport-native-epoll:4.1.8.Final]
      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)[147:io.netty.transport-native-epoll:4.1.8.Final]
      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)[141:io.netty.common:4.1.8.Final]
      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)[141:io.netty.common:4.1.8.Final]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

      Attachments

        1. odl1logs.tar.gz
          458 kB
          Guy Sela
        2. odl2logs.tar.gz
          512 kB
          Guy Sela
        3. odl3logs.tar.gz
          931 kB
          Guy Sela
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Unassigned Unassigned
            guy.sela@hpe.com Guy Sela
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: