Uploaded image for project: 'netconf'
  1. netconf
  2. NETCONF-187

Clustering: Akka timeout after node outage

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Cannot Reproduce
    • None
    • None
    • netconf
    • None
    • Operating System: All
      Platform: All

    • 5762

    Description

      1. Setup a 3-node cluster
      2. Configure and connect a Netconf device and wait until all 3 nodes see it as connected.
      3. Kill node 1.
      4. Here is a large bunch of steps about which I am not sure whether they are needed, in order:
      a) "create data using node 2"
      b) "check the data is seen in the nodes 2 and 3"
      c) "bring node 1 up"
      d) "check the data is seen on node 1" (this fails due to NETCONF-186)
      e) "kill node 1 again"
      f) "delete the test data using node 2"
      g) "wait for the test data to disappear on the nodes 2 and 3"
      5. Bring node 1 up again.
      6. Here is another step about which I am not sure whether it is needed (maybe not): "check that the data disappears on the node 1" (this also fails due to NETCONF-186).
      7. Attempt to create data using node 3

      The result of step 7 will be 500 Server Internal Error:

      <errors xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf"><error><error-type>application</error-type><error-tag>operation-failed</error-tag><error-message>Error creating data </error-message><error-info>TransactionCommitFailedException

      {message=Transaction commit failed, errorList=[RpcError [message=Transaction commit failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.util.concurrent.ExecutionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://opendaylight-cluster-data@10.30.32.104:2550/user/topology-netconf/netconf-test-device/mountpoint#-630093980]] after [5000 ms]]]}

      at org.opendaylight.netconf.topology.pipeline.tx.ProxyWriteOnlyTransaction$1.apply(ProxyWriteOnlyTransaction.java:68)
      at org.opendaylight.netconf.topology.pipeline.tx.ProxyWriteOnlyTransaction$1.apply(ProxyWriteOnlyTransaction.java:64)
      at com.google.common.util.concurrent.Futures$MappingCheckedFuture.mapException(Futures.java:1809)
      at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:85)
      at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:869)
      at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:816)
      at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.createConfigurationData(StatisticsRestconfServiceWrapper.java:152)
      at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.createConfigurationData(RestconfCompositeWrapper.java:89)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
      at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
      at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)
      at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
      at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
      at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
      at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
      at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
      at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
      at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
      at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:256)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
      at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
      at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
      at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
      at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
      at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
      at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
      at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
      at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
      at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
      at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
      at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
      at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:370)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.ExecutionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://opendaylight-cluster-data@10.30.32.104:2550/user/topology-netconf/netconf-test-device/mountpoint#-630093980]] after [5000 ms]
      at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
      at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
      at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
      at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
      at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:78)
      ... 70 more
      Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://opendaylight-cluster-data@10.30.32.104:2550/user/topology-netconf/netconf-test-device/mountpoint#-630093980]] after [5000 ms]
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
      at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
      at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
      at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
      at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
      ... 1 more
      </error-info></error></errors>

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              rudolf.brisuda@pantheon.tech Rudolf Brisuda
              jbehran@cisco.com Jozef Behran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: