SUT: http://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.4.0-SNAPSHOT/distribution-karaf-0.4.0-20160208.111345-3983.zip

Robot & Karaf logs: https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-3node-bgpclustering-only-beryllium/19/console (the faster reaction the more data - alias - please take into account the regular clean-up of sandbox on weekends ...)

Leader:
=======

2016-02-09 10:09:20,654 | INFO | h for user karaf | command | 220 - org.apache.karaf.log.command - 3.0.3 | ROBOT MESSAGE: Starting test Start_Talking_BGP_Speaker
2016-02-09 10:09:20,807 | INFO | oupCloseable-6-2 | StrictBGPPeerRegistry | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | BGP Open message session parameters differ, session still accepted.
2016-02-09 10:09:20,808 | INFO | h for user karaf | command | 220 - org.apache.karaf.log.command - 3.0.3 | ROBOT MESSAGE: Starting test Wait_For_Stable_Talking_Ipv4_Topology_1
2016-02-09 10:09:20,814 | INFO | oupCloseable-6-2 | BGPSessionImpl | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | BGP HoldTimer new value: 180
2016-02-09 10:09:20,825 | INFO | oupCloseable-6-2 | BGPPeer | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | Session with peer 10.30.32.10 went up with tables: [BgpTableTypeImpl [getAfi()=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.Ipv4AddressFamily, getSafi()=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.UnicastSubsequentAddressFamily]]
2016-02-09 10:09:20,843 | INFO | oupCloseable-6-2 | AbstractBGPSessionNegotiator | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | BGP Session with peer [id: 0x5ec9f71d, /10.30.32.10:17900 => /10.30.32.19:1790] established successfully.
2016-02-09 10:10:20,334 | WARN | lt-dispatcher-15 | Shard | 140 - org.opendaylight.controller.sal-akka-raft - 1.3.0.SNAPSHOT | ApplyState took more time than expected. Elapsed Time = 179 ms ApplyState = ApplyState

{identifier='member-1-chn-27-txn-5753', replicatedLogEntry.index =5766, startTime=1264028676460}

2016-02-09 10:10:28,807 | WARN | lt-dispatcher-49 | ClusterCoreDaemon | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node [akka.tcp://opendaylight-cluster-data@10.30.32.19:2550] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://opendaylight-cluster-data@10.30.32.147:2550, status = Up)]
2016-02-09 10:10:28,818 | INFO | lt-dispatcher-38 | kka://opendaylight-cluster-data) | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node [akka.tcp://opendaylight-cluster-data@10.30.32.19:2550] - Marking node(s) as REACHABLE [Member(address = akka.tcp://opendaylight-cluster-data@10.30.32.147:2550, status = Up)]
2016-02-09 10:10:28,818 | INFO | lt-dispatcher-15 | EntityOwnershipShard | 140 - org.opendaylight.controller.sal-akka-raft - 1.3.0.SNAPSHOT | member-1-shard-entity-ownership-operational: onPeerDown: PeerDown [memberName=member-3, peerId=member-3-shard-entity-ownership-operational]

Follower:
=========

2016-02-09 10:12:50,483 | WARN | tp1409018534-532 | BrokerFacade | 211 - org.opendaylight.netconf.sal-rest-connector - 1.3.0.SNAPSHOT | Exception by reading OPERATIONAL via Restconf: /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}

]
java.util.concurrent.ExecutionException: ReadFailedException{message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}

], errorList=[RpcError [message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}

], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@10.30.32.19:2550/), Path(/user/shardmanager-operational/member-1-shard-topology-operational/shard-member-2-txn-1#-1694239398)]] after [5000 ms]]]}
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:63)
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:76)
at org.opendaylight.netconf.sal.restconf.impl.BrokerFacade.readDataViaTransaction(BrokerFacade.java:199)[211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT]
at org.opendaylight.netconf.sal.restconf.impl.BrokerFacade.readOperationalData(BrokerFacade.java:97)[211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.readOperationalData(RestconfImpl.java:684)[211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT]
at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.readOperationalData(StatisticsRestconfServiceWrapper.java:114)[211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT]
at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.readOperationalData(RestconfCompositeWrapper.java:77)[211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_85]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_85]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)[49:com.sun.jersey.jersey-server:1.17.0]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)[177:com.sun.jersey.servlet:1.17.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[177:com.sun.jersey.servlet:1.17.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[177:com.sun.jersey.servlet:1.17.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)[145:org.apache.geronimo.specs.geronimo-servlet_3.0_spec:1.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)[205:org.apache.shiro.core:1.2.3]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)[205:org.apache.shiro.core:1.2.3]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)[205:org.apache.shiro.core:1.2.3]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)[204:org.apache.shiro.web:1.2.3]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)[204:org.apache.shiro.web:1.2.3]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)[160:org.ops4j.pax.web.pax-web-jetty:3.1.4]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)[160:org.ops4j.pax.web.pax-web-jetty:3.1.4]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75)[160:org.ops4j.pax.web.pax-web-jetty:3.1.4]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.Server.handle(Server.java:370)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_85]
Caused by: ReadFailedException{message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}

], errorList=[RpcError [message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}

], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@10.30.32.19:2550/), Path(/user/shardmanager-operational/member-1-shard-topology-operational/shard-member-2-txn-1#-1694239398)]] after [5000 ms]]]}
at org.opendaylight.controller.cluster.datastore.RemoteTransactionContext$1.onComplete(RemoteTransactionContext.java:201)[143:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at akka.dispatch.OnComplete.internal(Future.scala:247)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.OnComplete.internal(Future.scala:245)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)[128:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[128:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)[128:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[128:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@10.30.32.19:2550/), Path(/user/shardmanager-operational/member-1-shard-topology-operational/shard-member-2-txn-1#-1694239398)]] after [5000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)[128:com.typesafe.akka.actor:2.3.14]
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)[128:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)[125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)[128:com.typesafe.akka.actor:2.3.14]
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)[128:com.typesafe.akka.actor:2.3.14]
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)[128:com.typesafe.akka.actor:2.3.14]
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)[128:com.typesafe.akka.actor:2.3.14]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_85]