<!-- 
RSS generated by JIRA (8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d) at Wed Feb 07 19:55:40 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>OpenDaylight JIRA</title>
    <link>https://jira.opendaylight.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>8.20.10</version>
        <build-number>820010</build-number>
        <build-date>22-06-2022</build-date>
    </build-info>


<item>
            <title>[CONTROLLER-1483] akka.pattern.AskTimeoutException on follower while BGP peer introduces 1M prefixes</title>
                <link>https://jira.opendaylight.org/browse/CONTROLLER-1483</link>
                <project id="10113" key="CONTROLLER">controller</project>
                    <description>&lt;p&gt;SUT: &lt;a href=&quot;http://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.4.0-SNAPSHOT/distribution-karaf-0.4.0-20160208.111345-3983.zip&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.4.0-SNAPSHOT/distribution-karaf-0.4.0-20160208.111345-3983.zip&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Robot &amp;amp; Karaf logs: &lt;a href=&quot;https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-3node-bgpclustering-only-beryllium/19/console&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-3node-bgpclustering-only-beryllium/19/console&lt;/a&gt; (the faster reaction the more data &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.opendaylight.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; - alias - please take into account the regular clean-up of sandbox on weekends ...)&lt;/p&gt;


&lt;p&gt;Leader:&lt;br/&gt;
=======&lt;/p&gt;

&lt;p&gt;2016-02-09 10:09:20,654 | INFO  | h for user karaf | command                          | 220 - org.apache.karaf.log.command - 3.0.3 | ROBOT MESSAGE: Starting test Start_Talking_BGP_Speaker&lt;br/&gt;
2016-02-09 10:09:20,807 | INFO  | oupCloseable-6-2 | StrictBGPPeerRegistry            | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | BGP Open message session parameters differ, session still accepted.&lt;br/&gt;
2016-02-09 10:09:20,808 | INFO  | h for user karaf | command                          | 220 - org.apache.karaf.log.command - 3.0.3 | ROBOT MESSAGE: Starting test Wait_For_Stable_Talking_Ipv4_Topology_1&lt;br/&gt;
2016-02-09 10:09:20,814 | INFO  | oupCloseable-6-2 | BGPSessionImpl                   | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | BGP HoldTimer new value: 180&lt;br/&gt;
2016-02-09 10:09:20,825 | INFO  | oupCloseable-6-2 | BGPPeer                          | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | Session with peer 10.30.32.10 went up with tables: [BgpTableTypeImpl &lt;span class=&quot;error&quot;&gt;&amp;#91;getAfi()=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.Ipv4AddressFamily, getSafi()=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.UnicastSubsequentAddressFamily&amp;#93;&lt;/span&gt;]&lt;br/&gt;
2016-02-09 10:09:20,843 | INFO  | oupCloseable-6-2 | AbstractBGPSessionNegotiator     | 287 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | BGP Session with peer &lt;span class=&quot;error&quot;&gt;&amp;#91;id: 0x5ec9f71d, /10.30.32.10:17900 =&amp;gt; /10.30.32.19:1790&amp;#93;&lt;/span&gt; established successfully.&lt;br/&gt;
2016-02-09 10:10:20,334 | WARN  | lt-dispatcher-15 | Shard                            | 140 - org.opendaylight.controller.sal-akka-raft - 1.3.0.SNAPSHOT | ApplyState took more time than expected. Elapsed Time = 179 ms ApplyState = ApplyState&lt;/p&gt;
{identifier=&apos;member-1-chn-27-txn-5753&apos;, replicatedLogEntry.index =5766, startTime=1264028676460}
&lt;p&gt;2016-02-09 10:10:28,807 | WARN  | lt-dispatcher-49 | ClusterCoreDaemon                | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node &lt;span class=&quot;error&quot;&gt;&amp;#91;akka.tcp://opendaylight-cluster-data@10.30.32.19:2550&amp;#93;&lt;/span&gt; - Marking node(s) as UNREACHABLE &lt;span class=&quot;error&quot;&gt;&amp;#91;Member(address = akka.tcp://opendaylight-cluster-data@10.30.32.147:2550, status = Up)&amp;#93;&lt;/span&gt;&lt;br/&gt;
2016-02-09 10:10:28,818 | INFO  | lt-dispatcher-38 | kka://opendaylight-cluster-data) | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node &lt;span class=&quot;error&quot;&gt;&amp;#91;akka.tcp://opendaylight-cluster-data@10.30.32.19:2550&amp;#93;&lt;/span&gt; - Marking node(s) as REACHABLE &lt;span class=&quot;error&quot;&gt;&amp;#91;Member(address = akka.tcp://opendaylight-cluster-data@10.30.32.147:2550, status = Up)&amp;#93;&lt;/span&gt;&lt;br/&gt;
2016-02-09 10:10:28,818 | INFO  | lt-dispatcher-15 | EntityOwnershipShard             | 140 - org.opendaylight.controller.sal-akka-raft - 1.3.0.SNAPSHOT | member-1-shard-entity-ownership-operational: onPeerDown: PeerDown &lt;span class=&quot;error&quot;&gt;&amp;#91;memberName=member-3, peerId=member-3-shard-entity-ownership-operational&amp;#93;&lt;/span&gt;&lt;/p&gt;


&lt;p&gt;Follower:&lt;br/&gt;
=========&lt;/p&gt;

&lt;p&gt;2016-02-09 10:12:50,483 | WARN  | tp1409018534-532 | BrokerFacade                     | 211 - org.opendaylight.netconf.sal-rest-connector - 1.3.0.SNAPSHOT | Exception by reading OPERATIONAL via Restconf: /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[&lt;/p&gt;
{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}
&lt;p&gt;]&lt;br/&gt;
java.util.concurrent.ExecutionException: ReadFailedException{message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[&lt;/p&gt;
{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}
&lt;p&gt;], errorList=[RpcError [message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[&lt;/p&gt;
{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}
&lt;p&gt;], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection&lt;span class=&quot;error&quot;&gt;&amp;#91;Anchor(akka.tcp://opendaylight-cluster-data@10.30.32.19:2550/), Path(/user/shardmanager-operational/member-1-shard-topology-operational/shard-member-2-txn-1#-1694239398)&amp;#93;&lt;/span&gt;] after &lt;span class=&quot;error&quot;&gt;&amp;#91;5000 ms&amp;#93;&lt;/span&gt;]]}&lt;br/&gt;
	at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:63)&lt;br/&gt;
	at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:76)&lt;br/&gt;
	at org.opendaylight.netconf.sal.restconf.impl.BrokerFacade.readDataViaTransaction(BrokerFacade.java:199)&lt;span class=&quot;error&quot;&gt;&amp;#91;211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.opendaylight.netconf.sal.restconf.impl.BrokerFacade.readOperationalData(BrokerFacade.java:97)&lt;span class=&quot;error&quot;&gt;&amp;#91;211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.readOperationalData(RestconfImpl.java:684)&lt;span class=&quot;error&quot;&gt;&amp;#91;211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.readOperationalData(StatisticsRestconfServiceWrapper.java:114)&lt;span class=&quot;error&quot;&gt;&amp;#91;211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.readOperationalData(RestconfCompositeWrapper.java:77)&lt;span class=&quot;error&quot;&gt;&amp;#91;211:org.opendaylight.netconf.sal-rest-connector:1.3.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.7.0_85&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.7.0_85&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.7.0_85&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at java.lang.reflect.Method.invoke(Method.java:606)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.7.0_85&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)&lt;span class=&quot;error&quot;&gt;&amp;#91;49:com.sun.jersey.jersey-server:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)&lt;span class=&quot;error&quot;&gt;&amp;#91;177:com.sun.jersey.servlet:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)&lt;span class=&quot;error&quot;&gt;&amp;#91;177:com.sun.jersey.servlet:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)&lt;span class=&quot;error&quot;&gt;&amp;#91;177:com.sun.jersey.servlet:1.17.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)&lt;span class=&quot;error&quot;&gt;&amp;#91;145:org.apache.geronimo.specs.geronimo-servlet_3.0_spec:1.0.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)&lt;span class=&quot;error&quot;&gt;&amp;#91;205:org.apache.shiro.core:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)&lt;span class=&quot;error&quot;&gt;&amp;#91;205:org.apache.shiro.core:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)&lt;span class=&quot;error&quot;&gt;&amp;#91;205:org.apache.shiro.core:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)&lt;span class=&quot;error&quot;&gt;&amp;#91;204:org.apache.shiro.web:1.2.3&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)&lt;span class=&quot;error&quot;&gt;&amp;#91;160:org.ops4j.pax.web.pax-web-jetty:3.1.4&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)&lt;span class=&quot;error&quot;&gt;&amp;#91;160:org.ops4j.pax.web.pax-web-jetty:3.1.4&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75)&lt;span class=&quot;error&quot;&gt;&amp;#91;160:org.ops4j.pax.web.pax-web-jetty:3.1.4&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.Server.handle(Server.java:370)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)&lt;span class=&quot;error&quot;&gt;&amp;#91;151:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.7.0_85&amp;#93;&lt;/span&gt;&lt;br/&gt;
Caused by: ReadFailedException{message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[&lt;/p&gt;
{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}
&lt;p&gt;], errorList=[RpcError [message=Error checking ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[&lt;/p&gt;
{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=example-ipv4-topology-1}
&lt;p&gt;], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection&lt;span class=&quot;error&quot;&gt;&amp;#91;Anchor(akka.tcp://opendaylight-cluster-data@10.30.32.19:2550/), Path(/user/shardmanager-operational/member-1-shard-topology-operational/shard-member-2-txn-1#-1694239398)&amp;#93;&lt;/span&gt;] after &lt;span class=&quot;error&quot;&gt;&amp;#91;5000 ms&amp;#93;&lt;/span&gt;]]}&lt;br/&gt;
	at org.opendaylight.controller.cluster.datastore.RemoteTransactionContext$1.onComplete(RemoteTransactionContext.java:201)&lt;span class=&quot;error&quot;&gt;&amp;#91;143:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.OnComplete.internal(Future.scala:247)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.OnComplete.internal(Future.scala:245)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection&lt;span class=&quot;error&quot;&gt;&amp;#91;Anchor(akka.tcp://opendaylight-cluster-data@10.30.32.19:2550/), Path(/user/shardmanager-operational/member-1-shard-topology-operational/shard-member-2-txn-1#-1694239398)&amp;#93;&lt;/span&gt;] after &lt;span class=&quot;error&quot;&gt;&amp;#91;5000 ms&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)&lt;span class=&quot;error&quot;&gt;&amp;#91;125:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)&lt;span class=&quot;error&quot;&gt;&amp;#91;128:com.typesafe.akka.actor:2.3.14&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.7.0_85&amp;#93;&lt;/span&gt;&lt;/p&gt;</description>
                <environment>&lt;p&gt;Operating System: All&lt;br/&gt;
Platform: All&lt;/p&gt;</environment>
        <key id="26037">CONTROLLER-1483</key>
            <summary>akka.pattern.AskTimeoutException on follower while BGP peer introduces 1M prefixes</summary>
                <type id="10104" iconUrl="https://jira.opendaylight.org/secure/viewavatar?size=xsmall&amp;avatarId=10303&amp;avatarType=issuetype">Bug</type>
                                                <status id="5" iconUrl="https://jira.opendaylight.org/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10000">Done</resolution>
                                        <assignee username="rovarga">Robert Varga</assignee>
                                    <reporter username="rsajben@cisco.com">Radovan Sajben</reporter>
                        <labels>
                    </labels>
                <created>Tue, 9 Feb 2016 10:34:32 +0000</created>
                <updated>Thu, 19 Oct 2017 14:59:14 +0000</updated>
                            <resolved>Tue, 2 May 2017 12:56:11 +0000</resolved>
                                    <version>Beryllium</version>
                                                    <component>clustering</component>
                        <due></due>
                            <votes>0</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="51260" author="rsajben@cisco.com" created="Tue, 9 Feb 2016 10:38:42 +0000"  >&lt;p&gt;Test references:&lt;br/&gt;
================&lt;/p&gt;

&lt;p&gt;bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_wait_for_stable_talking_ipv4_topology_2&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_wait_for_stable_talking_ipv4_topology_3&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_check_talking_ipv4_topology_count_2&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_check_talking_ipv4_topology_count_3&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_wait_for_stable_ipv4_topology_after_listening_2&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_wait_for_stable_ipv4_topology_after_listening_3&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_check_for_empty_ipv4_topology_after_listening_2&lt;br/&gt;
bgpcep_bgpclustering_txt_010_singlepeer_prefixcount_check_for_empty_ipv4_topology_after_listening_3&lt;/p&gt;</comment>
                            <comment id="51261" author="tpantelis" created="Mon, 15 Feb 2016 06:31:46 +0000"  >&lt;p&gt;Please supply the full karaf logs.&lt;/p&gt;</comment>
                            <comment id="51262" author="tpantelis" created="Mon, 15 Feb 2016 06:32:17 +0000"  >&lt;p&gt;And attach to the bug.&lt;/p&gt;

&lt;p&gt;(In reply to Tom Pantelis from comment #2)&lt;br/&gt;
&amp;gt; Please supply the full karaf logs.&lt;/p&gt;</comment>
                            <comment id="51263" author="rovarga" created="Thu, 25 Feb 2016 22:02:52 +0000"  >&lt;p&gt;Is the operational datastore replicated? What is the JVM sizing/parameters in this test?&lt;/p&gt;</comment>
                            <comment id="51264" author="rovarga" created="Wed, 13 Apr 2016 11:35:19 +0000"  >&lt;p&gt;We have analyzed the problem manifested here. Essentially it boils down to the ShardLeader being too busy to respond to the frontend message before the timeout expires. This opened the question of how state transitions occur in the system, as leadership changes have an impact on running transactions.&lt;/p&gt;

&lt;p&gt;This issue has two essential causes:&lt;/p&gt;

&lt;p&gt;1) Frontend-&amp;gt;Backend messages are not idempotent.&lt;/p&gt;

&lt;p&gt;Frontend needs to cover the case when a message gets lost (for whatever reasons), hence it places an upper bound on how long it waits for the response to arrive. When that happens, it cannot re-submit the message because the previous message may still be processed, which could mean that we run the same transaction twice (for the case of BatchedModifications, other messages have similar problems).&lt;/p&gt;

&lt;p&gt;2) Leader-&amp;gt;Follower state is not fully synchronized.&lt;/p&gt;

&lt;p&gt;We only persist the results of a transaction, not the intermediate state leading to those results. This works out (mostly) fine for cases where the client and leader are co-located, as all required state is kept on the client. For client talking to a remote Leader, we have state on both client and leader. In the event of a Leader change we end up losing the leader-side state of the transaction, hence a transaction spanning the transition will fail. The client does not keep the state it has communicated to the Leader via BatchedModifications and so it cannot replay it to the new leader. Further problems arise around transaction chain existence (e.g. the follower does not have the corresponding actors), hence these will inherently fail, too.&lt;/p&gt;

&lt;p&gt;There is a third problem, reported in BUG-5153, where a client restart does not flush Leader-side state. While that problem has been partially addressed by using wall-time timestamps, we need to solved it again so that we minimize the replicated (and persisted) state we keep about transactions. More specifically, if the backend detects a frontend restart, it should prune any state pertaining to that frontend instance.&lt;/p&gt;

&lt;p&gt;The solution involves introducing/formalizing/modifying a few concepts already present in the implementation and describing their semantic meaning and defining the frontend/backend interactions using those concepts. This will allow us to reason about recovery strategies in face of frontend/backend communication failures, including issues stemming from leader restarts and leader location changes. That will then drive requirements on what information the actual Shard FSM looks like, and by extension, define what events need to be persisted (and replicated via RAFT).&lt;/p&gt;

&lt;p&gt;I will post an asciidoc document describing the intent and post a patch here.&lt;/p&gt;</comment>
                            <comment id="51265" author="tpantelis" created="Wed, 13 Apr 2016 12:49:32 +0000"  >&lt;p&gt;Actually the front-end does keep the modification state prior to ready so it can be replayed - I have a prototype draft I did a while back for this.&lt;/p&gt;

&lt;p&gt;(In reply to Robert Varga from comment #5)&lt;br/&gt;
The client does not keep the state it has communicated to the Leader&lt;br/&gt;
&amp;gt; via BatchedModifications and so it cannot replay it to the new leader.&lt;/p&gt;</comment>
                            <comment id="51266" author="rovarga" created="Fri, 15 Apr 2016 12:15:51 +0000"  >&lt;p&gt;I am not saying it cannot &amp;#8211; but it currently keeps only the last batch (in RemoteTransactionContext). I also think that keeping around the entire transaction will be problematic for large transactions.&lt;/p&gt;</comment>
                            <comment id="51267" author="tpantelis" created="Fri, 15 Apr 2016 12:59:00 +0000"  >&lt;p&gt;I changed it to keep the entire trail of transaction operations but in RemoteTransactionContextSupport such that it can retry the FindPrimaryShard. The RemoteTransactionContextSupport already cached the client transaction operations up to the point that if finds the primary shard - I just extended it to cache behind that point in case the RemoteTransactionContext failed. It was just a prototype, obviously needed more work to handle duplicate messages on the leader.&lt;/p&gt;

&lt;p&gt;Anyway, I see few scenarios for front-end transaction disruption due to loss of known leader:&lt;/p&gt;

&lt;p&gt; 1) The current leader is deposed for some reason but is still running. In this case it will forward all current front-end transaction state to the new leader and subsequent messages.&lt;/p&gt;

&lt;p&gt; 2) The current leader is shutdown and successfully transfers leadership. It will forward all current front-end transaction state to the new leader (or attempt to). However subsequent messages will be lost as the front-end continues to send to the prior leader actor selection. &lt;/p&gt;

&lt;p&gt; 3) The current leader process dies ungracefully. All cached leader state is lost and all subsequent messages will also be lost.&lt;/p&gt;

&lt;p&gt; 4) The current leader is still running but the front-end loses connection due to network failure. Subsequent messages could succeed if the messages are retried and the network heals in time.&lt;/p&gt;

&lt;p&gt;My prototype patch is an attempt to handle 2, 3 and 4 by finding the new leader and replaying all cached client operations (up to and including &quot;ready&quot;). Interspersed read operations for R/W transactions are problematic though as well as CanCommit and Commit messages for cross-shard transactions.&lt;/p&gt;

&lt;p&gt;Maybe instead of caching and replaying front-end operations on loss of current leader, we can encapsulate message retries in a custom AskPattern that internally attempts to find the new leader if a message times out. The RemoteTransactionContext would create the pattern instance up-front with the initial actor selection.&lt;/p&gt;

&lt;p&gt;No matter how and if we implemented front-end retries, the messages would have to be idempotent in the shard.&lt;/p&gt;</comment>
                            <comment id="51268" author="tpantelis" created="Fri, 15 Apr 2016 13:14:41 +0000"  >&lt;p&gt;The custom ask pattern that retries the current message wouldn&apos;t handle #3, in that case the front-end would have to replay all prior operations after finding the new leader. &lt;/p&gt;

&lt;p&gt;Or, another possibility, would be to replicate intermediate transaction operations prior to the final commit to the followers so they could pick it up should they become leader. This would be complicated for R/W transactions where the intermediate operations are handled by a separate transaction actor (we would probably have to eliminate this actor).&lt;/p&gt;</comment>
                            <comment id="51269" author="rovarga" created="Fri, 15 Apr 2016 14:10:44 +0000"  >&lt;p&gt;Good points, but I do believe we have problems also around transaction chains (and their associated actors), hence I am writing up the analysis.&lt;/p&gt;

&lt;p&gt;For addressing this issue, I want to start we a clean-slate design and then map it to actual implementation. State management and replication needs to be documented so that it can be reviewed and known to be correct &amp;#8211; for all interactions and all failure modes.&lt;/p&gt;

&lt;p&gt;I know this is not a quick win and it is going to take a lot of effort, hence the approach with well-documented design first.&lt;/p&gt;

&lt;p&gt;In any case, the journal will have to contain more events than just DataTreeCandidatePayload, and state related to transactions (like their identifiers) will have to be persisted.&lt;/p&gt;</comment>
                            <comment id="51270" author="tpantelis" created="Fri, 15 Apr 2016 14:35:51 +0000"  >&lt;p&gt;Yeah this really needs to be thought thru in detail and documentation would help. There is no quick win for handling all the failure scenarios &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.opendaylight.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; But there&apos;s time to do it right - no need to rush on it.&lt;/p&gt;

&lt;p&gt;(In reply to Robert Varga from comment #10)&lt;br/&gt;
&amp;gt; Good points, but I do believe we have problems also around transaction&lt;br/&gt;
&amp;gt; chains (and their associated actors), hence I am writing up the analysis.&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; For addressing this issue, I want to start we a clean-slate design and then&lt;br/&gt;
&amp;gt; map it to actual implementation. State management and replication needs to&lt;br/&gt;
&amp;gt; be documented so that it can be reviewed and known to be correct &amp;#8211; for all&lt;br/&gt;
&amp;gt; interactions and all failure modes.&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; I know this is not a quick win and it is going to take a lot of effort,&lt;br/&gt;
&amp;gt; hence the approach with well-documented design first.&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; In any case, the journal will have to contain more events than just&lt;br/&gt;
&amp;gt; DataTreeCandidatePayload, and state related to transactions (like their&lt;br/&gt;
&amp;gt; identifiers) will have to be persisted.&lt;/p&gt;</comment>
                            <comment id="51271" author="rovarga" created="Mon, 18 Apr 2016 12:12:12 +0000"  >&lt;p&gt;initial drop of analysis: &lt;a href=&quot;https://git.opendaylight.org/gerrit/37544&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/37544&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="51272" author="anipbu" created="Tue, 6 Sep 2016 18:13:30 +0000"  >&lt;p&gt;The following patch was submitted to the stable/boron branch to fix this bug: &lt;a href=&quot;https://git.opendaylight.org/gerrit/#/c/45127&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/#/c/45127&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;To better assess the impact of this bug and fix, could someone from your team please help us identify the following:&lt;br/&gt;
Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Boron without it? Is there a workaround such that we can write a release note and fix in future Boron SR1?&lt;br/&gt;
Testing: Could you also elaborate on the testing of this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests?&lt;br/&gt;
Impact: Does this fix impact any dependent projects?&lt;/p&gt;</comment>
                            <comment id="51273" author="cdgasparini" created="Thu, 23 Feb 2017 18:24:18 +0000"  >&lt;p&gt;Issue still present on boron&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-3node-periodic-bgpclustering-only-boron/9/archives/odl2_karaf.log.gz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-3node-periodic-bgpclustering-only-boron/9/archives/odl2_karaf.log.gz&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;2017-02-23 14:55:28,852 | INFO  | lt-dispatcher-61 | EntityOwnershipShard             | 187 - org.opendaylight.controller.sal-akka-raft - 1.4.3.SNAPSHOT | Peer address for peer member-1-shard-entity-ownership-operational set to akka.tcp://opendaylight-cluster-data@10.29.12.157:2550/user/shardmanager-operational/member-1-shard-entity-ownership-operational&lt;br/&gt;
2017-02-23 14:55:32,455 | WARN  | on-dispatcher-66 | OperationLimiter                 | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Failed to acquire operation permit for transaction member-2-datastore-operational-fe-0-chn-30-txn-1150&lt;br/&gt;
2017-02-23 14:55:33,766 | WARN  | entLoopGroup-9-1 | OperationLimiter                 | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Failed to acquire operation permit for transaction member-2-datastore-operational-fe-0-chn-29-txn-1157&lt;br/&gt;
2017-02-23 14:55:37,455 | WARN  | on-dispatcher-66 | OperationLimiter                 | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Failed to acquire operation permit for transaction member-2-datastore-operational-fe-0-chn-30-txn-1150&lt;br/&gt;
2017-02-23 14:55:38,767 | WARN  | entLoopGroup-9-1 | OperationLimiter                 | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Failed to acquire operation permit for transaction member-2-datastore-operational-fe-0-chn-29-txn-1157&lt;br/&gt;
2017-02-23 14:55:42,456 | WARN  | on-dispatcher-66 | OperationLimiter                 | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Failed to acquire operation permit for transaction member-2-datastore-operational-fe-0-chn-30-txn-1150&lt;br/&gt;
2017-02-23 14:55:43,726 | ERROR | lt-dispatcher-17 | TransactionChainProxy            | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Tx: member-2-datastore-operational-fe-0-chn-29-txn-1157 - ready future failed for previous Tx member-2-datastore-operational-fe-0-chn-29-txn-1157&lt;br/&gt;
2017-02-23 14:55:43,726 | ERROR | lt-dispatcher-49 | LocalThreePhaseCommitCohort      | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Failed to prepare transaction member-2-datastore-operational-fe-0-chn-29-txn-1156 on backend&lt;br/&gt;
akka.pattern.AskTimeoutException: Ask timed out on &lt;a href=&quot;#1626255105)]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-2-shard-default-operational#1626255105)]&lt;/a&gt; after &lt;span class=&quot;error&quot;&gt;&amp;#91;30000 ms&amp;#93;&lt;/span&gt;. Sender&lt;span class=&quot;error&quot;&gt;&amp;#91;null&amp;#93;&lt;/span&gt; sent message of type &quot;org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction&quot;.&lt;br/&gt;
	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)&lt;span class=&quot;error&quot;&gt;&amp;#91;170:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;170:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;170:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_121&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-02-23 14:55:43,727 | WARN  | lt-dispatcher-62 | ConcurrentDOMDataBroker          | 192 - org.opendaylight.controller.sal-distributed-datastore - 1.4.3.SNAPSHOT | Tx: DOM-CHAIN-28-1156 Error during phase CAN_COMMIT, starting Abort&lt;br/&gt;
akka.pattern.AskTimeoutException: Ask timed out on &lt;a href=&quot;#1626255105)]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-2-shard-default-operational#1626255105)]&lt;/a&gt; after &lt;span class=&quot;error&quot;&gt;&amp;#91;30000 ms&amp;#93;&lt;/span&gt;. Sender&lt;span class=&quot;error&quot;&gt;&amp;#91;null&amp;#93;&lt;/span&gt; sent message of type &quot;org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction&quot;.&lt;br/&gt;
	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)&lt;span class=&quot;error&quot;&gt;&amp;#91;170:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;170:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;170:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)&lt;span class=&quot;error&quot;&gt;&amp;#91;174:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_121&amp;#93;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="51274" author="srini.seetharaman@gmail.com" created="Wed, 15 Mar 2017 02:58:35 +0000"  >&lt;p&gt;Any update on this? I too am seeing this with Boron, even though the SyncStatus for that shard reports &apos;true&apos; over JMX. The way I reproduced this was by doing 2 rounds of bring down the quorum and healing. All write transactions fail now. Is there any logs you need to help debug this more?&lt;/p&gt;</comment>
                            <comment id="51275" author="srini.seetharaman@gmail.com" created="Wed, 15 Mar 2017 03:25:03 +0000"  >&lt;p&gt;(In reply to Srini Seetharaman from comment #17)&lt;br/&gt;
&amp;gt; Any update on this? I too am seeing this with Boron, even though the&lt;br/&gt;
&amp;gt; SyncStatus for that shard reports &apos;true&apos; over JMX. The way I reproduced this&lt;br/&gt;
&amp;gt; was by doing 2 rounds of bring down the quorum and healing. All write&lt;br/&gt;
&amp;gt; transactions fail now. Is there any logs you need to help debug this more?&lt;/p&gt;

&lt;p&gt;Just to add to my note - My cluster is based on 3 ODL docker containers running on the same machine. It is running a simple app that is monitoring the machine&apos;s CPU and memory using JMX, and doing 1 write transaction to the datastore every sec. I&apos;m not stressing the system, but the writes fail with the following commit error on the shard leader:&lt;/p&gt;

&lt;p&gt;2017-03-15 02:37:24,099 | WARN  | ult-dispatcher-2 | ShardDataTree                    | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | member-2-shard-infinera-system-operational: Current transaction member-2-datastore-operational-fe-0-txn&lt;br/&gt;
-5383 has timed out after 15000 ms in state COMMIT_PENDING&lt;br/&gt;
2017-03-15 02:37:24,100 | WARN  | ult-dispatcher-2 | ShardDataTree                    | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | member-2-shard-infinera-system-operational: Transaction member-2-datastore-operational-fe-0-txn-5383 is&lt;br/&gt;
 still committing, cannot abort&lt;br/&gt;
2017-03-15 02:37:25,690 | ERROR | lt-dispatcher-16 | LocalThreePhaseCommitCohort      | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | Failed to prepare transaction member-2-datastore-operational-fe-0-txn-5824 on backend&lt;br/&gt;
akka.pattern.AskTimeoutException: Ask timed out on &lt;a href=&quot;#742189228)]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-2-shard-infinera-system-operational#742189228)]&lt;/a&gt; after &lt;span class=&quot;error&quot;&gt;&amp;#91;30000 ms&amp;#93;&lt;/span&gt;. Sender&lt;span class=&quot;error&quot;&gt;&amp;#91;null&amp;#93;&lt;/span&gt; sent message of type &quot;org.opendaylight&lt;br/&gt;
.controller.cluster.datastore.messages.ReadyLocalTransaction&quot;.&lt;br/&gt;
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_111&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-03-15 02:37:25,690 | WARN  | ult-dispatcher-2 | ConcurrentDOMDataBroker          | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | Tx: DOM-6044 Error during phase CAN_COMMIT, starting Abort&lt;br/&gt;
akka.pattern.AskTimeoutException: Ask timed out on &lt;a href=&quot;#742189228)]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-2-shard-infinera-system-operational#742189228)]&lt;/a&gt; after &lt;span class=&quot;error&quot;&gt;&amp;#91;30000 ms&amp;#93;&lt;/span&gt;. Sender&lt;span class=&quot;error&quot;&gt;&amp;#91;null&amp;#93;&lt;/span&gt; sent message of type &quot;org.opendaylight&lt;br/&gt;
.controller.cluster.datastore.messages.ReadyLocalTransaction&quot;.&lt;br/&gt;
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_111&amp;#93;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="51276" author="vzelcamo@cisco.com" created="Wed, 15 Mar 2017 10:04:13 +0000"  >&lt;p&gt;patches merged: &lt;a href=&quot;https://git.opendaylight.org/gerrit/#/q/message:5280+is:closed&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/#/q/message:5280+is:closed&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="51277" author="srini.seetharaman@gmail.com" created="Wed, 15 Mar 2017 14:44:35 +0000"  >&lt;p&gt;Thanks! Will this be pushed to Boron-sr3 too?&lt;/p&gt;</comment>
                            <comment id="51278" author="vzelcamo@cisco.com" created="Wed, 15 Mar 2017 16:31:02 +0000"  >&lt;p&gt;Milestone will be merged in Carbon only. This is new development and therefore not fit for bugfix description within service releases.&lt;/p&gt;</comment>
                            <comment id="51279" author="srini.seetharaman@gmail.com" created="Wed, 15 Mar 2017 17:35:29 +0000"  >&lt;p&gt;Thanks for the note. Without this bug fix, my company&apos;s &quot;Powered by OpenDaylight&quot; controller is going to be pretty unstable in clustered setting because it just takes two cluster downs to freeze up all subsequent write transactions with the akka timeout errors. What is the rationale to wait to Carbon and not support Boron?&lt;/p&gt;</comment>
                            <comment id="51280" author="rovarga" created="Wed, 15 Mar 2017 17:52:30 +0000"  >&lt;p&gt;What do you mean by &apos;two cluster downs&apos;?&lt;/p&gt;

&lt;p&gt;The scope of this issue is a huge rewrite of frontend/backend communication protocol along with incompatible journal layout. Therefore I do not think it is appropriate to introduce it in an SR. The soonest this could get into Boron is SR4, which ships after the initial Carbon release...&lt;/p&gt;</comment>
                            <comment id="51281" author="srini.seetharaman@gmail.com" created="Wed, 15 Mar 2017 18:06:40 +0000"  >&lt;p&gt;(In reply to Robert Varga from comment #23)&lt;br/&gt;
&amp;gt; What do you mean by &apos;two cluster downs&apos;?&lt;/p&gt;

&lt;p&gt;In my stable/boron based setup, I do the following and am able to reproduce this timeout issue very easily. Sometimes it  takes doing these steps once. Almost always I get it within two runs.&lt;/p&gt;

&lt;p&gt;1. Bring up three controller instances each within a Docker container. &lt;br/&gt;
2. Once the cluster is up and steadily updating the operational store with the memory/CPU usage info, I bring down the cluster by making the interface of two Docker containers as down. &lt;br/&gt;
3. sleep 60 secs&lt;br/&gt;
4. Bring up the two interfaces for the containers and verify the shard members in all three instances. Once data store sync status says &quot;True&quot; and log prints &quot;shard-manager-operational: All shard are ready&quot;, I look at the logs. I see the following every second after that.&lt;/p&gt;

&lt;p&gt;2017-03-15 18:01:47,368 | ERROR | lt-dispatcher-35 | LocalThreePhaseCommitCohort      | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | Failed to prepare transaction member-3-datastore-operational-fe-0-txn-5445 on backend&lt;br/&gt;
akka.pattern.AskTimeoutException: Ask timed out on &lt;a href=&quot;#-949816139)]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-3-shard-infinera-system-operational#-949816139)]&lt;/a&gt; after &lt;span class=&quot;error&quot;&gt;&amp;#91;30000 ms&amp;#93;&lt;/span&gt;. Sender&lt;span class=&quot;error&quot;&gt;&amp;#91;null&amp;#93;&lt;/span&gt; sent message of type &quot;org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction&quot;.&lt;br/&gt;
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_111&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-03-15 18:01:47,368 | WARN  | lt-dispatcher-20 | ConcurrentDOMDataBroker          | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | Tx: DOM-5445 Error during phase CAN_COMMIT, starting Abort&lt;br/&gt;
akka.pattern.AskTimeoutException: Ask timed out on &lt;a href=&quot;#-949816139)]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-3-shard-infinera-system-operational#-949816139)]&lt;/a&gt; after &lt;span class=&quot;error&quot;&gt;&amp;#91;30000 ms&amp;#93;&lt;/span&gt;. Sender&lt;span class=&quot;error&quot;&gt;&amp;#91;null&amp;#93;&lt;/span&gt; sent message of type &quot;org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction&quot;.&lt;br/&gt;
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)&lt;span class=&quot;error&quot;&gt;&amp;#91;171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)&lt;span class=&quot;error&quot;&gt;&amp;#91;175:com.typesafe.akka.actor:2.4.7&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at java.lang.Thread.run(Thread.java:745)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_111&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-03-15 18:01:47,374 | WARN  | r-ControlerTimer | GenericTransactionUtils          | 299 - com.infinera.sdn.utils.transaction - 0.1.0.SNAPSHOT | Transaction for add of object ControllerInfoAugment &lt;span class=&quot;error&quot;&gt;&amp;#91;_controllerRole=Slave, _controllerStatus=Up, _memberName=member-3&amp;#93;&lt;/span&gt; failed with error {}&lt;br/&gt;
2017-03-15 18:01:57,518 | WARN  | lt-dispatcher-33 | ShardDataTree                    | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | member-3-shard-infinera-system-operational: Current transaction member-3-datastore-operational-fe-0-txn-5380 has timed out after 15000 ms in state COMMIT_PENDING&lt;br/&gt;
2017-03-15 18:01:57,519 | WARN  | lt-dispatcher-33 | ShardDataTree                    | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | member-3-shard-infinera-system-operational: Transaction member-3-datastore-operational-fe-0-txn-5380 is still committing, cannot abort&lt;/p&gt;</comment>
                            <comment id="51282" author="srini.seetharaman@gmail.com" created="Wed, 15 Mar 2017 18:36:10 +0000"  >&lt;p&gt;I wonder perhaps if my akka.conf is incorrect. I pulled the latest from boron-sr2 and am using it. Here it is. Can you comment on whether anything is missing or can potential trigger this issue? Thanks!&lt;/p&gt;

&lt;p&gt;odl-cluster-data {&lt;br/&gt;
  akka {&lt;br/&gt;
    loglevel = &quot;INFO&quot;&lt;br/&gt;
    loggers = &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;akka.event.slf4j.Slf4jLogger&amp;quot;&amp;#93;&lt;/span&gt;&lt;br/&gt;
    logger-startup-timeout = 300s&lt;/p&gt;

&lt;p&gt;    remote {&lt;br/&gt;
      netty.tcp &lt;/p&gt;
{
        hostname = &quot;172.17.4.23&quot;
        port = 2550
      }
&lt;p&gt;    }&lt;/p&gt;

&lt;p&gt;    cluster &lt;/p&gt;
{
      seed-nodes = [&quot;akka.tcp://opendaylight-cluster-data@172.17.4.21:2550&quot;,
                    &quot;akka.tcp://opendaylight-cluster-data@172.17.4.22:2550&quot;,
                    &quot;akka.tcp://opendaylight-cluster-data@172.17.4.23:2550&quot;]
      seed-node-timeout = 12s
      auto-down-unreachable-after = 300s
      roles = [&quot;member-3&quot;]
    }

&lt;p&gt;    persistence {&lt;br/&gt;
        journal {&lt;br/&gt;
            leveldb &lt;/p&gt;
{
                # Set native = off to use a Java-only implementation of leveldb.
                # Note that the Java-only version is not currently considered by Akka to be production quality.
                # native = off
            }
&lt;p&gt;        }&lt;br/&gt;
    }&lt;br/&gt;
  }&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;odl-cluster-rpc {&lt;br/&gt;
  akka {&lt;br/&gt;
    loglevel = &quot;INFO&quot;&lt;br/&gt;
    loggers = &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;akka.event.slf4j.Slf4jLogger&amp;quot;&amp;#93;&lt;/span&gt;&lt;br/&gt;
    logger-startup-timeout = 300s&lt;/p&gt;

&lt;p&gt;    remote {&lt;br/&gt;
      netty.tcp &lt;/p&gt;
{
        hostname = &quot;172.17.4.23&quot;
        port = 2551
      }
&lt;p&gt;    }&lt;/p&gt;

&lt;p&gt;    cluster &lt;/p&gt;
{
      seed-nodes = [&quot;akka.tcp://opendaylight-cluster-data@172.17.4.21:2551&quot;,
                    &quot;akka.tcp://opendaylight-cluster-data@172.17.4.22:2551&quot;,
                    &quot;akka.tcp://opendaylight-cluster-data@172.17.4.23:2551&quot;]
      auto-down-unreachable-after = 300s
    }
&lt;p&gt;  }&lt;br/&gt;
}&lt;/p&gt;</comment>
                            <comment id="51283" author="tpantelis" created="Wed, 15 Mar 2017 19:24:15 +0000"  >&lt;p&gt;This bug is related to timeouts due to high-volume transactions by the BGP plugin. It doesn&apos;t seem like yours is the same issue. AskTimeoutExceptions can occur for various reasons. &lt;/p&gt;

&lt;p&gt;One thing is you shouldn&apos;t set auto-down-unreachable-after. I would suggest re-testing with auto-down-unreachable-after off and, if necessary, the current Boron-SP3.&lt;/p&gt;

&lt;p&gt;(In reply to Srini Seetharaman from comment #25)&lt;br/&gt;
&amp;gt; I wonder perhaps if my akka.conf is incorrect. I pulled the latest from&lt;br/&gt;
&amp;gt; boron-sr2 and am using it. Here it is. Can you comment on whether anything&lt;br/&gt;
&amp;gt; is missing or can potential trigger this issue? Thanks!&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; odl-cluster-data {&lt;br/&gt;
&amp;gt;   akka {&lt;br/&gt;
&amp;gt;     loglevel = &quot;INFO&quot;&lt;br/&gt;
&amp;gt;     loggers = &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;akka.event.slf4j.Slf4jLogger&amp;quot;&amp;#93;&lt;/span&gt;&lt;br/&gt;
&amp;gt;     logger-startup-timeout = 300s&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt;     remote {&lt;br/&gt;
&amp;gt;       netty.tcp &lt;/p&gt;
{
&amp;gt;         hostname = &quot;172.17.4.23&quot;
&amp;gt;         port = 2550
&amp;gt;       }
&lt;p&gt;&amp;gt;     }&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt;     cluster &lt;/p&gt;
{
&amp;gt;       seed-nodes = [&quot;akka.tcp://opendaylight-cluster-data@172.17.4.21:2550&quot;,
&amp;gt;                     &quot;akka.tcp://opendaylight-cluster-data@172.17.4.22:2550&quot;,
&amp;gt;                     &quot;akka.tcp://opendaylight-cluster-data@172.17.4.23:2550&quot;]
&amp;gt;       seed-node-timeout = 12s
&amp;gt;       auto-down-unreachable-after = 300s
&amp;gt;       roles = [&quot;member-3&quot;]
&amp;gt;     }
&lt;p&gt;&amp;gt; &lt;br/&gt;
&amp;gt;     persistence {&lt;br/&gt;
&amp;gt;         journal {&lt;br/&gt;
&amp;gt;             leveldb &lt;/p&gt;
{
&amp;gt;                 # Set native = off to use a Java-only implementation of
&amp;gt; leveldb.
&amp;gt;                 # Note that the Java-only version is not currently
&amp;gt; considered by Akka to be production quality.
&amp;gt;                 # native = off
&amp;gt;             }
&lt;p&gt;&amp;gt;         }&lt;br/&gt;
&amp;gt;     }&lt;br/&gt;
&amp;gt;   }&lt;br/&gt;
&amp;gt; }&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; odl-cluster-rpc {&lt;br/&gt;
&amp;gt;   akka {&lt;br/&gt;
&amp;gt;     loglevel = &quot;INFO&quot;&lt;br/&gt;
&amp;gt;     loggers = &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;akka.event.slf4j.Slf4jLogger&amp;quot;&amp;#93;&lt;/span&gt;&lt;br/&gt;
&amp;gt;     logger-startup-timeout = 300s&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt;     remote {&lt;br/&gt;
&amp;gt;       netty.tcp &lt;/p&gt;
{
&amp;gt;         hostname = &quot;172.17.4.23&quot;
&amp;gt;         port = 2551
&amp;gt;       }
&lt;p&gt;&amp;gt;     }&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt;     cluster &lt;/p&gt;
{
&amp;gt;       seed-nodes = [&quot;akka.tcp://opendaylight-cluster-data@172.17.4.21:2551&quot;,
&amp;gt;                     &quot;akka.tcp://opendaylight-cluster-data@172.17.4.22:2551&quot;,
&amp;gt;                     &quot;akka.tcp://opendaylight-cluster-data@172.17.4.23:2551&quot;]
&amp;gt;       auto-down-unreachable-after = 300s
&amp;gt;     }
&lt;p&gt;&amp;gt;   }&lt;br/&gt;
&amp;gt; }&lt;/p&gt;</comment>
                            <comment id="51284" author="srini.seetharaman@gmail.com" created="Thu, 16 Mar 2017 06:40:57 +0000"  >&lt;p&gt;(In reply to Tom Pantelis from comment #26)&lt;br/&gt;
&amp;gt; This bug is related to timeouts due to high-volume transactions by the BGP&lt;br/&gt;
&amp;gt; plugin. It doesn&apos;t seem like yours is the same issue. AskTimeoutExceptions&lt;br/&gt;
&amp;gt; can occur for various reasons. &lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; One thing is you shouldn&apos;t set auto-down-unreachable-after. I would suggest&lt;br/&gt;
&amp;gt; re-testing with auto-down-unreachable-after off and, if necessary, the&lt;br/&gt;
&amp;gt; current Boron-SP3.&lt;/p&gt;

&lt;p&gt;Just removing that line didn&apos;t help. But, reintroducing some of the parameters that I used to have from pre-Boron helped solve the issue and make my build stable and not have akka-timeouts. Here are the additional things it had. Not sure exactly which one helped.&lt;/p&gt;

&lt;p&gt; odl-cluster-data {&lt;br/&gt;
+  bounded-mailbox &lt;/p&gt;
{
+    mailbox-type = &quot;org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox&quot;
+    mailbox-capacity = 1000
+    mailbox-push-timeout-time = 100ms
+  }
&lt;p&gt;+&lt;br/&gt;
+  metric-capture-enabled = true&lt;br/&gt;
+&lt;br/&gt;
   akka {&lt;br/&gt;
     loglevel = &quot;INFO&quot;&lt;br/&gt;
     loggers = &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;akka.event.slf4j.Slf4jLogger&amp;quot;&amp;#93;&lt;/span&gt;&lt;br/&gt;
     logger-startup-timeout = 300s&lt;/p&gt;

&lt;p&gt;+    actor {&lt;br/&gt;
+      provider = &quot;akka.cluster.ClusterActorRefProvider&quot;&lt;br/&gt;
+      serializers &lt;/p&gt;
{
+        java = &quot;akka.serialization.JavaSerializer&quot;
+        proto = &quot;akka.remote.serialization.ProtobufSerializer&quot;
+        readylocal = &quot;org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransactionSerializer&quot;
+      }
&lt;p&gt;+&lt;br/&gt;
+      serialization-bindings &lt;/p&gt;
{
+        &quot;com.google.protobuf.Message&quot; = proto
+        &quot;org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction&quot; = readylocal
+      }
&lt;p&gt;+&lt;br/&gt;
+      default-dispatcher &lt;/p&gt;
{
+        # Setting throughput to 1 makes the dispatcher fair. It processes 1 message from
+        # the mailbox before moving on to the next mailbox
+        throughput = 1
+      }
&lt;p&gt;+&lt;br/&gt;
+      default-mailbox &lt;/p&gt;
{
+        # When not using a BalancingDispatcher it is recommended that we use the SingleConsumerOnlyUnboundedMailbox
+        # as it is the most efficient for multiple producer/single consumer use cases
+        mailbox-type=&quot;akka.dispatch.SingleConsumerOnlyUnboundedMailbox&quot;
+      }
&lt;p&gt;+    }&lt;br/&gt;
     remote {&lt;br/&gt;
+      log-remote-lifecycle-events = off&lt;br/&gt;
       netty.tcp &lt;/p&gt;
{
         hostname = &quot;127.0.0.1&quot;
         port = 2550
+        maximum-frame-size = 419430400
+        send-buffer-size = 52428800
+        receive-buffer-size = 52428800
       }
&lt;p&gt;     }&lt;/p&gt;</comment>
                            <comment id="51285" author="tpantelis" created="Thu, 16 Mar 2017 13:09:54 +0000"  >&lt;p&gt;You don&apos;t need all that stuff in akka.conf anymore. All of the necessary stuff is in configuration/factory/akka.conf which gets re-written with updated changes on feature re-install. Take a look at the current configuration/initial/akka.conf - it now contains just the lines needed to setup a cluster - all the rest is inherited from factory/akka.conf:&lt;/p&gt;

&lt;p&gt;odl-cluster-data {&lt;br/&gt;
  akka {&lt;br/&gt;
    remote {&lt;br/&gt;
      netty.tcp &lt;/p&gt;
{
        hostname = &quot;127.0.0.1&quot;
        port = 2550
      }
&lt;p&gt;    }&lt;/p&gt;

&lt;p&gt;    cluster &lt;/p&gt;
{
      seed-nodes = [&quot;akka.tcp://opendaylight-cluster-data@127.0.0.1:2550&quot;]

      roles = [
        &quot;member-1&quot;
      ]
    }
&lt;p&gt;}&lt;/p&gt;

&lt;p&gt;Note that the odl-cluster-rpc section is no longer needed nor used.&lt;/p&gt;

&lt;p&gt;(In reply to Srini Seetharaman from comment #27)&lt;br/&gt;
&amp;gt; (In reply to Tom Pantelis from comment #26)&lt;br/&gt;
&amp;gt; &amp;gt; This bug is related to timeouts due to high-volume transactions by the BGP&lt;br/&gt;
&amp;gt; &amp;gt; plugin. It doesn&apos;t seem like yours is the same issue. AskTimeoutExceptions&lt;br/&gt;
&amp;gt; &amp;gt; can occur for various reasons. &lt;br/&gt;
&amp;gt; &amp;gt; &lt;br/&gt;
&amp;gt; &amp;gt; One thing is you shouldn&apos;t set auto-down-unreachable-after. I would suggest&lt;br/&gt;
&amp;gt; &amp;gt; re-testing with auto-down-unreachable-after off and, if necessary, the&lt;br/&gt;
&amp;gt; &amp;gt; current Boron-SP3.&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; Just removing that line didn&apos;t help. But, reintroducing some of the&lt;br/&gt;
&amp;gt; parameters that I used to have from pre-Boron helped solve the issue and&lt;br/&gt;
&amp;gt; make my build stable and not have akka-timeouts. Here are the additional&lt;br/&gt;
&amp;gt; things it had. Not sure exactly which one helped.&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt;  odl-cluster-data {&lt;br/&gt;
&amp;gt; +  bounded-mailbox &lt;/p&gt;
{
&amp;gt; +    mailbox-type =
&amp;gt; &quot;org.opendaylight.controller.cluster.common.actor.MeteredBoundedMailbox&quot;
&amp;gt; +    mailbox-capacity = 1000
&amp;gt; +    mailbox-push-timeout-time = 100ms
&amp;gt; +  }
&lt;p&gt;&amp;gt; +&lt;br/&gt;
&amp;gt; +  metric-capture-enabled = true&lt;br/&gt;
&amp;gt; +&lt;br/&gt;
&amp;gt;    akka {&lt;br/&gt;
&amp;gt;      loglevel = &quot;INFO&quot;&lt;br/&gt;
&amp;gt;      loggers = &lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;akka.event.slf4j.Slf4jLogger&amp;quot;&amp;#93;&lt;/span&gt;&lt;br/&gt;
&amp;gt;      logger-startup-timeout = 300s&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; +    actor {&lt;br/&gt;
&amp;gt; +      provider = &quot;akka.cluster.ClusterActorRefProvider&quot;&lt;br/&gt;
&amp;gt; +      serializers &lt;/p&gt;
{
&amp;gt; +        java = &quot;akka.serialization.JavaSerializer&quot;
&amp;gt; +        proto = &quot;akka.remote.serialization.ProtobufSerializer&quot;
&amp;gt; +        readylocal =
&amp;gt; &quot;org.opendaylight.controller.cluster.datastore.messages.
&amp;gt; ReadyLocalTransactionSerializer&quot;
&amp;gt; +      }
&lt;p&gt;&amp;gt; +&lt;br/&gt;
&amp;gt; +      serialization-bindings &lt;/p&gt;
{
&amp;gt; +        &quot;com.google.protobuf.Message&quot; = proto
&amp;gt; +       
&amp;gt; &quot;org.opendaylight.controller.cluster.datastore.messages.
&amp;gt; ReadyLocalTransaction&quot; = readylocal
&amp;gt; +      }
&lt;p&gt;&amp;gt; +&lt;br/&gt;
&amp;gt; +      default-dispatcher &lt;/p&gt;
{
&amp;gt; +        # Setting throughput to 1 makes the dispatcher fair. It processes 1
&amp;gt; message from
&amp;gt; +        # the mailbox before moving on to the next mailbox
&amp;gt; +        throughput = 1
&amp;gt; +      }
&lt;p&gt;&amp;gt; +&lt;br/&gt;
&amp;gt; +      default-mailbox &lt;/p&gt;
{
&amp;gt; +        # When not using a BalancingDispatcher it is recommended that we
&amp;gt; use the SingleConsumerOnlyUnboundedMailbox
&amp;gt; +        # as it is the most efficient for multiple producer/single consumer
&amp;gt; use cases
&amp;gt; +        mailbox-type=&quot;akka.dispatch.SingleConsumerOnlyUnboundedMailbox&quot;
&amp;gt; +      }
&lt;p&gt;&amp;gt; +    }&lt;br/&gt;
&amp;gt;      remote {&lt;br/&gt;
&amp;gt; +      log-remote-lifecycle-events = off&lt;br/&gt;
&amp;gt;        netty.tcp &lt;/p&gt;
{
&amp;gt;          hostname = &quot;127.0.0.1&quot;
&amp;gt;          port = 2550
&amp;gt; +        maximum-frame-size = 419430400
&amp;gt; +        send-buffer-size = 52428800
&amp;gt; +        receive-buffer-size = 52428800
&amp;gt;        }
&lt;p&gt;&amp;gt;      }&lt;/p&gt;</comment>
                            <comment id="51286" author="rovarga" created="Thu, 16 Mar 2017 16:25:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://git.opendaylight.org/gerrit/49263&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/49263&lt;/a&gt; is the last code patch, this is now waiting for review and testing.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                            <outwardlinks description="blocks">
                                        <issuelink>
            <issuekey id="25412">CONTROLLER-858</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="26040">CONTROLLER-1486</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="23632">BGPCEP-392</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10002">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="26040">CONTROLLER-1486</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="26172">CONTROLLER-1618</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="26726">PERSISTENC-6</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                            <customfield id="customfield_11400" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10208" key="com.atlassian.jira.plugin.system.customfieldtypes:textfield">
                        <customfieldname>External issue ID</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5280</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10201" key="com.atlassian.jira.plugin.system.customfieldtypes:url">
                        <customfieldname>External issue URL</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[https://bugs.opendaylight.org/show_bug.cgi?id=5280]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10206" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Issue Type</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10300"><![CDATA[Bug]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10204" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>ODL SR Target Milestone</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10321"><![CDATA[Carbon]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10000" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i02qun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>