<!-- 
RSS generated by JIRA (8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d) at Wed Feb 07 19:56:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>OpenDaylight JIRA</title>
    <link>https://jira.opendaylight.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>8.20.10</version>
        <build-number>820010</build-number>
        <build-date>22-06-2022</build-date>
    </build-info>


<item>
            <title>[CONTROLLER-1665] C: write-transactions does not return if local leader moved</title>
                <link>https://jira.opendaylight.org/browse/CONTROLLER-1665</link>
                <project id="10113" key="CONTROLLER">controller</project>
                    <description>&lt;p&gt;Write-transactions rcp invoked on the leader of default module shard. &lt;br/&gt;
url: &lt;a href=&quot;http://10.29.15.39:8181/restconf/operations/odl-mdsal-lowlevel-control:write-transactions&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://10.29.15.39:8181/restconf/operations/odl-mdsal-lowlevel-control:write-transactions&lt;/a&gt;, data: &amp;lt;input xmlns=&quot;tag:opendaylight.org,2017:controller:yang:lowlevel:control&quot;&amp;gt;&lt;br/&gt;
  &amp;lt;id&amp;gt;prefix-1&amp;lt;/id&amp;gt;&lt;br/&gt;
  &amp;lt;seconds&amp;gt;30&amp;lt;/seconds&amp;gt;&lt;br/&gt;
  &amp;lt;transactions-per-second&amp;gt;1000&amp;lt;/transactions-per-second&amp;gt;&lt;br/&gt;
  &amp;lt;chained-transactions&amp;gt;True&amp;lt;/chained-transactions&amp;gt;&lt;br/&gt;
&amp;lt;/input&amp;gt;&lt;/p&gt;

&lt;p&gt;Within this time the leader was moved to different node. But after 30s the write-transactions rpc did not return and after 90s is was aborted from robot side&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/12/archives/log.html.gz#s1-s1-t1-k2-k8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/12/archives/log.html.gz#s1-s1-t1-k2-k8&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The bug is similar to &lt;a href=&quot;https://bugs.opendaylight.org/show_bug.cgi?id=8372&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugs.opendaylight.org/show_bug.cgi?id=8372&lt;/a&gt;, but that time is was without any leader movement, just write-transactions invoked.&lt;/p&gt;

&lt;p&gt;Logs of trans producer node:&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/12/archives/odl1_karaf.log.gz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/12/archives/odl1_karaf.log.gz&lt;/a&gt;&lt;br/&gt;
2017-05-09 04:55:51,788 | INFO  | h for user karaf | command                          | 266 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test Local_To_Remote_Movement&lt;/p&gt;</description>
                <environment>&lt;p&gt;Operating System: All&lt;br/&gt;
Platform: All&lt;/p&gt;</environment>
        <key id="26219">CONTROLLER-1665</key>
            <summary>C: write-transactions does not return if local leader moved</summary>
                <type id="10104" iconUrl="https://jira.opendaylight.org/secure/viewavatar?size=xsmall&amp;avatarId=10303&amp;avatarType=issuetype">Bug</type>
                                                <status id="5" iconUrl="https://jira.opendaylight.org/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10000">Done</resolution>
                                        <assignee username="rovarga">Robert Varga</assignee>
                                    <reporter username="pgubka@cisco.com">Peter Gubka</reporter>
                        <labels>
                    </labels>
                <created>Tue, 9 May 2017 09:08:23 +0000</created>
                <updated>Tue, 25 Jul 2023 08:24:35 +0000</updated>
                            <resolved>Wed, 7 Jun 2017 12:25:03 +0000</resolved>
                                                                    <component>clustering</component>
                        <due></due>
                            <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="52136" author="pgubka@cisco.com" created="Tue, 9 May 2017 09:34:13 +0000"  >&lt;p&gt;Same behavior happened when the write-transactions was run on the default shard leader node and this was isolated(iptables).&lt;br/&gt;
Instead of aborting the rpc after hard timeout, is was aborted after 150s from robot side&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/13/archives/log.html.gz#s1-s1-t1-k2-k18-k3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/13/archives/log.html.gz#s1-s1-t1-k2-k18-k3&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Logs:&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/13/archives/odl1_karaf.log.gz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/13/archives/odl1_karaf.log.gz&lt;/a&gt;&lt;br/&gt;
2017-05-09 05:16:59,386 | INFO  | h for user karaf | command                          | 266 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test Producer_On_Shard_Leader_Node_ChainedTx&lt;/p&gt;</comment>
                            <comment id="52137" author="rovarga" created="Mon, 15 May 2017 14:27:55 +0000"  >&lt;p&gt;For tell-based protocol, after BUG-8422 the deadline for requests is 120 seconds, not 30 seconds (which is reconnect timer).&lt;/p&gt;</comment>
                            <comment id="52138" author="vrpolak" created="Thu, 18 May 2017 15:00:14 +0000"  >&lt;p&gt;&amp;gt; the deadline for requests is 120 seconds&lt;/p&gt;

&lt;p&gt;Re-opening.&lt;br/&gt;
write-transactions have not returned &lt;span class=&quot;error&quot;&gt;&amp;#91;4&amp;#93;&lt;/span&gt; within 180 seconds (30 seconds of work, so 150 seconds overhead, which is more than 120 seconds).&lt;br/&gt;
The scenario was removing shard replica on the leader; module-based shard with tell-based protocol.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;4&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/694/archives/log.html.gz#s1-s20-t1-k2-k8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/694/archives/log.html.gz#s1-s20-t1-k2-k8&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52139" author="rovarga" created="Fri, 19 May 2017 14:20:37 +0000"  >&lt;p&gt;One troublesome thing is:&lt;/p&gt;

&lt;p&gt;java.util.ConcurrentModificationException&lt;br/&gt;
        at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643)&lt;span class=&quot;error&quot;&gt;&amp;#91;:1.8.0_121&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at com.google.common.collect.Iterators$5.next(Iterators.java:558)&lt;span class=&quot;error&quot;&gt;&amp;#91;57:com.google.guava:18.0.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.replayMessages(AbstractProxyTransaction.java:538)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$ReconnectCohort.replayRequests(ProxyHistory.java:232)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractClientHistory$1.replayRequests(AbstractClientHistory.java:310)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractDataStoreClientBehavior.lambda$connectionUp$0(AbstractDataStoreClientBehavior.java:147)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.backendConnectFinished(ClientActorBehavior.java:280)&lt;span class=&quot;error&quot;&gt;&amp;#91;197:org.opendaylight.controller.cds-access-client:1.1.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.lambda$null$3(ClientActorBehavior.java:329)&lt;span class=&quot;error&quot;&gt;&amp;#91;197:org.opendaylight.controller.cds-access-client:1.1.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:118)&lt;span class=&quot;error&quot;&gt;&amp;#91;197:org.opendaylight.controller.cds-access-client:1.1.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:42)&lt;span class=&quot;error&quot;&gt;&amp;#91;197:org.opendaylight.controller.cds-access-client:1.1.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at org.opendaylight.controller.cluster.access.client.AbstractClientActor.onReceiveCommand(AbstractClientActor.java:59)&lt;span class=&quot;error&quot;&gt;&amp;#91;197:org.opendaylight.controller.cds-access-client:1.1.0.Carbon&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)&lt;span class=&quot;error&quot;&gt;&amp;#91;180:com.typesafe.akka.persistence:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.Actor$class.aroundReceive(Actor.scala:497)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)&lt;span class=&quot;error&quot;&gt;&amp;#91;180:com.typesafe.akka.persistence:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:664)&lt;span class=&quot;error&quot;&gt;&amp;#91;180:com.typesafe.akka.persistence:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)&lt;span class=&quot;error&quot;&gt;&amp;#91;180:com.typesafe.akka.persistence:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)&lt;span class=&quot;error&quot;&gt;&amp;#91;180:com.typesafe.akka.persistence:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)&lt;span class=&quot;error&quot;&gt;&amp;#91;173:com.typesafe.akka.actor:2.4.17&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)&lt;span class=&quot;error&quot;&gt;&amp;#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)&lt;span class=&quot;error&quot;&gt;&amp;#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)&lt;span class=&quot;error&quot;&gt;&amp;#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;br/&gt;
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)&lt;span class=&quot;error&quot;&gt;&amp;#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I&apos;ll try to see what the heck is going on.&lt;/p&gt;</comment>
                            <comment id="52140" author="rovarga" created="Fri, 19 May 2017 15:40:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://git.opendaylight.org/gerrit/57438&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/57438&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52141" author="vrpolak" created="Mon, 22 May 2017 07:10:24 +0000"  >&lt;p&gt;&amp;gt; &lt;a href=&quot;https://git.opendaylight.org/gerrit/57438&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/57438&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Even after that merged, this Bug still occurs &lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt; (module-based shard, tell-based protocol, graceful leader move).&lt;br/&gt;
There is nothing suspicious in karaf.log &lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; after 00:38:39,727.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s24-t1-k2-k10&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s24-t1-k2-k10&lt;/a&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/odl1_karaf.log.gz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/odl1_karaf.log.gz&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52142" author="pgubka@cisco.com" created="Mon, 22 May 2017 13:16:57 +0000"  >&lt;p&gt;There is a sandbox run for patches 57598+99&lt;br/&gt;
&lt;a href=&quot;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-3rd/3/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-3rd/3/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52143" author="pgubka@cisco.com" created="Tue, 23 May 2017 07:16:22 +0000"  >&lt;p&gt;Another sandbox run (odl includes patches 57598+99) with debug logging for org.opendaylight.controller.cluster.databroker.actors.dds and org.opendaylight.controller.cluster.access.client&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-3rd/4/archives/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-3rd/4/archives/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52144" author="vrpolak" created="Tue, 23 May 2017 10:25:40 +0000"  >&lt;p&gt;Still occasionally happens &lt;span class=&quot;error&quot;&gt;&amp;#91;9&amp;#93;&lt;/span&gt; when a (module-based, tell protocol) shard replica is removed from the leader member. &lt;/p&gt;

&lt;p&gt;Possibly just because the fix &lt;span class=&quot;error&quot;&gt;&amp;#91;10&amp;#93;&lt;/span&gt; has not been merged yet.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;9&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/719/archives/log.html.gz#s1-s22-t1-k2-k9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/719/archives/log.html.gz#s1-s22-t1-k2-k9&lt;/a&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;10&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://git.opendaylight.org/gerrit/57423&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/57423&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52145" author="vrpolak" created="Tue, 23 May 2017 11:02:37 +0000"  >&lt;p&gt;&amp;gt; shard replica is removed from the leader member.&lt;/p&gt;

&lt;p&gt;In controller-csit-3node-clustering-only-carbon/720 this happened reliably, both for module-based and prefix based shard, but only when the writer/producer was located on the leader member.&lt;/p&gt;</comment>
                            <comment id="52146" author="pgubka@cisco.com" created="Tue, 23 May 2017 17:22:08 +0000"  >&lt;p&gt;New run with debug logs here:&lt;br/&gt;
&lt;a href=&quot;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/8/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/8/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Not returned rest response from write-transactions rpc &lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/8/archives/log.html.gz#s1-s2-t1-k2-k11&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/8/archives/log.html.gz#s1-s2-t1-k2-k11&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Odl zip from: &lt;a href=&quot;https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/37/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/37/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52147" author="pgubka@cisco.com" created="Thu, 25 May 2017 08:57:18 +0000"  >&lt;p&gt;Debug logs including akka&lt;br/&gt;
&lt;a href=&quot;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/11/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/11/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Odl built from controller=63/57763/2:99/57699/3 at &lt;br/&gt;
&lt;a href=&quot;https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/43&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/43&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Not returned rpc&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/11/archives/log.html.gz#s1-s2-t1-k2-k11&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/11/archives/log.html.gz#s1-s2-t1-k2-k11&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52148" author="pgubka@cisco.com" created="Thu, 25 May 2017 10:18:13 +0000"  >&lt;p&gt;New debug logs (with akka)&lt;br/&gt;
&lt;a href=&quot;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/14/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/14/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Odl built from controller=70/57770/4:99/57699/3 at&lt;br/&gt;
&lt;a href=&quot;https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/45/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/45/&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;Not-returned rpc&lt;br/&gt;
&lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/14/archives/log.html.gz#s1-s2-t1-k2-k11&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/14/archives/log.html.gz#s1-s2-t1-k2-k11&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52149" author="rovarga" created="Thu, 25 May 2017 16:18:29 +0000"  >&lt;p&gt;It seems we are hitting TransmitQueue throttling when sending a purge request after the transaction completes. Since there is a &lt;b&gt;ton&lt;/b&gt; of requests in the queue at that time, we naturally get throttled.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://git.opendaylight.org/gerrit/57822&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/57822&lt;/a&gt; eliminates that delay and adds debug when we get blocked for more than 100msec.&lt;/p&gt;</comment>
                            <comment id="52150" author="vrpolak" created="Fri, 26 May 2017 10:42:00 +0000"  >&lt;p&gt;This still happens &lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt;, also on Carbon post-release codebase &lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;.&lt;br/&gt;
The difference is that this now does not happen after remove-shard replica (&lt;a href=&quot;https://jira.opendaylight.org/browse/CONTROLLER-1692&quot; title=&quot;C: produce-transactions rpc returned NotLeaderException when local prefix shard replica removed&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CONTROLLER-1692&quot;&gt;&lt;del&gt;CONTROLLER-1692&lt;/del&gt;&lt;/a&gt; is seen there instead), only after make-leader-local or become-prefix-leader. &lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/15/archives/log.html.gz#s1-s24-t1-k2-k10&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/15/archives/log.html.gz#s1-s24-t1-k2-k10&lt;/a&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://git.opendaylight.org/gerrit/#/c/57822/1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/#/c/57822/1&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52151" author="rovarga" created="Mon, 29 May 2017 13:58:06 +0000"  >&lt;p&gt;In general this is a manifestation of the client actor dying and being restarted &amp;#8211; the restarted instance has no state, so there is noone to complete the futures the old instance has no knowledge of it.&lt;/p&gt;

&lt;p&gt;The latest failure is an ISE splat during replay of a transaction which is waiting for purge to complete, as the successor txchain is not transitioned to account for the transaction being closed:&lt;/p&gt;

&lt;p&gt;2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | ProxyHistory                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | LocalHistoryIdentifier{client=ClientIdentifier&lt;/p&gt;
{frontend=member-1-frontend-datastore-config, generation=0}, history=2, cookie=0} creating successor transaction proxy for RemoteProxyTransaction{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=successor}&lt;br/&gt;
2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | ProxyHistory                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Proxy org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$Local@3d6366fb open transaction LocalReadWriteProxyTransaction{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open}&lt;br/&gt;
2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | ProxyHistory                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Allocated proxy member-1-datastore-config-fe-0-chn-2-txn-2-0 for transaction member-1-datastore-config-fe-0-chn-2-txn-2-0&lt;br/&gt;
2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | ProxyHistory                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0}
&lt;p&gt;, history=2, cookie=0} created successor transaction proxy LocalReadWriteProxyTransaction&lt;/p&gt;
{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open}&lt;br/&gt;
2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | AbstractProxyTransaction         | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Replaying queued request TransactionPurgeRequest{target=member-1-datastore-config-fe-0-chn-2-txn-2-0, sequence=2, replyTo=Actor[akka://opendaylight-cluster-data/user/$a#1472136728]} to successor LocalReadWriteProxyTransaction{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open}
&lt;p&gt;2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | LocalReadWriteProxyTransaction   | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Applying replayed request TransactionPurgeRequest&lt;/p&gt;
{target=member-1-datastore-config-fe-0-chn-2-txn-2-0, sequence=2, replyTo=Actor[akka://opendaylight-cluster-data/user/$a#1472136728]}
&lt;p&gt;2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | AbstractProxyTransaction         | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Transaction LocalReadWriteProxyTransaction&lt;/p&gt;
{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open} allocated sequence 0&lt;br/&gt;
2017-05-29 03:22:27,394 | DEBUG | lt-dispatcher-22 | AbstractProxyTransaction         | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Transaction proxy LocalReadWriteProxyTransaction{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open}
&lt;p&gt; enqueing request TransactionPurgeRequest&lt;/p&gt;
{target=member-1-datastore-config-fe-0-chn-2-txn-2-0, sequence=0, replyTo=Actor[akka://opendaylight-cluster-data/user/$a#1472136728]}
&lt;p&gt; callback org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction$$Lambda$135/333503310@5ed81725&lt;br/&gt;
2017-05-29 03:22:27,395 | DEBUG | lt-dispatcher-22 | AbstractClientConnection         | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.SNAPSHOT | member-1-frontend-datastore-config: scheduling timeout in 30 seconds&lt;br/&gt;
2017-05-29 03:22:27,395 | DEBUG | lt-dispatcher-22 | TransmitQueue                    | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.SNAPSHOT | Queue ConnectionEntry{request=TransactionPurgeRequest&lt;/p&gt;
{target=member-1-datastore-config-fe-0-chn-2-txn-2-0, sequence=0, replyTo=Actor[akka://opendaylight-cluster-data/user/$a#1472136728]}
&lt;p&gt;, enqueuedTicks=1989409436188} transmitting entry {}&lt;br/&gt;
2017-05-29 03:22:27,398 | DEBUG | lt-dispatcher-22 | ProxyHistory                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | LocalHistoryIdentifier{client=ClientIdentifier&lt;/p&gt;
{frontend=member-1-frontend-datastore-config, generation=0}, history=2, cookie=0} creating successor transaction proxy for RemoteProxyTransaction{identifier=member-1-datastore-config-fe-0-chn-2-txn-3-0, state=successor}&lt;br/&gt;
2017-05-29 03:22:27,399 | DEBUG | lt-dispatcher-22 | AbstractClientHistory            | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Client history org.opendaylight.controller.cluster.databroker.actors.dds.SingleClientHistory@125db7b7 finishing reconnect to ConnectedClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0}
&lt;p&gt;, cookie=0, backend=ShardBackendInfo{actor=Actor&lt;a href=&quot;#-773763026&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-773763026&lt;/a&gt;, sessionId=1, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}}&lt;br/&gt;
2017-05-29 03:22:27,399 | DEBUG | lt-dispatcher-22 | ProxyHistory                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Finished reconnecting proxy history org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$ReconnectCohort@41c76af8&lt;br/&gt;
2017-05-29 03:22:27,399 | DEBUG | lt-dispatcher-22 | AbstractClientHistory            | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Client history org.opendaylight.controller.cluster.databroker.actors.dds.ClientLocalHistory@61fea21f finishing reconnect to ConnectedClientConnection{client=ClientIdentifier&lt;/p&gt;
{frontend=member-1-frontend-datastore-config, generation=0}
&lt;p&gt;, cookie=0, backend=ShardBackendInfo{actor=Actor&lt;a href=&quot;#-773763026&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-773763026&lt;/a&gt;, sessionId=1, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}}&lt;br/&gt;
2017-05-29 03:22:27,399 | DEBUG | lt-dispatcher-22 | AbstractProxyTransaction         | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Finishing reconnect of proxy RemoteProxyTransaction&lt;/p&gt;
{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=successor}
&lt;p&gt;2017-05-29 03:22:27,399 | DEBUG | lt-dispatcher-22 | AbstractProxyTransaction         | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Finishing reconnect of proxy RemoteProxyTransaction&lt;/p&gt;
{identifier=member-1-datastore-config-fe-0-chn-2-txn-3-0, state=successor}

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;...&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;2017-05-29 03:22:27,499 | ERROR | lt-dispatcher-30 | OneForOneStrategy                | 174 - com.typesafe.akka.slf4j - 2.4.17 | Proxy org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$Local@3d6366fb has LocalReadWriteProxyTransaction&lt;/p&gt;
{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open} currently open&lt;br/&gt;
java.lang.IllegalStateException: Proxy org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$Local@3d6366fb has LocalReadWriteProxyTransaction{identifier=member-1-datastore-config-fe-0-chn-2-txn-2-0, state=open}
&lt;p&gt; currently open&lt;br/&gt;
    at com.google.common.base.Preconditions.checkState(Preconditions.java:197)&lt;span class=&quot;error&quot;&gt;&amp;#91;57:com.google.guava:18.0.0&amp;#93;&lt;/span&gt;&lt;br/&gt;
    at org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$Local.doCreateTransactionProxy(ProxyHistory.java:90)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
    at org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory.createTransactionProxy(ProxyHistory.java:367)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
    at org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$ReconnectCohort.replayRequests(ProxyHistory.java:235)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
    at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractClientHistory$1.replayRequests(AbstractClientHistory.java:311)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
    at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractDataStoreClientBehavior.lambda$connectionUp$0(AbstractDataStoreClientBehavior.java:147)&lt;span class=&quot;error&quot;&gt;&amp;#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;br/&gt;
    at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.backendConnectFinished(ClientActorBehavior.java:293)&lt;span class=&quot;error&quot;&gt;&amp;#91;197:org.opendaylight.controller.cds-access-client:1.1.0.SNAPSHOT&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;after which the actor restarts and receives:&lt;/p&gt;

&lt;p&gt;2017-05-29 03:22:27,512 | DEBUG | lt-dispatcher-22 | AbstractClientActor              | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.SNAPSHOT | member-1-frontend-datastore-config: switched from org.opendaylight.controller.cluster.access.client.SavingClientActorBehavior@140e3fe1 to org.opendaylight.controller.cluster.databroker.actors.dds.DistributedDataStoreClientBehavior@329ed459&lt;br/&gt;
2017-05-29 03:22:27,793 | INFO  | ult-dispatcher-4 | ClientActorBehavior              | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.SNAPSHOT | member-1-frontend-datastore-config: Ignoring unknown response Envelope{sessionId=1, txSequence=0, message=TransactionPurgeResponse&lt;/p&gt;
{target=member-1-datastore-config-fe-0-chn-2-txn-2-0, sequence=0}
&lt;p&gt;}&lt;/p&gt;</comment>
                            <comment id="52152" author="rovarga" created="Wed, 31 May 2017 21:19:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://git.opendaylight.org/gerrit/57994&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/57994&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52153" author="vrpolak" created="Mon, 5 Jun 2017 12:01:22 +0000"  >&lt;p&gt;&amp;gt; &lt;a href=&quot;https://git.opendaylight.org/gerrit/57994&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/57994&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;After that got merged, this Bug stopped appearing in most scenarios, except one.&lt;/p&gt;

&lt;p&gt;The scenario uses single writer located at a follower and then the leader is moved to that member. This scenario fails both for module-based and prefix-based shard (tell-based protocol), and in both cases UnreachableMember happens shortly after movement is started.&lt;br/&gt;
Possibly, akka timer tweaking could get rid of this Bug.&lt;/p&gt;

&lt;p&gt;Here is a karaf.log &lt;span class=&quot;error&quot;&gt;&amp;#91;24&amp;#93;&lt;/span&gt; segment from recent module-based RelEng test:&lt;/p&gt;

&lt;p&gt;2017-06-04 14:13:16,733 | INFO  | qtp897580718-78  | ClusterAdminRpcService           | 201 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.1.SNAPSHOT | Moving leader to local node member-2 for shard default, datastoreType Config&lt;br/&gt;
2017-06-04 14:13:19,527 | WARN  | ult-dispatcher-3 | ClusterCoreDaemon                | 174 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node &lt;span class=&quot;error&quot;&gt;&amp;#91;akka.tcp://opendaylight-cluster-data@10.29.14.91:2550&amp;#93;&lt;/span&gt; - Marking node(s) as UNREACHABLE &lt;span class=&quot;error&quot;&gt;&amp;#91;Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.16:2550, status = Up)&amp;#93;&lt;/span&gt;. Node roles &lt;span class=&quot;error&quot;&gt;&amp;#91;member-2&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-06-04 14:13:19,530 | INFO  | lt-dispatcher-24 | ShardManager                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Received UnreachableMember: memberName MemberName&lt;/p&gt;
{name=member-1}, address: akka.tcp://opendaylight-cluster-data@10.29.15.16:2550&lt;br/&gt;
2017-06-04 14:13:19,530 | INFO  | lt-dispatcher-31 | ShardManager                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Received UnreachableMember: memberName MemberName{name=member-1}
&lt;p&gt;, address: akka.tcp://opendaylight-cluster-data@10.29.15.16:2550&lt;br/&gt;
2017-06-04 14:13:19,531 | INFO  | lt-dispatcher-24 | EntityOwnershipShard             | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-2-shard-entity-ownership-operational: onPeerDown: PeerDown &lt;span class=&quot;error&quot;&gt;&amp;#91;memberName=member-1, peerId=member-1-shard-entity-ownership-operational&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-06-04 14:13:20,526 | INFO  | ult-dispatcher-5 | kka://opendaylight-cluster-data) | 174 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node &lt;span class=&quot;error&quot;&gt;&amp;#91;akka.tcp://opendaylight-cluster-data@10.29.14.91:2550&amp;#93;&lt;/span&gt; - Marking node(s) as REACHABLE &lt;span class=&quot;error&quot;&gt;&amp;#91;Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.16:2550, status = Up)&amp;#93;&lt;/span&gt;. Node roles &lt;span class=&quot;error&quot;&gt;&amp;#91;member-2&amp;#93;&lt;/span&gt;&lt;br/&gt;
2017-06-04 14:13:20,527 | INFO  | ult-dispatcher-4 | ShardManager                     | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Received ReachableMember: memberName MemberName&lt;/p&gt;
{name=member-1}
&lt;p&gt;, address: akka.tcp://opendaylight-cluster-data@10.29.15.16:2550&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;24&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/733/odl2_karaf.log.gz&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/733/odl2_karaf.log.gz&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52154" author="vrpolak" created="Mon, 5 Jun 2017 14:01:01 +0000"  >&lt;p&gt;Link to a failing Sandbox longevity job &lt;span class=&quot;error&quot;&gt;&amp;#91;25&amp;#93;&lt;/span&gt; with code &lt;span class=&quot;error&quot;&gt;&amp;#91;26&amp;#93;&lt;/span&gt; not merged yet and verbose logs.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;25&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/2/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/2/&lt;/a&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;26&amp;#93;&lt;/span&gt; &lt;a href=&quot;https://git.opendaylight.org/gerrit/58194&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/58194&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="52155" author="rovarga" created="Mon, 5 Jun 2017 17:55:56 +0000"  >&lt;p&gt;The last patch had a thinko, &lt;a href=&quot;https://git.opendaylight.org/gerrit/58274&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/58274&lt;/a&gt; fixes that.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                            <outwardlinks description="blocks">
                                        <issuelink>
            <issuekey id="26254">CONTROLLER-1700</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is blocked by">
                                        <issuelink>
            <issuekey id="26222">CONTROLLER-1668</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="26243">CONTROLLER-1689</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                            <customfield id="customfield_11400" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10208" key="com.atlassian.jira.plugin.system.customfieldtypes:textfield">
                        <customfieldname>External issue ID</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8403</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10201" key="com.atlassian.jira.plugin.system.customfieldtypes:url">
                        <customfieldname>External issue URL</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[https://bugs.opendaylight.org/show_bug.cgi?id=8403]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10206" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Issue Type</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10300"><![CDATA[Bug]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10204" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>ODL SR Target Milestone</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10347"><![CDATA[Carbon-RC3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10000" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i02rz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>