https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1751 controller <p>Please see attached karaf log. We see this behavior sporadically at different places in the OpenFlow cluster test after we kill an instance and bring it back. Please help identifying the root cause.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1751

Sporadic cluster failure when member is restarted in OF cluster test

Bug Highest Resolved Done Luis Gomez Luis Gomez Thu, 17 Aug 2017 17:47:53 +0000 Tue, 25 Jul 2023 08:24:46 +0000 Thu, 24 May 2018 02:03:38 +0000 clustering 0 9 <p>Attachment karaf_member_restart.txt has been added with description: karaf log for restarted member</p> <p>WARN messages in attached karaf log:</p> <p>2017-08-15 14:29:10,004 | WARN | saction-28-30'}} | DeadlockMonitor | 119 - org.opendaylight.controller.config-manager - 0.6.2.Carbon | ModuleIdentifier</p> {factoryName='runtime-generated-mapping', instanceName='runtime-mapping-singleton'} did not finish after 274970 ms<br/> 2017-08-15 14:29:15,004 | WARN | saction-28-30'}} | DeadlockMonitor | 119 - org.opendaylight.controller.config-manager - 0.6.2.Carbon | ModuleIdentifier{factoryName='runtime-generated-mapping', instanceName='runtime-mapping-singleton'} <p> did not finish after 279970 ms</p> <p>A place in Robot test showing member-2 not syncing within 5 minutes: <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/733/log.html.gz#s1-s1-t34-k2-k2-k8-k1-k2-k1-k1-k6-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/733/log.html.gz#s1-s1-t34-k2-k2-k8-k1-k2-k1-k1-k6-k1</a></p> <p>This is also happening in Netconf <span class="error">&#91;1&#93;</span>, and similar symptom is occasionally seen in Carbon <span class="error">&#91;2&#93;</span>.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-nitrogen/116/log.html.gz#s1-s7-t19-k2-k2-k8-k1-k2-k1-k1-k6-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-nitrogen/116/log.html.gz#s1-s7-t19-k2-k2-k8-k1-k2-k1-k1-k6-k1</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8999" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8999</a></p> <p>Issue also seen in Netvirt csit: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/12/log.html.gz#s1-s2-s2-t7" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/12/log.html.gz#s1-s2-s2-t7</a></p> <p>All the shards timed out and went to Candidate and stayed there repeatedly starting new elections. This means it could not connect to any of the other nodes. Since this intermittent, it seems like an issue in the jenkins environment. </p> <p>I think the CSS timeout is a consequence of this as we block CDS bundle startup while waiting for shards to obtain leaders.</p> <p>It looks like it couldn't contact any other seed node in time so joined itself and made itself leader. Since I assume ODL1 is the first seed node, it can do that. This wouldn't occur with the other 2 nodes.</p> <p>2017-08-15 14:24:55,472 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 171 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.103:2550&#93;</span> - Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.103:2550&#93;</span> is JOINING, roles <span class="error">&#91;member-1&#93;</span><br/> 2017-08-15 14:24:55,480 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 171 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.103:2550&#93;</span> - Leader is moving node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.103:2550&#93;</span> to <span class="error">&#91;Up&#93;</span></p> <p>Tom, this is weird because all we do in these tests is to kill the java process running the karaf container and after that start karaf again. So if there was an issue in the network it should be there all the time not just at the moment of the test.</p> <p>yeah it seems weird. I'm just noting what I saw in the log - ODL1 joined itself which means it didn't connect to another node within a period of time. I don't know why. We don't have the logs from the other nodes. Perhaps something in those logs might yield something, eg maybe they refused to let ODL1 join for some reason. Or maybe the seed-node-timeout need to be increased.</p> <p>Looking at these recent logs:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/118/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/118/odl1_karaf.log.gz</a><br/> <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/118/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/118/odl2_karaf.log.gz</a><br/> <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/118/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/118/odl3_karaf.log.gz</a></p> <p>The restarted member 2 starts listening to 2550 at 14:27:58:</p> <p>2017-08-23 14:27:58,670 | INFO | ult-dispatcher-3 | Remoting | 41 - com.typesafe.akka.slf4j - 2.4.18 | Remoting started; listening on addresses :<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.14.87:2550&#93;</span></p> <p>The other member 1 sends connection attempts every 5 secs:</p> <p>2017-08-23 14:27:53,898 | WARN | lt-dispatcher-48 | ReliableDeliverySupervisor | 41 - com.typesafe.akka.slf4j - 2.4.18 | Association with remote system <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.14.87:2550&#93;</span> has failed, address is now gated for <span class="error">&#91;5000&#93;</span> ms. Reason: [Association failed with <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.14.87:2550&#93;</span>] Caused by: <span class="error">&#91;Connection refused: /10.29.14.87:2550&#93;</span></p> <p>But almost same time member 2 starts listening to 2550 I see these errors in member 1 and no more connection attempts are being sent:</p> <p>2017-08-23 14:27:57,593 | INFO | lt-dispatcher-48 | kka://opendaylight-cluster-data) | 41 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span> - Leader can currently not perform its duties, reachability status: [akka.tcp://opendaylight-cluster-data@10.29.12.157:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.14.87:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (1), akka.tcp://opendaylight-cluster-data@10.29.15.229:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.14.87:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (1)], member status: <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550 Up seen=true, akka.tcp://opendaylight-cluster-data@10.29.14.87:2550 Up seen=false, akka.tcp://opendaylight-cluster-data@10.29.15.229:2550 Up seen=true&#93;</span></p> <p>2017-08-23 14:28:57,606 | INFO | ult-dispatcher-4 | kka://opendaylight-cluster-data) | 41 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span> - Leader can currently not perform its duties, reachability status: [akka.tcp://opendaylight-cluster-data@10.29.12.157:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.14.87:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (1), akka.tcp://opendaylight-cluster-data@10.29.15.229:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.14.87:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (1)], member status: <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550 Up seen=true, akka.tcp://opendaylight-cluster-data@10.29.14.87:2550 Up seen=false, akka.tcp://opendaylight-cluster-data@10.29.15.229:2550 Up seen=true&#93;</span></p> <p>"Leader can currently not perform its duties" is normal when a node becomes unreachable. By duties, it means it can't let any new nodes join the cluster until previously unreachable re-connect/re-join or are "downed" and removed from the cluster..</p> <p>On member-2, it restarted around 2017-08-23 14:27:45:</p> <p>2017-08-23 14:28:22,849 | WARN | lt-dispatcher-23 | JoinSeedNodeProcess | 41 - com.typesafe.akka.slf4j - 2.4.18 | Couldn't join seed nodes after <span class="error">&#91;2&#93;</span> attmpts, will try again. seed-nodes=<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550, akka.tcp://opendaylight-cluster-data@10.29.15.229:2550&#93;</span></p> <p>This continued for 87 attempts until it apparently gave up:</p> <p>2017-08-23 14:45:27,598 | WARN | lt-dispatcher-21 | ReliableDeliverySupervisor | 41 - com.typesafe.akka.slf4j - 2.4.18 | Association with remote system <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span> has failed, address is now gated for <span class="error">&#91;5000&#93;</span> ms. Reason: <span class="error">&#91;Disassociated&#93;</span> </p> <p>2017-08-23 14:45:46,975 | WARN | lt-dispatcher-26 | NettyTransport | 41 - com.typesafe.akka.slf4j - 2.4.18 | Remote connection to null failed with java.net.ConnectException: Connection refused: /10.29.12.157:2550</p> <p>2017-08-23 14:45:46,983 | WARN | ult-dispatcher-4 | ReliableDeliverySupervisor | 41 - com.typesafe.akka.slf4j - 2.4.18 | Association with remote system <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span> has failed, address is now gated for <span class="error">&#91;5000&#93;</span> ms. Reason: [Association failed with <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span>] Caused by: <span class="error">&#91;Connection refused: /10.29.12.157:2550&#93;</span></p> <p>Then it finally connects:</p> <p>2017-08-23 14:45:58,855 | INFO | ult-dispatcher-6 | kka://opendaylight-cluster-data) | 41 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.14.87:2550&#93;</span> - Welcome from <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span><br/> 2017-08-23 14:45:58,864 | INFO | rd-dispatcher-48 | ShardManager | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.6.0 | shard-manager-config: Received MemberUp: memberName: MemberName</p> {name=member-1} <p>, address: akka.tcp://opendaylight-cluster-data@10.29.12.157:2550</p> <p>On member-1, it looks like 10.29.15.229 (ODL3 I assume) was also stopped:</p> <p>2017-08-23 14:37:51,592 | WARN | lt-dispatcher-20 | ClusterCoreDaemon | 41 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span> - Marking node(s) as UNREACHABLE <span class="error">&#91;Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.229:2550, status = Up)&#93;</span>. Node roles <span class="error">&#91;member-1&#93;</span></p> <p>In that case, 2 nodes were down so both nodes have to re-connect or be "downed" before it allows any to join. The latter happened for ODL3:</p> <p>2017-08-23 14:40:18,419 | INFO | lt-dispatcher-22 | kka://opendaylight-cluster-data) | 41 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.157:2550&#93;</span> - Marking unreachable node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.229:2550&#93;</span> as <span class="error">&#91;Down&#93;</span></p> <p>Did the test manually down it?</p> <p>So it looks like the test stops 2 of the nodes. I'm not sure what the expectations are.</p> <p>Hi Tom, you are right, we take down member-3 but that happens 10 mins after (14:37:42) we start member-2 (14:27:45). In our test we give max 5 mins for a member to boot up, join the cluster and load all features. Do you think this is not enough? because when the test pass it only takes 30 secs to go through the above.</p> <p>You can try giving it more time as it did eventually connect although it took over 15 min. Maybe there's something in the CSIT env (it gets busy?) where network communications get bogged down sometimes. member-2 was trying to continuously connect but was getting "Connection refused" for some reason until it finally connected.</p> <p>OK, I think as next step I can try to see if this reproduces outside CI.</p> <p>Preliminary target set at Carbon SR3, pending netvirt CSIT (<a href="https://jira.opendaylight.org/browse/CONTROLLER-1755" title="RaftActor lastApplied index moves backwards" class="issue-link" data-issue-key="CONTROLLER-1755"><del>CONTROLLER-1755</del></a>) may become a blocker for SR2.</p> <p>NetVirt is hitting this exact same issue. NetVirt tests copied the openflowplugin test pattern to take a node down and bring it back. Then wait 5 minutes. What I don't understand is why taking 1 node down out of the three leads to instability? We have three nodes in the cluster. Take 1 down leave other 2 alone. Attempt to bring back the 1 node, wait 5 minutes, that fails and now the cluster is in a bad state causing the further tests to fail.</p> <p>Sometimes there are random failures where a node does not come back properly such as in job <span class="error">&#91;4&#93;</span>. We try to bring ODL1 back into the cluster but it fails to come back within 5 minutes. Then we move to the next tests and they fail. That ODL1 is hitting the below issue. Is there anything we can do to get past that? We can increase the timeout but why is the cluster in a bad shape? I don't think the infra is loaded since everything else is moving along properly - the robot vm is driving the other two nodes. We can also see odl1 restarting but taking it's time in the failing case.</p> <p>2017-08-25 02:02:38,430 | WARN | saction-32-34'}} | DeadlockMonitor | 126 - org.opendaylight.controller.config-manager - 0.6.2.SNAPSHOT | ModuleIdentifier</p> {factoryName='runtime-generated-mapping', instanceName='runtime-mapping-singleton'} <p> did not finish after 284864 ms</p> <p><span class="error">&#91;2&#93;</span> <a href="https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/25/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/25/</a></p> <p><span class="error">&#91;3&#93;</span> <a href="https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/</a></p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/24/log.html.gz#s1-s1-t13-k2-k2-k8" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/24/log.html.gz#s1-s1-t13-k2-k2-k8</a></p> <p>I assume this is intermittent and not every run? Based on analysis of the controller test run logs that Luis provided, it's due to connectivity issues between the nodes for whatever reason. I would suggest manually running the tests outside CI and see if it reproduces as Luis noted. Or manually going thru the steps yourself.</p> <p>Attachment karaf-1.log has been added with description: Member 1 restarting</p> <p>Attachment karaf-2.log has been added with description: Member 2 cannot connect to member 1</p> <p>Attachment karaf-3.log has been added with description: Member 3 cannot connect to member 1</p> <p>I could reproduce the original issue in my local setup (see last 3 attached logs) after trying for 9 times killing/restarting owner instance. From the logs it seems like the restarting instance (member-1) can connect to the other 2 but these cannot connect back to member-1. </p> <p>I could also verify the restarting instance did not opened the akka port:</p> <p>telnet 127.0.0.1 2550<br/> Trying 127.0.0.1...<br/> telnet: Unable to connect to remote host: Connection refused</p> <p>Even when you see in the log:</p> <p>2017-08-27 21:32:00,780 | INFO | ult-dispatcher-2 | Remoting | 179 - com.typesafe.akka.slf4j - 2.4.18 | Remoting started; listening on addresses :<span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span><br/> 2017-08-27 21:32:00,797 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> - Starting up...<br/> 2017-08-27 21:32:00,873 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> - Registered cluster JMX MBean <span class="error">&#91;akka:type=Cluster&#93;</span><br/> 2017-08-27 21:32:00,873 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> - Started up successfully</p> <p>So this seems so far a cluster related issue rather than infra issue.</p> <p>This is the same issue that originated this bug and is collected in this run:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/733/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/733/</a></p> <p>Same behavior: member-2 restarts at 2:25:07 and it is never "akka" reachable from the other members.</p> <p>Tom, Can you look at the additional data Luis added and provide input as to what we can do about this problem? As it stands, this issue will block the Nitrogen release. Thanks!</p> <p>I'll take a look but I've already looked at the other logs and can't ascertain why it can't connect - I don't know if it's an actual network issue or something fluky in akka. I don't anticipate this run being any different. Perhaps enabling akka debug may help. </p> <p>In one run I looked at it did eventually connect but not in time for the test. Luis mentioned he would increase the timeout.</p> <p>In another run, ODL1 couldn't connect to the other nodes so it joined itself. As I mentioned, we can try increasing the seed-node-timeout in the akka.conf for the tests. </p> <p>Either way I don't think this needs to block Nitrogen. There's nothing that could've been introduced in Nitrogen that could've caused this. As I've mentioned, it may be the environment.</p> <p>(In reply to Luis Gomez from comment #21)<br/> &gt; I could reproduce the original issue in my local setup (see last 3 attached<br/> &gt; logs) after trying for 9 times killing/restarting owner instance. From the<br/> &gt; logs it seems like the restarting instance (member-1) can connect to the<br/> &gt; other 2 but these cannot connect back to member-1. <br/> &gt; <br/> &gt; I could also verify the restarting instance did not opened the akka port:<br/> &gt; <br/> &gt; telnet 127.0.0.1 2550<br/> &gt; Trying 127.0.0.1...<br/> &gt; telnet: Unable to connect to remote host: Connection refused<br/> &gt; <br/> &gt; Even when you see in the log:<br/> &gt; <br/> &gt; 2017-08-27 21:32:00,780 | INFO | ult-dispatcher-2 | Remoting <br/> &gt; | 179 - com.typesafe.akka.slf4j - 2.4.18 | Remoting started; listening on<br/> &gt; addresses :<span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span><br/> &gt; 2017-08-27 21:32:00,797 | INFO | ult-dispatcher-2 |<br/> &gt; kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 |<br/> &gt; Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> -<br/> &gt; Starting up...<br/> &gt; 2017-08-27 21:32:00,873 | INFO | ult-dispatcher-2 |<br/> &gt; kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 |<br/> &gt; Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> -<br/> &gt; Registered cluster JMX MBean <span class="error">&#91;akka:type=Cluster&#93;</span><br/> &gt; 2017-08-27 21:32:00,873 | INFO | ult-dispatcher-2 |<br/> &gt; kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 |<br/> &gt; Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> -<br/> &gt; Started up successfully<br/> &gt; <br/> &gt; So this seems so far a cluster related issue rather than infra issue.</p> <p>In this case, member-1 apparently couldn't connect to another node in time so joined itself about 23s later and formed a single-node island:</p> <p>2017-08-27 21:32:23,944 | INFO | ult-dispatcher-5 | kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> - Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> is JOINING, roles <span class="error">&#91;member-1&#93;</span></p> <p>Since member-1 is the first seed node, which has special sematice, it is allowed to do that. </p> <p>Looking at member-2, member-1 did initially connect and try to join but member-2 has to first "down" it and remove the old incarnation from the cluster before it allows the new incarnation to join, which it did:</p> <p>17-08-27 21:32:01,291 | INFO | ult-dispatcher-4 | kka://openaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.102:2550&#93;</span> - New incarnation of existing member <span class="error">&#91;Member(address = akka.tcp://opendaylight-cluster-data@192.168.0.101:2550, status = Up)&#93;</span> is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.<br/> 2017-08-27 21:32:01,292 | INFO | ult-dispatcher-4 | kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.102:2550&#93;</span> - Marking unreachable node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> as <span class="error">&#91;Down&#93;</span><br/> 2017-08-27 21:32:02,376 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 179 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.102:2550&#93;</span> </p> <ul class="alternate" type="square"> <li>Leader is removing unreachable node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span></li> </ul> <p>There's no other log activity for 192.168.0.101 until about 5 min later when akka reports "Connection refused" after member-1 was stopped again. </p> <p>2017-08-27 21:36:49,429 | INFO | lt-dispatcher-22 | Remoting | 179 - com.typesafe.akka.slf4j - 2.4.18 | Quarantined address <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.0.101:2550&#93;</span> is still unreachable or has not been restarted. Keeping it quarantined.<br/> 2017-08-27 21:36:49,815 | WARN | ult-dispatcher-4 | NettyTransport | 179 - com.typesafe.akka.slf4j - 2.4.18 | Remote connection to null failed with java.net.ConnectException: Connection refused: /192.168.0.101:2550</p> <p>From my understanding, member-1 should retry connection every second up to the seed-node-timeout (which we set to 12s by default) and member-2 should eventually let it join if it is able to connect. For some reason it wasn't able to reconnect/join within that period of time. </p> <p>Gary Wu saw this behavior a while back and opened an issue with akka <a href="https://github.com/akka/akka/issues/18757/" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/issues/18757/</a>. It turned out the default akka seed-node-timeout of 5s was sometimes too low in his docker environment and increasing to 12s alleviated it and we made that our default. </p> <p>As Gary mentioned in his environment "dnode0's startup creates a heavy load, so it's possible that dnode0 might be affecting the responsiveness of dnode1 and dnode2.". It may be that even 12s is sometimes too low in the jenkins test environment. I would suggest increasing it even more for the test, try 30s or even 60s.</p> <p>(In reply to Luis Gomez from comment #22)<br/> &gt; This is the same issue that originated this bug and is collected in this run:<br/> &gt; <br/> &gt; <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-</a><br/> &gt; clustering-only-carbon/733/<br/> &gt; <br/> &gt; Same behavior: member-2 restarts at 2:25:07 and it is never "akka" reachable<br/> &gt; from the other members.</p> <p>In this case member-2 is not the first seed node so it has to join with another and cannot join itself. We see that it tried to connect/join with another node 23 times over 5 min until apparently robot shut it down:</p> <p>2017-08-14 02:30:08,093 | WARN | ult-dispatcher-5 | JoinSeedNodeProcess | 171 - com.typesafe.akka.slf4j - 2.4.18 | Couldn't join seed nodes after <span class="error">&#91;23&#93;</span> attmpts, will try again. seed-nodes=<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.94:2550, akka.tcp://opendaylight-cluster-data@10.29.15.142:2550&#93;</span></p> <p>On member-1, during that time we see it try to connect to member-2 every 5s but fails with "Connection refused":</p> <p>2017-08-14 02:25:15,880 | WARN | lt-dispatcher-23 | ReliableDeliverySupervisor | 171 - com.typesafe.akka.slf4j - 2.4.18 | Association with remote system <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.200:2550&#93;</span> has failed, address is now gated for <span class="error">&#91;5000&#93;</span> ms. Reason: [Association failed with <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.200:2550&#93;</span>] Caused by: <span class="error">&#91;Connection refused: /10.29.12.200:2550&#93;</span></p> <p>So it appears neither side was able to connect. It seems to me the environment was overloaded unless there's some bug in akka which is always possible but just doesn't seem likely as establishing TCP connections is pretty basic and "Connection refused" emanates from the lower-level TCP/socket layer. In a previous run we saw a node take about 17 min before it finally connected as I noted earlier.</p> <p>Yes, I believe it is a TCP socket problem as when I reproduced locally the TCP port 2550 never came up for the restarting instance. It could be AKKA issue as I guess this library is ultimately opening/using the port. Tom, is there any DEBUG to troubleshoot this more?</p> <p>(In reply to Luis Gomez from comment #27)<br/> &gt; Yes, I believe it is a TCP socket problem as when I reproduced locally the<br/> &gt; TCP port 2550 never came up for the restarting instance. It could be AKKA<br/> &gt; issue as I guess this library is ultimately opening/using the port. Tom, is<br/> &gt; there any DEBUG to troubleshoot this more?</p> <p>Akka reports it successfully opened the port and I doubt there could be a bug there - that's pretty basic and I've never seen an issue with that. Maybe it could be the networking layer in the VM environment is overloaded or threads/timers get delayed...</p> <p>The fact that akka keeps trying to periodically connect over minutes seems less likely an issue with akka, at least to me. I could see maybe a timing bug or something causing one attempt to fail but many ...?</p> <p>You can set akka.loglevel=DEBUG in the akka.conf and also<br/> <a href="http://doc.akka.io/docs/akka/2.4.0/scala/logging.html#Auxiliary_remote_logging_options" class="external-link" target="_blank" rel="nofollow noopener">http://doc.akka.io/docs/akka/2.4.0/scala/logging.html#Auxiliary_remote_logging_options</a> as was instructed in <a href="https://github.com/akka/akka/issues/18757/" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/issues/18757/</a>.</p> <p>OK, I tried to reproduce this one in my local setup and this time it did not fail for the many times I tried. So this puts the issue back to the CI where it is happening sporadically but still after few times. Because of this I think we can downgrade this to CRITICAL for now.</p> <p>FYI this issue is also present in Boron so all branches:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/411/log.html.gz#s1-s3" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/411/log.html.gz#s1-s3</a></p> <p>Unfortunately it is difficult to reproduce outside ODL CI, so we cannot discard test env problem.</p> <p>Some time ago my favorite explanation was that there is a subtle bug in Blueprint definitions somewhere in clustering features, which needs a specific timing to result in this Bug.</p> <p>If that is the case (and we still have trouble reproducing the failure reliably in CSIT), perhaps it would be a good idea to fix other Bugs, hoping this would get fix together with the other bug.</p> <p>SingleFeatureTest should be verifying blueprint, but some features are blacklisted, some of them related to clustering. Specifically, Odlparent 2.0.4 skips <span class="error">&#91;8&#93;</span>: odl-mdsal-broker-local, odl-mdsal-clustering-commons, odl-mdsal-distributed-datastore, and odl-mdsal-remoterpc-connector.</p> <p>(This list is strangely similar to features affected by <span class="error">&#91;9&#93;</span>, no idea if there is a causal relation.)</p> <p>There is already <a href="https://jira.opendaylight.org/browse/CONTROLLER-1584" title="Fix broken controller features failing the new extended SingleFeatureTest incl. TestBundleDiag due to IllegalStateException: ./configuration/initial/akka.conf is missing" class="issue-link" data-issue-key="CONTROLLER-1584"><del>CONTROLLER-1584</del></a> opened for not skipping those features, so perhaps we should attempt to tackle <a href="https://jira.opendaylight.org/browse/CONTROLLER-1584" title="Fix broken controller features failing the new extended SingleFeatureTest incl. TestBundleDiag due to IllegalStateException: ./configuration/initial/akka.conf is missing" class="issue-link" data-issue-key="CONTROLLER-1584"><del>CONTROLLER-1584</del></a> (again) before getting back to this? I recall the SFT failures on those features were quite reliable.</p> <p><span class="error">&#91;8&#93;</span> <a href="https://github.com/opendaylight/odlparent/blob/v2.0.4/features-test/src/main/java/org/opendaylight/odlparent/featuretest/SingleFeatureTest.java#L404-L409" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/odlparent/blob/v2.0.4/features-test/src/main/java/org/opendaylight/odlparent/featuretest/SingleFeatureTest.java#L404-L409</a><br/> <span class="error">&#91;9&#93;</span> <a href="https://git.opendaylight.org/gerrit/63329" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/63329</a></p> <p>This is an issue with akka sometimes not being able to connect to other nodes or, at least, it takes longer than expected. From all the logs I've looked at it appears to be at the TCP level. It is not related to blueprint wiring or anything in ODL code. As Luis mentioned, it is also present in Boron which indicates it is not tied to any akka version or the karaf 4 upgrade. From everything I can tell, it seems related to the CSIT env, ie intermittent load. I would suggest adjusting time outs to take that into account.</p> <p>OK, I am trying to reproduce the issue in sandbox so I can apply some of the suggested cluster configurations but so far I am also having trouble to reproduce there <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>&gt; akka sometimes not being able to connect to other nodes or,<br/> &gt; at least, it takes longer than expected.</p> <p>Is there a logging setting which would enable us to compare such connect timings (without spamming karaf.log too much)?<br/> Possibly without additional changes such as <span class="error">&#91;10&#93;</span>?</p> <p><span class="error">&#91;10&#93;</span> <a href="https://git.opendaylight.org/gerrit/60727" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/60727</a></p> <p>After adding some network connection printouts (netstat) in the test, I found a pattern for the failure: Every time a restarting member fails to join I can see there are 2 connection (inbound and outbound) established between the restarting member and the same remote member (see 2 first lines below):</p> <p>tcp6 0 0 10.29.15.190:2550 10.29.15.107:58300 ESTABLISHED 12020/java <br/> tcp6 0 0 10.29.15.190:44008 10.29.15.107:2550 ESTABLISHED 12020/java</p> <p>tcp6 0 0 10.29.15.190:47210 10.29.15.113:2550 ESTABLISHED 12020/java </p> <p>This is weird as normally 1 connection (inbound or outbound) is enough to setup the akka association. I am still investigating and adding more printouts to understand how this double connection gets generated.</p> <p>Hi Tom,</p> <p>From the netstat printouts for a recent failure:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-oxygen/19/log.html.gz#s1-s6" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-oxygen/19/log.html.gz#s1-s6</a></p> <p>It seems clear there is a relation between duplicated akka connections and the issue:</p> <p>Failing restarting instance (member-1):<br/> tcp6 0 0 10.29.15.220:2550 :::* LISTEN 18273/java <br/> tcp6 0 0 10.29.15.220:46724 10.29.14.239:2550 ESTABLISHED 18273/java <br/> tcp6 0 0 10.29.15.220:2550 10.29.13.230:33632 ESTABLISHED 18273/java <br/> tcp6 0 0 10.29.15.220:2550 10.29.14.239:49522 ESTABLISHED 18273/java <br/> tcp6 0 0 10.29.15.220:57166 10.29.13.230:2550 ESTABLISHED 18273/java </p> <p>Other instance (member-2):<br/> tcp6 0 0 10.29.13.230:2550 :::* LISTEN 16951/java <br/> tcp6 0 0 10.29.13.230:2550 10.29.15.220:57166 ESTABLISHED 16951/java <br/> tcp6 0 428 10.29.13.230:46226 10.29.14.239:2550 ESTABLISHED 16951/java <br/> tcp6 0 0 10.29.13.230:33632 10.29.15.220:2550 ESTABLISHED 16951/java </p> <p>Other instance (member-3):<br/> tcp6 0 0 10.29.14.239:2550 :::* LISTEN 16693/java <br/> tcp6 0 450 10.29.14.239:2550 10.29.13.230:46226 ESTABLISHED 16693/java <br/> tcp6 0 0 10.29.14.239:2550 10.29.15.220:46724 ESTABLISHED 16693/java <br/> tcp6 0 0 10.29.14.239:49522 10.29.15.220:2550 ESTABLISHED 16693/java</p> <p>Also looking at the logs it seems member-2 and member-3 try to connect the restarting instance at the same time the tcp socket gets available in member-1:</p> <p>2017-10-17 09:45:26,352 | INFO | ult-dispatcher-2 | Remoting | 41 - com.typesafe.akka.slf4j - 2.4.18 | Remoting started; listening on addresses :<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.220:2550&#93;</span></p> <p>So is there any possibility of a race condition in the restarting instance so that it accepts both connections from member-2 &amp; member-3 and at the same time creates different connections to member-2 and member-3 to join the cluster? in such scenario is it possible member-1 sends and expects messages from one connection while member-2 and member-3 sends and expects messages from other connection?</p> <p>I don't know. I haven't studied akka code to that detail. I would suggest opening a case with akka to answer your questions.</p> <p>Well, it is not yet clear whether the double connection is the origin or a consequence of the issue. I think next step will be to enable some akka debug in CSIT. In my local setup I changed the akka.conf as instructed:</p> <p> loglevel = "DEBUG"<br/> log-config-on-start = on<br/> actor {<br/> debug </p> { autoreceive = on lifecycle = on unhandled = on fsm = on } <p> }</p> <p>But I do not see any DEBUG message in karaf.log if this is the place where akka log is dumped, maybe I need to enable some DEBUG in the karaf logger itself, any idea of what is missing?</p> <p>&gt; I need to enable some DEBUG in the karaf logger itself</p> <p>Yes. See <span class="error">&#91;12&#93;</span>.</p> <p><span class="error">&#91;12&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/58933/14/jjb/integration/integration-configure-clustering.sh@95" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/58933/14/jjb/integration/integration-configure-clustering.sh@95</a></p> <p>Thanks Vratko, it works but unfortunately the akka messaging debug is very chatty so I will start with some other debug and see what happens.</p> <p>After adding the DEBUGs, I can confirm issue happens when the restarting member and the cluster leader initiates AKKA connection at the same time. The restarting member initiates the connection as part of the boot while the cluster leader is trying every 5 secs. Can we do something in our code to handle this conflict? if not I guess I will open an issue in akka. Tom, which version of akka are we using in ODL?</p> <p>Node 3 (restarting member) log in <span class="error">&#91;1&#93;</span>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2017-10-31 22:47:55,553 | DEBUG | lt-dispatcher-25 | AkkaProtocolManager | 41 - com.typesafe.akka.slf4j - 2.4.18 | now supervising Actor[akka:<span class="code-comment">//opendaylight-cluster-data/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fopendaylight-cluster-data%4010.29.14.65%3A2550-1#175430158] </span>2017-10-31 22:47:55,554 | DEBUG | lt-dispatcher-25 | AkkaProtocolManager | 41 - com.typesafe.akka.slf4j - 2.4.18 | now supervising Actor[akka:<span class="code-comment">//opendaylight-cluster-data/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fopendaylight-cluster-data%4010.29.13.208%3A2550-2#-2037739644] </span>2017-10-31 22:47:55,590 | INFO | rint Extender: 3 | AbstractDataStore | 217 - org.opendaylight.controller.sal-distributed-datastore - 1.6.1.SNAPSHOT | Creating ShardManager : shardmanager-config 2017-10-31 22:47:55,596 | DEBUG | ult-dispatcher-2 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | started (akka.remote.transport.ProtocolStateActor@35fbf1c1) 2017-10-31 22:47:55,597 | DEBUG | ult-dispatcher-2 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | started (akka.remote.transport.ProtocolStateActor@519a5d1b) 2017-10-31 22:47:55,597 | DEBUG | ult-dispatcher-2 | AkkaProtocolManager | 41 - com.typesafe.akka.slf4j - 2.4.18 | now supervising Actor[akka:<span class="code-comment">//opendaylight-cluster-data/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fopendaylight-cluster-data%4010.29.14.65%3A59194-3#1453714731] </span>2017-10-31 22:47:55,597 | DEBUG | ult-dispatcher-2 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | started (akka.remote.transport.ProtocolStateActor@76642d3b) 2017-10-31 22:47:55,601 | DEBUG | lt-dispatcher-25 | AkkaProtocolManager | 41 - com.typesafe.akka.slf4j - 2.4.18 | now supervising Actor[akka:<span class="code-comment">//opendaylight-cluster-data/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fopendaylight-cluster-data%4010.29.13.208%3A42796-4#562821358] </span>2017-10-31 22:47:55,601 | DEBUG | lt-dispatcher-25 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | started (akka.remote.transport.ProtocolStateActor@79b35ebe) 2017-10-31 22:47:55,675 | DEBUG | lt-dispatcher-25 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | stopped </pre> </div></div> <p>Node 2 (cluster leader) log in <span class="error">&#91;2&#93;</span>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2017-10-31 22:47:55,587 | DEBUG | ult-dispatcher-6 | AkkaProtocolManager | 41 - com.typesafe.akka.slf4j - 2.4.18 | now supervising Actor[akka:<span class="code-comment">//opendaylight-cluster-data/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fopendaylight-cluster-data%4010.29.15.87%3A2550-6#-1931382701] </span>2017-10-31 22:47:55,587 | DEBUG | ult-dispatcher-6 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | started (akka.remote.transport.ProtocolStateActor@6a4fb40c) 2017-10-31 22:47:55,595 | DEBUG | ult-dispatcher-5 | AkkaProtocolManager | 41 - com.typesafe.akka.slf4j - 2.4.18 | now supervising Actor[akka:<span class="code-comment">//opendaylight-cluster-data/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fopendaylight-cluster-data%4010.29.15.87%3A53374-7#691018071] </span>2017-10-31 22:47:55,596 | DEBUG | ult-dispatcher-5 | ProtocolStateActor | 41 - com.typesafe.akka.slf4j - 2.4.18 | started (akka.remote.transport.ProtocolStateActor@5146512) </pre> </div></div> <p><span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/208/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/208/odl3_karaf.log.gz</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/208/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-nitrogen/208/odl2_karaf.log.gz</a></p> <p>FYI I opened this issue in AKKA:<br/> <a href="https://github.com/akka/akka/issues/23959" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/issues/23959</a></p> <p>Since the bug is unassigned I'm currently assigning it to you.</p> <p>Please assign to the relevant person. </p> <p>Oxygen SR1 and Fluorine are using akka_2.12-2.5.11, scala 2.12.5.</p> <p>This was identified as an AKKA issue, I will recheck on mentioned branches.</p> <p>This seems like the issue is fixed after last AKKA update:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-bulkomatic-only-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-bulkomatic-only-oxygen/</a></p> Duplicate CONTROLLER-1754 Development External issue ID 9006 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02si7: