https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/INFRAUTILS-38 infrautils <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> is seeing this on particular environment (but not everywhere) :</p> <p>We initially had some confusion because 2 separate logs were submitted. This bug with the new log has nothing to do with <a href="https://jira.opendaylight.org/browse/INFRAUTILS-36" title="showSvcStatus: Status retrieval JMX Operation failed for node 127.0.0.1 (NoSuchElementException) at acquireServiceStatusDetailed(DiagStatusServiceMBeanImpl.java:119)" class="issue-link" data-issue-key="INFRAUTILS-36"><del>INFRAUTILS-36</del></a> (part of <a href="https://jira.opendaylight.org/browse/INFRAUTILS-31" title="diagstatus including cause of failures" class="issue-link" data-issue-key="INFRAUTILS-31"><del>INFRAUTILS-31</del></a>) nor is it related to the technical enhancement proposed in <a href="https://jira.opendaylight.org/browse/INFRAUTILS-37" title="diagstatus CLI should not to use JMX to obtain information from local node to avoid ConnectException on 127.0.0.1" class="issue-link" data-issue-key="INFRAUTILS-37"><del>INFRAUTILS-37</del></a>.</p> <p>I suspect this could perhaps be related to that environment having a particular networking config.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-05-02T14:01:50,905 | ERROR | pipe-showSvcStatus | DiagStatusCommand | 233 - org.opendaylight.infrautils.diagstatus-shell - 1.4.0.SNAPSHOT | Exception while reaching Host 127.0.0.1 java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused (Connection refused)] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) [?:?] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) [?:?] at org.opendaylight.infrautils.diagstatus.MBeanUtils.invokeRemoteJMXOperation(MBeanUtils.java:160) [231:org.opendaylight.infrautils.diagstatus-api:1.4.0.SNAPSHOT] at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.getRemoteStatusSummary(DiagStatusCommand.java:74) [233:org.opendaylight.infrautils.diagstatus-shell:1.4.0.SNAPSHOT] at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.execute(DiagStatusCommand.java:47) [233:org.opendaylight.infrautils.diagstatus-shell:1.4.0.SNAPSHOT] at org.apache.karaf.shell.commands.basic.AbstractCommand.execute(AbstractCommand.java:34) [49:org.apache.karaf.shell.core:4.1.5] at Proxybdb3a43e_422e_473e_8dcb_4c63b2f0ef41.execute(Unknown Source) [?:?] at org.apache.karaf.shell.compat.CommandTracker.execute(CommandTracker.java:120) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:68) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:86) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:571) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:497) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:386) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Pipe.doCall(Pipe.java:417) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:229) [49:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:59) [49:org.apache.karaf.shell.core:4.1.5] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?] Caused by: javax.naming.ServiceUnavailableException at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136) ~[?:?] at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[?:?] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:?] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:?] ... 19 more Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused (Connection refused) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[?:?] at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338) ~[?:?] at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112) ~[?:?] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[?:?] at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[?:?] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:?] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:?] ... 19 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:?] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?] at java.net.Socket.connect(Socket.java:589) ~[?:?] at java.net.Socket.connect(Socket.java:538) ~[?:?] at java.net.Socket.&lt;init&gt;(Socket.java:434) ~[?:?] at java.net.Socket.&lt;init&gt;(Socket.java:211) ~[?:?] at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[?:?] at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[?:?] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[?:?] at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338) ~[?:?] at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112) ~[?:?] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[?:?] at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[?:?] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:?] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:?] ... 19 more</pre> </div></div> INFRAUTILS-38

showSvcStatus: Status retrieval JMX Operation failed for node 127.0.0.1 due to rmi.ConnectException: Connection refused to host: 127.0.0.1

Bug Medium Verified Done Faseela K Michael Vorburger Wed, 2 May 2018 16:19:23 +0000 Tue, 22 May 2018 18:07:31 +0000 Tue, 22 May 2018 17:59:37 +0000 Fluorine diagstatus 0 4 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> do you want to assign this to yourself, add some logging, and follow up with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, or were you hoping I would? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : Do you hit the issue in a script, or when you run this manually? Self-address is derived from an akka based configuration, wondering whether your script is executing showSvcStatus CLI too early, before this is exposed. And I am wondering whether DNS is not configured properly on your system, that it results in the error.</p> <p> </p> <p>Else, if you are running this manually giving enough time, could you please give me the output of :</p> <p><a href="http://%3Ccic-ip/" class="external-link" target="_blank" rel="nofollow noopener">http://&lt;ODL-IP</a><ins>&gt;:8181/jolokia/read/akka:type=Cluster</ins></p> <p> </p> <p>ok, so I have learned more about this one. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/star_red.png" height="16" width="16" align="absmiddle" alt="" border="0"/><a href="https://jira.opendaylight.org/browse/INFRAUTILS-37" title="diagstatus CLI should not to use JMX to obtain information from local node to avoid ConnectException on 127.0.0.1" class="issue-link" data-issue-key="INFRAUTILS-37"><del>INFRAUTILS-37</del></a> and this are not the same root cause.</p> <p>TL;DR the /etc/hosts file matters, and stuff internally with odl/jmx/etc will choke if the $(hostname) is not there.</p> <p>This issue is something that used to show up for me in the karaf console log and I could never figure out what<br/> was wrong, but did not try very hard as it didn't seem to bother anything I was doing. I don't have the old error<br/> I'd see on the console, but it was jmx related and errors connecting to 127.0.0.1</p> <p>So, the fix for me is to have a proper /etc/hosts file. It was this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 127.0.0.1 localhost localhost.localdomain jamo-rc701.rdocloud ::1 localhost localhost.localdomain jamo-rc701.rdocloud </pre> </div></div> <p>but, after changing it to this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 127.0.0.1 localhost localhost.localdomain ::1 localhost localhost.localdomain </pre> </div></div> <p>it works.</p> <p>This begs the question of what is going on under the hood, because the error says that it's a connection<br/> refused to 127.0.0.1, because I am able to open a connection to 127.0.0.1 on port 6886 which is the<br/> relevant port here. see below with 6886 working and 6887 being refused like we'd expect:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ nc 127.0.0.1 6886 ^C ]$ nc 127.0.0.1 6887 Ncat: Connection refused. </pre> </div></div> <p>I can't think of how this matters yet, but we can chalk this up to a user configuration issue, but<br/> that was pretty lame to hunt down.</p> <p>Jamo, JMX RMI uses InetAddress.getHostByName() inside, and if /etc/hosts does not have the hostname, it does not work.</p> <p>But, what is still not clear to me is, why we go via JMX RMI for 127.0.0.1 which is local address.</p> <p>Output of this will help me to analyse that further : <a href="http://%3Ccic-ip/" class="external-link" target="_blank" rel="nofollow noopener">http://&lt;ODL-IP</a><ins>&gt;:8181/jolokia/read/akka:type=Cluster</ins></p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ curl -u <span class="code-quote">"admin:admin"</span> http:<span class="code-comment">//localhost:8181/jolokia/read/akka:type=Cluster | python -m json.tool </span> % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 651 0 651 0 0 83482 0 --:--:-- --:--:-- --:--:-- 93000 { <span class="code-quote">"request"</span>: { <span class="code-quote">"mbean"</span>: <span class="code-quote">"akka:type=Cluster"</span>, <span class="code-quote">"type"</span>: <span class="code-quote">"read"</span> }, <span class="code-quote">"status"</span>: 200, <span class="code-quote">"timestamp"</span>: 1525358717, <span class="code-quote">"value"</span>: { <span class="code-quote">"Available"</span>: <span class="code-keyword">true</span>, <span class="code-quote">"ClusterStatus"</span>: <span class="code-quote">"{\n \"</span>members\<span class="code-quote">": [\n {\n \"</span>address\<span class="code-quote">": \"</span>akka.tcp:<span class="code-comment">//opendaylight-cluster-data@127.0.0.1:2550\<span class="code-quote">",\n \"</span>roles\<span class="code-quote">": [\n \"</span>dc-<span class="code-keyword">default</span>\<span class="code-quote">",\n \"</span>member-1\<span class="code-quote">"\n ],\n \"</span>status\<span class="code-quote">": \"</span>Up\<span class="code-quote">"\n }\n ],\n \"</span>self-address\<span class="code-quote">": \"</span>akka.tcp://opendaylight-cluster-data@127.0.0.1:2550\<span class="code-quote">",\n \"</span>unreachable\<span class="code-quote">": []\n}\n"</span>, </span> <span class="code-quote">"Leader"</span>: <span class="code-quote">"akka.tcp:<span class="code-comment">//opendaylight-cluster-data@127.0.0.1:2550"</span>, </span> <span class="code-quote">"MemberStatus"</span>: <span class="code-quote">"Up"</span>, <span class="code-quote">"Members"</span>: <span class="code-quote">"akka.tcp:<span class="code-comment">//opendaylight-cluster-data@127.0.0.1:2550"</span>, </span> <span class="code-quote">"Singleton"</span>: <span class="code-keyword">true</span>, <span class="code-quote">"Unreachable"</span>: "" } } </pre> </div></div> <p>So, what are you looking for in that output?</p> <p>in this output, if self-address matches the ip in the cluster-members, we do localStatusSummary(). I was wondering why the exception you have given shows that RemoteStatusSummary() failed for 127.0.0.1.</p> <p> </p> <p>Could you please confirm if you are using a build which has this fix?</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/70842/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70842/</a></p> <p> </p> <p>I think it was, but it was from 4/17, which is when that patch was merged. Anyway,<br/> below is the same request on a distro built today. looks like the same output:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ curl -u <span class="code-quote">"admin:admin"</span> http:<span class="code-comment">//localhost:8181/jolokia/read/akka:type=Cluster | python -m json.tool </span>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                  Dload  Upload   Total   Spent    Left  Speed 100   651    0   651    0     0   5482      0 --:--:-- --:--:-- --:--:--  5516 {     <span class="code-quote">"request"</span>: {         <span class="code-quote">"mbean"</span>: <span class="code-quote">"akka:type=Cluster"</span>,         <span class="code-quote">"type"</span>: <span class="code-quote">"read"</span>     },     <span class="code-quote">"status"</span>: 200,     <span class="code-quote">"timestamp"</span>: 1525458172,     <span class="code-quote">"value"</span>: {         <span class="code-quote">"Available"</span>: <span class="code-keyword">true</span>,         <span class="code-quote">"ClusterStatus"</span>: <span class="code-quote">"\{\n  \"</span>members\<span class="code-quote">": [\n    {\n      \"</span>address\<span class="code-quote">": \"</span>akka.tcp:<span class="code-comment">//opendaylight-cluster-data@127.0.0.1:2550\<span class="code-quote">",\n      \"</span>roles\<span class="code-quote">": [\n        \"</span>dc-<span class="code-keyword">default</span>\<span class="code-quote">",\n        \"</span>member-1\<span class="code-quote">"\n      ],\n      \"</span>status\<span class="code-quote">": \"</span>Up\<span class="code-quote">"\n    }\n  ],\n  \"</span>self-address\<span class="code-quote">": \"</span>akka.tcp://opendaylight-cluster-data@127.0.0.1:2550\<span class="code-quote">",\n  \"</span>unreachable\<span class="code-quote">": []\n}\n"</span>, </span>        <span class="code-quote">"Leader"</span>: <span class="code-quote">"akka.tcp:<span class="code-comment">//opendaylight-cluster-data@127.0.0.1:2550"</span>, </span>        <span class="code-quote">"MemberStatus"</span>: <span class="code-quote">"Up"</span>,         <span class="code-quote">"Members"</span>: <span class="code-quote">"akka.tcp:<span class="code-comment">//opendaylight-cluster-data@127.0.0.1:2550"</span>, </span>        <span class="code-quote">"Singleton"</span>: <span class="code-keyword">true</span>,         <span class="code-quote">"Unreachable"</span>: ""     } } </pre> </div></div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, anything more I can do here? I don't think this is probably a high priority/severity bug, but just making sure the</p> <p>ball is not in my court <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> only thing is whether you had the patch I mentioned above while testing? If it is there the error would not have come even if your /etc/hosts is not proper for single node . Anyways should we close the bug saying it is a missing config?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, no I confirmed that I had the patch and gave the output in my last comment. Do you want me to try again?</p> <p>Yes, I made <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=sathwiksai" class="user-hover" rel="sathwiksai">sathwiksai</a> to try this out locally, and we always see that for 127.0.0.1 it fetches the local status and never go via RMI. In fact that was the main reason why the above patch was pushed. Yesterday, even Vishal tried this out, and we did not see the error. Probably we should try removing hostname from /etc/hosts and see if it works? Will try that out and update.</p> <p>I tried this locally, removing the hostname in etc/hosts and it worked. I have seen the same before this patch (<a href="https://git.opendaylight.org/gerrit/#/c/70842/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70842/</a>) pushed. that's the main reason the patch was pushed.</p> <p>just for kicks I double checked with and without a good /etc/hosts file and I don't see the high level symptoms with those<br/> exceptions. closing this as resolved and verified.</p> Development Rank 0|i03e53: