https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/INFRAUTILS-54 infrautils <p>Genius CSIT 3 node is failing on master in ShowSVCStatus. </p> <p>Due to this, <a href="https://git.opendaylight.org/gerrit/#/c/76345/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76345/</a> changes it is failing which is regarding Diagstatus and Cluster.</p> <p>Please Find Logs here, <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/434/nidhigenius-csit-3node-gate-all-neon/10/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/434/nidhigenius-csit-3node-gate-all-neon/10/</a></p> INFRAUTILS-54

Genius CSIT 3 Node fails following infrautils changes in diagstatus

Bug Highest Resolved Done Michael Vorburger nidhi adhvaryu Thu, 27 Sep 2018 05:36:01 +0000 Wed, 3 Oct 2018 05:20:41 +0000 Tue, 2 Oct 2018 17:05:15 +0000 Neon Neon 0 4 <p>I'll look more into and sort this out (fix it) on Monday, I promise! <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/tongue.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Thank you Michael. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=enidadh" class="user-hover" rel="enidadh">enidadh</a> I just looked more into it now, but I don't see the actual error yet... the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/434/nidhigenius-csit-3node-gate-all-neon/10/" class="external-link" target="_blank" rel="nofollow noopener">odl_1-3/odl1-3_karaf.log.gz</a> do not contain any ERROR (which seems related to this) and the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/434/nidhigenius-csit-3node-gate-all-neon/10/robot-plugin/log.html.gz" class="external-link" target="_blank" rel="nofollow noopener">Robot Log</a> shows:</p> <p><em>"Keyword 'ClusterManagement.Check Status Of Services Is OPERATIONAL' failed after retrying for 1 minute. The last error was: '[?1l&gt;[?2004l[?1h=[?2004hopendaylight-user@root&gt;' does not contain 'ACTIVE'"</em></p> <p>but does not show what the output of the "showSvcStatus -n 10.30.170.65" it seems to do actually is - or I'm just not seeing it? That output should contain the failure, hopefully with exception, which I need to understand what broke here... can someone help find that?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> perhaps CSIT output could be improved?</p> <p>Just FTR, a freshly built local genius without clustering is OK, so this is something particular in a clustered CSIT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>opendaylight-user@root&gt;showSvcStatus Timestamp: Mon Oct 01 12:29:52 CEST 2018 Node IP Address: 127.0.0.1 System is operational: true System ready state: ACTIVE OPENFLOW : OPERATIONAL (OF::PORTS:: 6653 and 6633 are up.) IFM : OPERATIONAL ITM : OPERATIONAL OVSDB : OPERATIONAL (OVSDB initialization complete) DATASTORE : OPERATIONAL </pre> </div></div> <p>I need to see the output like above from the clustered CSIT to understand what's failing...</p> <p>that's the exact problem. there is no output coming back. It's just coming back with the<br/> karaf prompt.</p> <p>maybe you can try <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a>'s cluster tool to pop up a local cluster to reproduce it?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> csit does log the showsvcstatus output, may be it is not priniting anything. Does the karaf log help?</p> <p>&gt; maybe you can try <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a>'s cluster tool to pop up a local cluster to reproduce it?</p> <p><a href="https://github.com/jhershberg/netvirt-ha-docker" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jhershberg/netvirt-ha-docker</a> sounds useful here - lemme see if I can get this up and running</p> <p>I've just tried it! The basics seem to work just fine.. here are my exact steps (using netvirt instead of genius, but is most probably the same):</p> <p><tt>sudo setenforce 0</tt><br/> <tt>git clone git@github.com:jhershberg/netvirt-ha-docker.git ; cd netvirt-ha-docker</tt><br/> <tt>./build /home/vorburger/dev/ODL/git/netvirt/</tt><br/> <tt>./up</tt><br/> <tt>./start</tt><br/> <tt>./wait-for-active</tt><br/> <tt>./karaf-client 1 showSvcStatus</tt><br/> <tt>./karaf-client 2 showSvcStatus</tt><br/> <tt>./karaf-client 3 showSvcStatus</tt></p> <p>but <tt>showSvcStatus -n</tt> indeed seems to have somehow broken, I'll dig into this:</p> <p><tt>./karaf-client 1</tt><br/> <tt>opendaylight-user@root&gt;shell:stack-traces-print</tt><br/> <tt>opendaylight-user@root&gt;showSvcStatus -n 172.28.5.1</tt><br/> <tt>opendaylight-user@root&gt;</tt></p> <p>PS: I initially got confused until I realized that <tt>./karaf-client 1 showSvcStatus -n 172.28.5.1</tt> <a href="https://github.com/jhershberg/netvirt-ha-docker/issues/11" class="external-link" target="_blank" rel="nofollow noopener">does not work because of this problem</a>, which is totally unrelated to what we are chasing here. (PPS: Also stumbled upon <a href="https://jira.opendaylight.org/browse/ODLPARENT-170" title="Karaf CLI &quot;java.util.regex.PatternSyntaxException: Illegal repetition near index&quot; when running in container and using dumb terminal" class="issue-link" data-issue-key="ODLPARENT-170"><del>ODLPARENT-170</del></a> here.)</p> <p>Looking at the code of <tt>DiagStatusCommand</tt> it sees to be virtually impossible for it to return without printing anything - it should either at the very least have "Timestamp: ..." OR an Exception - unless somehow it's possible that there is nothing in the log (I checked) and <tt>shell:stack-traces-print</tt> is broken?! I can try a custom build with an extra <tt>catch</tt>, but this is weird...</p> <p>&gt; try a custom build with an extra catch</p> <p>with <a href="https://git.opendaylight.org/gerrit/#/c/76562/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76562/</a>, I'm finally seeing this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.lang.IncompatibleClassChangeError: Found interface org.opendaylight.infrautils.diagstatus.ClusterMemberInfo, but class was expected at Proxy566b334a_4fa8_428a_a448_4444fecf36c0.isLocalAddress(Unknown Source) at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.getNodeSpecificStatus(DiagStatusCommand.java:118) at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.execute2(DiagStatusCommand.java:66) at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.execute(DiagStatusCommand.java:50) at org.apache.karaf.shell.impl.action.command.ActionCommand.execute(ActionCommand.java:84) at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:68) at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:86) at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:571) at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:497) at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:386) at org.apache.felix.gogo.runtime.Pipe.doCall(Pipe.java:417) at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:229) at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:59) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)</pre> </div></div> <p>PS: I'm also seeing that during boot there are a lot of new exceptions while we are not yet ready...</p> <p>&gt; java.lang.IncompatibleClassChangeError: Found interface org.opendaylight.infrautils.diagstatus.ClusterMemberInfo, but class was expected</p> <p>similar to but not exactly <a href="https://issues.apache.org/jira/browse/KARAF-5086" class="external-link" target="_blank" rel="nofollow noopener">https://issues.apache.org/jira/browse/KARAF-5086</a></p> <p><a href="https://git.opendaylight.org/gerrit/#/c/76564/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76564/</a> fixes this.</p> <p>&gt; PS: I'm also seeing that during boot there are a lot of new exceptions while we are not yet ready...</p> <p>That's those below, but that is just because the <tt>./wait-for-active</tt> script from <a href="https://github.com/jhershberg/netvirt-ha-docker" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jhershberg/netvirt-ha-docker</a> uses <tt>showSvcStatus</tt> "too soon". <a href="https://github.com/jhershberg/netvirt-ha-docker/issues/12" class="external-link" target="_blank" rel="nofollow noopener">If it used /diagstatus instead, then this wouldn't happen</a>. This therefore also should not happen in CSIT and real world, so ignoring this here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-10-02T14:49:48,323 | INFO | pipe-showsvcstatus | DiagStatusCommand | 265 - org.opendaylight.infrautils.diagstatus-shell - 1. 5.0.SNAPSHOT | invokeRemoteJMXOperation() JMX service URL: service:jmx:rmi://172.28.5.2/jndi/rmi://172.28.5.2:6886/server 2018-10-02T14:49:48,323 | INFO | pipe-showsvcstatus | DiagStatusCommand | 265 - org.opendaylight.infrautils.diagstatus-shell - 1. 5.0.SNAPSHOT | fetching status summary for node : 172.28.5.2 2018-10-02T14:49:48,331 | ERROR | pipe-showsvcstatus | DiagStatusCommand | 265 - org.opendaylight.infrautils.diagstatus-shell - 1. 5.0.SNAPSHOT | Exception while reaching Host 172.28.5.2 java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Co nnection refused to host: 172.28.5.2; nested exception is: java.net.ConnectException: Connection refused (Connection refused)] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) [?:?] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) [?:?] at org.opendaylight.infrautils.diagstatus.MBeanUtils.invokeRemoteMBeanOperation(MBeanUtils.java:159) [263:org.opendaylight.infrautils.diagstatus-api:1.5.0.SNAPSHOT] at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.getRemoteStatusSummary(DiagStatusCommand.java:104) [265:org.opendaylight.infrautils.diagstatus-shell:1.5.0.SNAPSHOT] at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.execute2(DiagStatusCommand.java:76) [265:org.opendaylight.infrautils.diagstatus-shell:1.5.0.SNAPSHOT] at org.opendaylight.infrautils.diagstatus.shell.DiagStatusCommand.execute(DiagStatusCommand.java:50) [265:org.opendaylight.infrautils.diagstatus-shell:1.5.0.SNAPSHOT] at org.apache.karaf.shell.impl.action.command.ActionCommand.execute(ActionCommand.java:84) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:68) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:86) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:571) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:497) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:386) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Pipe.doCall(Pipe.java:417) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:229) [126:org.apache.karaf.shell.core:4.1.5] at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:59) [126:org.apache.karaf.shell.core:4.1.5] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?] Caused by: javax.naming.ServiceUnavailableException at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136) ~[?:?] at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[?:?] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:?] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:?] ... 18 more Caused by: java.rmi.ConnectException: Connection refused to host: 172.28.5.2; nested exception is: java.net.ConnectException: Connection refused (Connection refused) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[?:?] at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338) ~[?:?] at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112) ~[?:?] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[?:?] at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[?:?] at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:?] at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:?] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:?] ... 18 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:?] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?] at java.net.Socket.connect(Socket.java:589) ~[?:?] at java.net.Socket.connect(Socket.java:538) ~[?:?] at java.net.Socket.&lt;init&gt;(Socket.java:434) ~[?:?] at java.net.Socket.&lt;init&gt;(Socket.java:211) ~[?:?] at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[?:?] at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[?:?] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[?:?] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[?:?] ... 2018-10-02T14:49:48,341 | INFO | pipe-showsvcstatus | DiagStatusCommand | 265 - org.opendaylight.infrautils.diagstatus-shell - 1.5.0.SNAPSHOT | invokeRemoteJMXOperation() JMX service URL: service:jmx:rmi://172.28.5.3/jndi/rmi://172.28.5.3:6886/server 2018-10-02T14:49:48,342 | INFO | pipe-showsvcstatus | DiagStatusCommand | 265 - org.opendaylight.infrautils.diagstatus-shell - 1.5.0.SNAPSHOT | fetching status summary for node : 172.28.5.3 </pre> </div></div> <p>&gt; with <a href="https://git.opendaylight.org/gerrit/#/c/76562/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76562/</a>, I'm finally seeing this:</p> <p>let's discuss this further in <a href="https://jira.opendaylight.org/browse/INFRAUTILS-55" title="Intro. new Wrapper to make sure Exceptions from Karaf CLI commands are certain to get logged" class="issue-link" data-issue-key="INFRAUTILS-55"><del>INFRAUTILS-55</del></a>.</p> <p>Thank you Michael. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Relates INFRAUTILS-55 Development Rank 0|i03j2f: