https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/GENIUS-138 genius <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=trozet" class="user-hover" rel="trozet">trozet</a> in private internal email asked this re. new /diagstatus from <del><a href="https://jira.opendaylight.org/browse/INFRAUTILS-33" title="Expose ready and/or diagstatus via a non-authenticated URL" class="issue-link" data-issue-key="INFRAUTILS-33"><del>INFRAUTILS-33</del></a></del> :</p> <blockquote><p>Would it be possible for the endpoint to also reveal if the node was an isolated leader?</p></blockquote> <p>I'm personally not familiar enough with clustering implementation details yet, but assuming that this information is available somewhere in controller, and hopefully already exposed by its JMX bean/s, then it may not be hard to improve our <a href="https://github.com/opendaylight/genius/blob/master/mdsalutil/mdsalutil-impl/src/main/java/org/opendaylight/genius/mdsalutil/diagstatus/internal/DatastoreServiceStatusProvider.java" class="external-link" target="_blank" rel="nofollow noopener">org.opendaylight.genius.mdsalutil.diagstatus.internal.DatastoreServiceStatusProvider</a> to report a <tt>ServiceDescriptor</tt> in <tt>ServiceState.ERROR</tt> if it's an isolated leader?</p> GENIUS-138

Improve Datastore Cluster diagstatus to indicate if node is currently an isolated leader

Improvement Medium Resolved Done Michael Vorburger Michael Vorburger Mon, 14 May 2018 14:21:08 +0000 Mon, 2 Jul 2018 10:38:35 +0000 Mon, 2 Jul 2018 10:38:35 +0000 Oxygen-SR3 Fluorine 0 5 <p>We typically use following curl commands on all 3 nodes of ODL cluster to collect the shard details.</p> <p>Output would be JMX output of all shards. So we need to parse "RaftState" attribute for each shard and RaftState can carry values - Leader, Follower as well as "Isolated Leader". Output can also provide other useful information such as involuntary leader moves, log index, repl index etc. </p> <p>We typically use below curl commands in an external script (shell, python etc.) for either polling / for on-demand dumping using Jolokia endpoint</p> <p>For DS to report ERROR status, we can follow simple logic as below</p> <p>If RaftState of <b>any</b> shard has value other than Leader OR Follower , then declare the entire DS (across shards) to be in ERROR state. Of course, that may not sound like a purist definition. But for all practical reasons, it should be good enough. </p> <p> </p> <p>For all Operational Shards</p> <p>=====================</p> <p>curl -u $jolokia_username:$jolokia_password <a href="http://%24cic_node_ip_address%24jolokia_port/" class="external-link" target="_blank" rel="nofollow noopener">http://$cluster_node_ip_address:$jolokia_port/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=Shards,name=*</a></p> <p> </p> <p>For all Config Shards </p> <p>================</p> <p>curl -u $jolokia_username:$jolokia_password <a href="http://%24cic_node_ip_address%24jolokia_port/" class="external-link" target="_blank" rel="nofollow noopener">http://$cluster_node_ip_address:$jolokia_port/jolokia/read/org.opendaylight.controller:type=DistributedConfigDatastore,Category=Shards,name=*</a></p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p>Yeah, I was more specifically looking for an HTTP error code if the node is isolated leader, rather than parsing HTTP response output.  Thought maybe that could be exposed with the new diagstatus.</p> <p>Could I suggest that instead of piling on more untraceable black magic with typeless querying of internal MBeans from one project to another we use this issue as the opportunity to do this right and strongly typed? Either entirely move <a href="https://github.com/opendaylight/genius/blob/master/mdsalutil/mdsalutil-impl/src/main/java/org/opendaylight/genius/mdsalutil/diagstatus/internal/DatastoreServiceStatusProvider.java" class="external-link" target="_blank" rel="nofollow noopener">org.opendaylight.genius.mdsalutil.diagstatus.internal.DatastoreServiceStatusProvider</a>  into controller (with a dependency from controller to infrautils.diagstatus; would that be acceptable <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> ?) OR create a real public OSGi Service API to query the required information in controller, and have genius...DatastoreServiceStatusProvider invoke that.</p> <p>JMX is available locally (<a href="https://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/ManagementFactory.html#getPlatformMBeanServer()" class="external-link" target="_blank" rel="nofollow noopener">https://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/ManagementFactory.html#getPlatformMBeanServer()</a> ), no need to connect all over the place. It does not really make sense to monitor remote nodes (i.e. if the cluster does not work, why do you think you can still access that node?).</p> <p>As for diagstatus, I don't particularly care as long as there is equivalent information available from a single source and the framework we are going to (diagstatus) is clearly superior than the one we are leaving (JMX).</p> <p>If anyone disagrees, with that point, I will underline the difference in management of your typical router via CLI and SNMP – this difference comes from having two disjoint management infrastructures, where SNMP is a second class citizen (because admins interact with the router via CLI, not through SNMP).</p> <p><a href="https://lists.opendaylight.org/pipermail/controller-dev/2018-June/014404.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/controller-dev/2018-June/014404.html</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=MuthukumaranK" class="user-hover" rel="MuthukumaranK">MuthukumaranK</a> thanks for tips above, that was very helpful, and is now codified in <a href="https://git.opendaylight.org/gerrit/#/c/72797/." class="external-link" target="_blank" rel="nofollow noopener">c/72797</a>.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> thank you for your feedback above; we'll keep DatastoreServiceStatusProvider in genius; no move to controller planned (by me).</p> <p>&gt; instead of piling on more untraceable black magic with typeless querying of internal MBeans from one project to another<br/> &gt; we use this issue as the opportunity to do this right and strongly typed</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/72789/" class="external-link" target="_blank" rel="nofollow noopener">72789</a> addresses this in a way that satisfies me.</p> <p>This is now fully done, and just merged: diagstatus (whether via its CLI or /diagstatus or /jolokia URLs) will now report OPERATIONAL for the DATASTORE only if all Shards are Leaders or Followers; if they are in other states (which presumably includes an <em>isolated leader</em>; the original requirement above), the status will be ERROR and the Description will provide details. Took a couple of things to get this, see <a href="https://git.opendaylight.org/gerrit/#/q/topic:GENIUS-138" class="external-link" target="_blank" rel="nofollow noopener">topic:GENIUS-138</a>. </p> <p>Re-opening this for backporting to Oxygen SR3 (requested by <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=trozet" class="user-hover" rel="trozet">trozet</a>)...</p> <p>&gt; Re-opening this for backporting to Oxygen SR3 (requested by <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=trozet" class="user-hover" rel="trozet">trozet</a>)...</p> <p>which would only really be useful if we also back-port <a href="https://jira.opendaylight.org/browse/INFRAUTILS-33" title="Expose ready and/or diagstatus via a non-authenticated URL" class="issue-link" data-issue-key="INFRAUTILS-33"><del>INFRAUTILS-33</del></a> ...</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> : Merged all the patches under the topic. Please double check if everything is in, and if so we can close the JIRA</p> Blocks NETVIRT-1003 INFRAUTILS-43 CONTROLLER-1835 Development Rank 0|i03eif: