https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/INFRAUTILS-33 infrautils <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1549218" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1549218</a> and related internal discussion have revealed that it would be useful for the TripleO installation orchestrator to health check a non-authenticated URL for ready and/or diagstatus status.</p> <p>They cannot use that jolokia JMX HTTP bridge URL which requires authentication at the point where they need to make that check.</p> <p>It should not be very hard to write a Servlet which exposes similar information than the diag CLI command, with a HTTP status code (like 200 vs 503) and the body containing output similar to that CLI command.</p> <p>This will not break the existing diagstatus JMX bean exposed via (authenticated) /jolokia, but will be in addition to that.</p> INFRAUTILS-33

Expose ready and/or diagstatus via a non-authenticated URL

Improvement Medium Resolved Done Michael Vorburger Michael Vorburger Wed, 11 Apr 2018 13:20:47 +0000 Wed, 22 Aug 2018 21:14:49 +0000 Mon, 2 Jul 2018 10:53:03 +0000 Oxygen-SR3 Fluorine diagstatus 0 5 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=trozet" class="user-hover" rel="trozet">trozet</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=JankiChhatbar" class="user-hover" rel="JankiChhatbar">JankiChhatbar</a> (and FYI <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>) I've started looking into, and want to clarify 2 points before code:</p> <ul> <li>the /diagstatus would return a JSON; not human readable text - makes sense, right? FYI the format would be as in <tt>org.opendaylight.infrautils.diagstatus.internal.DiagStatusServiceMBeanImpl.acquireServiceStatusAsJSON()</tt>, or close to it.  (There are just two minor things there to clarify which don't make sense to me, yet.)</li> <li>contrary to the <tt>diagstatus:showSvcStatus</tt> CLI command, I do not want all that JMX stuff we have in that code related to querying cluster nodes in this feature; my view is that whatever external system queries for such status has to already know about, or have an external way to find, the cluster members, and query them individually; that model seems more robust than one node (which could be down!) itself querying others. This was a personal view I've previously held, but seeing that is how e.g. Prometheus.io goes about the same for metrics, seems to confirm this is the right approach.</li> </ul> <p>Are both OK for all of you?</p> <blockquote><p>Jamo Luhrsen, Tim Rozet, Janki Chhatbar (and FYI Faseela K) I've started looking into, and want to clarify 2 points before code:</p> <p> the /diagstatus would return a JSON; not human readable text - makes sense, right? FYI the format would be as in org.opendaylight.infrautils.diagstatus.internal.DiagStatusServiceMBeanImpl.acquireServiceStatusAsJSON(), or close to it. (There are just two minor things there to clarify which don't make sense to me, yet.)</p></blockquote> <p>yeah, json response is fine. The healthcheck might/could just be as dumb as sending a curl to this endpoint and marking things<br/> healthy if we get a 200 response. Maybe it wouldn't even worry about the body of the response. I mean, it could and that would<br/> probably be smarter, but it doesn't have to.</p> <blockquote> <p> contrary to the diagstatus:showSvcStatus CLI command, I do not want all that JMX stuff we have in that code related to querying cluster nodes in this feature; my view is that whatever external system queries for such status has to already know about, or have an external way to find, the cluster members, and query them individually; that model seems more robust than one node (which could be down!) itself querying others. This was a personal view I've previously held, but seeing that is how e.g. Prometheus.io goes about the same for metrics, seems to confirm this is the right approach.</p> <p>Are both OK for all of you?</p></blockquote> <p>I'm not totally following what you are getting at here. I <b>think</b> this healthcheck endpoint is on a container by container basis<br/> and there doesn't need to be any broader knowledge of anything other than if that specific ODL container is healthy or<br/> not. But, maybe you are getting at a more intelligent healthcheck that not only tells us if that one container is healthy, but<br/> also if that container is also running in a healthy cluster?</p> <p>Agree to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>, the CLI is just an example implementation of a client to show how status can be retrieved from all nodes. Orchestrator will definitely know about all 3 nodes in a cluster, and hence should be easy for the orchestrator to fetch the status from all nodes by constructing the REST URLs, specific to the node IPs.</p> <p>Healthcheck just curls on specified URL and says container is healthy if the output of command is 0. This is run from inside the container. So each ODL container in a cluster will check for health for itself. Hence no need to worry about the cluster setup. They won't even know whether they are in cluster.</p> <p>A minor clarification. Does orchestrator use a static address list and and curls on the URL ?</p> <p>When one of cluster node is down during, I assume it would be treated as a timeout on curl for the corresponding node if the list is static. </p> <p>@everyone, I think we should ignore anything to do with clustering in this jira</p> <p>@janki, I don't think healthcheck has to be a simple curl, only. We can do whatever crazy bash-fu we want to do. So, in the end we could do curl to get the full response<br/> body and pipe it to some script that does anything we want and that script can set the exit code to 0 if healthy, or not.</p> <p>Completed implementation, see 3 linked Gerrit reviews (c/70987, c/71168 and c/71172).</p> <p>With this, the <tt>odl-infrautils-diagstatus</tt> feature will expose this on <tt><a href="http://localhost:8181/diagstatus/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/diagstatus/</a></tt> :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>{ "timeStamp": "Fri Apr 20 16:41:26 CEST 2018", "isOperational": true, "systemReadyState": "ACTIVE", "statusSummary": [] }</pre> </div></div> <p>in infrautils/common/karaf with no other features, or e.g. with <tt>odl-netvirt-openstack</tt> like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>{ "timeStamp": "Fri Apr 20 17:09:33 CEST 2018", "isOperational": true, "systemReadyState": "ACTIVE", "statusSummary": [ { "serviceName": "OPENFLOW", "effectiveStatus": "OPERATIONAL", "reportedStatusDescription": "switch connections started", "statusTimestamp": "2018-04-20T15:08:29.713Z" }, { "serviceName": "IFM", "effectiveStatus": "OPERATIONAL", "reportedStatusDescription": "Service started", "statusTimestamp": "2018-04-20T15:08:17.268Z" }, { "serviceName": "ITM", "effectiveStatus": "OPERATIONAL", "reportedStatusDescription": "Service started", "statusTimestamp": "2018-04-20T15:08:19.617Z" }, { "serviceName": "ELAN", "effectiveStatus": "OPERATIONAL", "reportedStatusDescription": "Service started", "statusTimestamp": "2018-04-20T15:08:20.191Z" }, { "serviceName": "DATASTORE", "effectiveStatus": "OPERATIONAL", "reportedStatusDescription": "OPERATIONAL", "statusTimestamp": "2018-04-20T15:09:33.509Z" } ] }</pre> </div></div> <p>BTW <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> idea in <a href="https://jira.opendaylight.org/browse/INFRAUTILS-31" title="diagstatus including cause of failures" class="issue-link" data-issue-key="INFRAUTILS-31"><del>INFRAUTILS-31</del></a> would be quite a nice complementary addition here. Anyone willing to wrap that up? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>FYI <tt><a href="http://localhost:8181/diagstatus/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/diagstatus/</a></tt> will return the following HTTP status codes: </p> <ul> <li><del>418</del> / 503 (see <a href="https://jira.opendaylight.org/browse/INFRAUTILS-46" title="diagstatus error code is 418 instead of 503" class="issue-link" data-issue-key="INFRAUTILS-46"><del>INFRAUTILS-46</del></a>) if isOperational = false, with details in the JSON; either systemReadyState not ACTIVE (=technical OSGi problem) OR systemReadyState = ACTIVE but one of the entries in statusSummary NOK (= more of a "functional" problem)</li> <li>401 <em>Problem accessing /diagstatus/. Reason: Unauthorized</em> indicates, confusingly, NOT that authentication is missing, but that there is nothing at that URL - i.e. infrautils has not even started (then things are REALLY screwed)</li> <li>500 if infrautils is up but has hit an internal error (details should be in karaf.log)</li> <li>200 OK if isOperational = true and all is well (will also have details in the JSON)</li> </ul> <p>HTH.</p> <p>This is all completely finished, done and dusted now - from my side.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> have internally suggested that this would be good to have not only on master for Fluorine but also on stable/oxygen already - I'm therefore reopening this issue and look into doing the back-port, hopefully some time next week.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> is now using jolokia/exec/org.opendaylight.infrautils.diagstatus/... instead of /diagstatus, so my understanding is there is no immediate need / request aanymore to back-port this to stable/oxygen amymore after all, so closing it again.</p> <p>Re-opening this for backporting to Oxygen SR3, so that <a href="https://jira.opendaylight.org/browse/GENIUS-138" title="Improve Datastore Cluster diagstatus to indicate if node is currently an isolated leader" class="issue-link" data-issue-key="GENIUS-138"><del>GENIUS-138</del></a> is really useful...</p> Blocks INTTEST-45 INFRAUTILS-45 Relates INFRAUTILS-47 INFRAUTILS-46 Development Rank 0|i03d73: