jluhrsen, trozet, JankiChhatbar (and FYI k.faseela) I've started looking into, and want to clarify 2 points before code:
- the /diagstatus would return a JSON; not human readable text - makes sense, right? FYI the format would be as in org.opendaylight.infrautils.diagstatus.internal.DiagStatusServiceMBeanImpl.acquireServiceStatusAsJSON(), or close to it. (There are just two minor things there to clarify which don't make sense to me, yet.)
- contrary to the diagstatus:showSvcStatus CLI command, I do not want all that JMX stuff we have in that code related to querying cluster nodes in this feature; my view is that whatever external system queries for such status has to already know about, or have an external way to find, the cluster members, and query them individually; that model seems more robust than one node (which could be down!) itself querying others. This was a personal view I've previously held, but seeing that is how e.g. Prometheus.io goes about the same for metrics, seems to confirm this is the right approach.
Are both OK for all of you?
Jamo Luhrsen, Tim Rozet, Janki Chhatbar (and FYI Faseela K) I've started looking into, and want to clarify 2 points before code:
the /diagstatus would return a JSON; not human readable text - makes sense, right? FYI the format would be as in org.opendaylight.infrautils.diagstatus.internal.DiagStatusServiceMBeanImpl.acquireServiceStatusAsJSON(), or close to it. (There are just two minor things there to clarify which don't make sense to me, yet.)
yeah, json response is fine. The healthcheck might/could just be as dumb as sending a curl to this endpoint and marking things
healthy if we get a 200 response. Maybe it wouldn't even worry about the body of the response. I mean, it could and that would
probably be smarter, but it doesn't have to.
contrary to the diagstatus:showSvcStatus CLI command, I do not want all that JMX stuff we have in that code related to querying cluster nodes in this feature; my view is that whatever external system queries for such status has to already know about, or have an external way to find, the cluster members, and query them individually; that model seems more robust than one node (which could be down!) itself querying others. This was a personal view I've previously held, but seeing that is how e.g. Prometheus.io goes about the same for metrics, seems to confirm this is the right approach.
Are both OK for all of you?
I'm not totally following what you are getting at here. I think this healthcheck endpoint is on a container by container basis
and there doesn't need to be any broader knowledge of anything other than if that specific ODL container is healthy or
not. But, maybe you are getting at a more intelligent healthcheck that not only tells us if that one container is healthy, but
also if that container is also running in a healthy cluster?
Agree to vorburger, the CLI is just an example implementation of a client to show how status can be retrieved from all nodes. Orchestrator will definitely know about all 3 nodes in a cluster, and hence should be easy for the orchestrator to fetch the status from all nodes by constructing the REST URLs, specific to the node IPs.
Healthcheck just curls on specified URL and says container is healthy if the output of command is 0. This is run from inside the container. So each ODL container in a cluster will check for health for itself. Hence no need to worry about the cluster setup. They won't even know whether they are in cluster.
A minor clarification. Does orchestrator use a static address list and and curls on the URL ?
When one of cluster node is down during, I assume it would be treated as a timeout on curl for the corresponding node if the list is static.
@everyone, I think we should ignore anything to do with clustering in this jira
@janki, I don't think healthcheck has to be a simple curl, only. We can do whatever crazy bash-fu we want to do. So, in the end we could do curl to get the full response
body and pipe it to some script that does anything we want and that script can set the exit code to 0 if healthy, or not.
Completed implementation, see 3 linked Gerrit reviews (c/70987, c/71168 and c/71172).
With this, the odl-infrautils-diagstatus feature will expose this on http://localhost:8181/diagstatus/ :
{
"timeStamp": "Fri Apr 20 16:41:26 CEST 2018",
"isOperational": true,
"systemReadyState": "ACTIVE",
"statusSummary": []
}
in infrautils/common/karaf with no other features, or e.g. with odl-netvirt-openstack like this:
{
"timeStamp": "Fri Apr 20 17:09:33 CEST 2018",
"isOperational": true,
"systemReadyState": "ACTIVE",
"statusSummary": [
{
"serviceName": "OPENFLOW",
"effectiveStatus": "OPERATIONAL",
"reportedStatusDescription": "switch connections started",
"statusTimestamp": "2018-04-20T15:08:29.713Z"
},
{
"serviceName": "IFM",
"effectiveStatus": "OPERATIONAL",
"reportedStatusDescription": "Service started",
"statusTimestamp": "2018-04-20T15:08:17.268Z"
},
{
"serviceName": "ITM",
"effectiveStatus": "OPERATIONAL",
"reportedStatusDescription": "Service started",
"statusTimestamp": "2018-04-20T15:08:19.617Z"
},
{
"serviceName": "ELAN",
"effectiveStatus": "OPERATIONAL",
"reportedStatusDescription": "Service started",
"statusTimestamp": "2018-04-20T15:08:20.191Z"
},
{
"serviceName": "DATASTORE",
"effectiveStatus": "OPERATIONAL",
"reportedStatusDescription": "OPERATIONAL",
"statusTimestamp": "2018-04-20T15:09:33.509Z"
}
]
}
BTW jluhrsen idea in INFRAUTILS-31 would be quite a nice complementary addition here. Anyone willing to wrap that up? 
FYI http://localhost:8181/diagstatus/ will return the following HTTP status codes:
418/ 503 (seeINFRAUTILS-46) if isOperational = false, with details in the JSON; either systemReadyState not ACTIVE (=technical OSGi problem) OR systemReadyState = ACTIVE but one of the entries in statusSummary NOK (= more of a "functional" problem)- 401 Problem accessing /diagstatus/. Reason: Unauthorized indicates, confusingly, NOT that authentication is missing, but that there is nothing at that URL - i.e. infrautils has not even started (then things are REALLY screwed)
- 500 if infrautils is up but has hit an internal error (details should be in karaf.log)
- 200 OK if isOperational = true and all is well (will also have details in the JSON)
HTH.
This is all completely finished, done and dusted now - from my side.
jluhrsen and thapar have internally suggested that this would be good to have not only on master for Fluorine but also on stable/oxygen already - I'm therefore reopening this issue and look into doing the back-port, hopefully some time next week.
jluhrsen is now using jolokia/exec/org.opendaylight.infrautils.diagstatus/... instead of /diagstatus, so my understanding is there is no immediate need / request aanymore to back-port this to stable/oxygen amymore after all, so closing it again.
Re-opening this for backporting to Oxygen SR3, so that GENIUS-138 is really useful...