Module Health Verification Endpoints and K8S Probes
-
Health endpoint: Will be used by monitoring to determine the health state of a Microservice. The health endpoint will also reflect the states of the Liveness, Readiness and Ping States. It is not intended to be used by the container management system, but for privileged users (authorization required) like operators or tools of an operator. For that reason the endpoints are exposed and will provide additional information to authorized users, and only a minimal healthy/unhealthy status in case of unprivileged access.
-
Readiness endpoint: Will be polled by Kubernetes to check, if a Microservice (pod) can accept traffic.
-
Liveness endpoint: Will be polled by Kubernetes to check, if a Microservice’s internal state is valid. Kubernetes will restart the Container if the liveness endpoint fails for the configured number of times.
-
Ping endpoint: Always returns "Healthy". It is intended to be used by other services in their health endpoint implementation to check availability of other services they are relying on. It only ensures that a service client is able to connect to a service.
IdBuilder service
IdBuilder service has the health and probes endpoints available at their respective base URL address (ex: \https://[serverName]/idbuilder/health)
Health and Probes
-
Health Endpoint
-
/health
-
up: 200
-
down: 503
-
Health endpoint is showing down and answering with http status code 503. IdBuilder service is considered to not be healthy e.g. in case of lost connection to MACMA and/or Database.
-
In case of authorized access, the Health endpoint will also show details about the individual health status of dependencies to external services like MACMA, Database and Portal.
-
-
-
Liveness Endpoint
-
/health/live
-
unhealthy: 503
-
only during startup and shutdown it will be down
-
-
healthy: 200
-
-
Readiness Endpoint
-
/health/readiness
-
unhealthy: 503
-
If Database or MACMA are down, the service will not accept requests as long as readiness isn’t restored
-
-
healthy: 200
-
-
Ping Endpoint
-
/ping
-
always healthy: 200
-
Dependencies
The status of probes (healthy/unhealthy) are based on health checks for service dependencies
-
Database
-
Use cases:
Lost connection to the Database.
Reasons:-
Database instance crashed, restarted etc
-
Network issue between microservice and Database
-
-
General Behavior
-
The lost connection to Database will be logged on error.
-
The microservice is trying to reconnect infinitely.
-
We keep the Service alive, but it won’t be ready to accept any requests.
-
-
Impact
-
The service will no longer be functional
-
-
-
Macma
-
Portal
-
Use cases:
Lost connection to Portal.
Reasons:-
Portal instance crashed, restarted etc
-
Network issue between microservice and Portal
-
Portal cannot accept traffic (readiness state is "Unhealty")
-
-
General Behavior
-
The lost connection to Portal will be logged on error.
-
The microservice is trying to reconnect infinitely.
-
We keep the Service alive, but e.g. access to UI might be unavailable
-
-
Impact
-
The service will still be functional as Portal is considered an optional dependency.
-
-
Infrastructure Outages
If a required infrastructure or module will become unavailable IdBuilder will immediately reflect the status in the health response. The general rule is that when a failed dependency becomes available again IdBuilder will automatically restore connection without any user intervention.
-
MACMA IdBuilder will try connection on every incoming authenticated request. There is no continuous background connection to MACMA.
-
Database IdBuilder will attempt a DB connection at every incoming service request and thus restore connectivity as soon as the database becomes available. The /health state has no influence on the db connection attempts.
-
Portal Once registered successfully, IdBuilder does not re-attempt registration if portal transitions between unhealthy and healthy states. The portal registration flow is not related in any way to the health monitoring endpoints.