Health verification endpoints

Service health verification endpoints

PQM provides kubernetes probes and a health endpoint. In contrast to the probes, the health endpoints are not intended to be used by the container management system, but for privileged (authorization required) users like operators or tools of an operator. For that reason the endpoints are exposed and will provide additional information than a simple ok, nok status.

Health endpoint

Authorization: Bearer Token
ResourceType: urn:com:bosch:bci:pqm:api
ResourceID: $process-quality.health
Role: PQM_OPERATOR

HTTP Method: GET
Path: /health

{
  "name": "Process Quality Module (PQM)",
  "instanceId": "2b0bc615-7ddb-48e9-b143-c630bb262106",
  "description": "Self contained system",
  "startupTime": "2023-12-18T08:56:03.231114292Z",
  "version": "3.2.0-rev",
  "ready": true,
  "health": "healthy",
  "dependencies": [
    {
      "name": "static-content",
      "description": "Webserver that provides static resources like the JS client application to the browser",
      "available": true,
      "details": {
        "instanceId": "N/A",
        "startupTime": null,
        "health": "healthy",
        "details": {},
        "version": "main",
        "dependencies": []
      }
    },
    {
      "name": "trinity-core-service",
      "available": true,
      "details": {
        "instanceId": "da050691-826a-4b9c-bfc5-6efaa28ee854",
        "startupTime": "2023-12-18T08:56:15.156302089Z",
        "health": "healthy",
        "details": {},
        "version": "main-20231214180723-DEV-SNAPSHOT",
        "dependencies": [
          {
            "name": "db",
            "available": true,
            "details": {
              "database": "Microsoft SQL Server",
              "validationQuery": "SELECT 1",
              "result": 1
            }
          },
          {
            "name": "diskSpace",
            "available": true,
            "details": {
              "total": 133003395072,
              "free": 22655647744,
              "threshold": 10485760,
              "path": "/app/.",
              "exists": true
            }
          },
          {
            "name": "migrationInfluxConnection",
            "available": true,
            "details": {
              "version": "1.7.11"
            }
          },
          {
            "name": "processInfluxConnection",
            "available": true,
            "details": {
              "version": "1.7.11"
            }
          },
          {
            "name": "livenessState",
            "available": true
          },
          {
            "name": "macma",
            "available": true
          },
          {
            "name": "mdm",
            "available": true
          },
          {
            "name": "portal",
            "available": true
          },
          {
            "name": "rabbit",
            "available": true,
            "details": {
              "version": "3.11.24"
            }
          },
          {
            "name": "readinessState",
            "available": true
          },
          {
            "name": "refreshScope",
            "available": true
          },
          {
            "name": "smdp",
            "available": true
          }
        ]
      }
    }
  ]
}

Kubernetes probes

Liveness Probe A liveness probe can return
1. "Yes, I’m alive!" which will be evaluated by the runtime environment and no further action is executed.
2. "No I’m not alive!" which will be evaluated by the runtime environment. The container will be killed and restarted by the runtime environment.
3. Or it simply fails since the container is not able to respond to the probe for some reasons. The container will be killed and restarted by the runtime environment.
Readiness Probe A readiness probe can return
1. "Yes, I’m ready and can serve functionality to my clients!" which will be evaluated by the runtime environment and no further action is executed.
2. "No, I’m not ready!" which will be evaluated by the runtime environment and no traffic will be routed to the container. The runtime envrionment will ask again with a configurable amount of retries and with a configurable delay. If this probe fails everytime the pod will marked as unready.
3. Or it simply fails since the container is not able to respond to the probe for some reasons. The runtime environment will stop routing traffic to the container. The runtime environment will ask again with a configurable amount of retries and with a configurable delay. If this probe fails everytime the pod will marked as unready
Startup Probe A startup probe is intendet to suppress other probes during conatainers startup or initialization phases for preventing restarts. PQM uses the liveness probe as startup probe. It can return
1. "I’m succesfully started." From now on the runtime environment will start executing the liveness probes.
2. "I’m in the startup phase." the runtime environment will ask again with a configurable amount of retries and with configurable delay. If this probe fails everytime the pod might be restarted (depending on the pods restart policy).
3. Or it simply fails since the container is not able to respond to the probe for some reasons.

Table shows how PQM’s microservices will behave for several scenarios of infrastructure outages.

Startup

Scenario	Spring context starting Database migration RabbitMQ setup
Impact	Service is not capable of accepting traffic
General Behaviour	-
Readiness Endpoint	http response: 503
Liveness Endpoint	http response: 503

Scenario

Spring context starting
Database migration
RabbitMQ setup

Impact

Service is not capable of accepting traffic

General Behaviour

Readiness Endpoint

http response: 503

Liveness Endpoint

http response: 503

RabbitMQ

Scenario	Lost connection to RabbitMQ Reasons: RabbitMQ instance crashed, restarted, etc. Network issue between microservice and RabbitMQ
Impact	Only in case if RabbitMQ outage: Lost of transient PPMP PPMP messages which will be send PQM internally to legacy microservices Several functionality will not work anymore e.g. no new PPMP data will be shown The UI will still respond and shows existing data from databases and cache
General Behaviour	The lost connection to RabbitMQ will be logged on error The microservice is trying to reconnect infinitely PQM keeps the service alive to keep the UI running RabbitMQ objects (e.g. queues and bindings) will be recreated once the microservice reconnected
Readiness Endpoint	http response: 200 PQM can still accept requests to the microservice e.g. from the UI Incoming PPMP data must be handled by the microservice itself by rejecting any incoming PPMP requests
Liveness Endpoint	http response: 200 The microservice will still be alive, the internal state is valid and does not need to get restarted

Scenario

Lost connection to RabbitMQ
Reasons:
- RabbitMQ instance crashed, restarted, etc.
- Network issue between microservice and RabbitMQ

Impact

Only in case if RabbitMQ outage:
- Lost of transient PPMP
  - PPMP messages which will be send PQM internally to legacy microservices
Several functionality will not work anymore e.g. no new PPMP data will be shown
The UI will still respond and shows existing data from databases and cache

General Behaviour

The lost connection to RabbitMQ will be logged on error
The microservice is trying to reconnect infinitely
PQM keeps the service alive to keep the UI running
RabbitMQ objects (e.g. queues and bindings) will be recreated once the microservice reconnected

Readiness Endpoint

http response: 200
PQM can still accept requests to the microservice e.g. from the UI
Incoming PPMP data must be handled by the microservice itself by rejecting any incoming PPMP requests

Liveness Endpoint

http response: 200
The microservice will still be alive, the internal state is valid and does not need to get restarted

Database

Scenario	Lost connection to the database Reasons: Database instance crashed, restarted, etc. Network issue between microservice and database
Impact	The UI will not respond and show any data Most of the functionality will not work anymore
General Behaviour	The lost connection to database will be logged on error The microservice is trying to reconnect infinitely PQM will keep the service alive
Readiness Endpoint	http response: 503 PQM will not accept requests to the microservice e.g. from the UI PQM will not accept incoming PPMP data
Liveness Endpoint	http response: 200 The microservice will still be alive, the internal state is valid and does not need to get restarted to not face more problems with RabbitMQ and loss of messages

Scenario

Lost connection to the database
Reasons:
- Database instance crashed, restarted, etc.
- Network issue between microservice and database

Impact

The UI will not respond and show any data
Most of the functionality will not work anymore

General Behaviour

The lost connection to database will be logged on error
The microservice is trying to reconnect infinitely
PQM will keep the service alive

Readiness Endpoint

http response: 503
PQM will not accept requests to the microservice e.g. from the UI
PQM will not accept incoming PPMP data

Liveness Endpoint

http response: 200
The microservice will still be alive, the internal state is valid and does not need to get restarted to not face more problems with RabbitMQ and loss of messages

Influx

Scenario	Lost connection to the InfluxDB Reasons: Influx instance crashed, restarted, etc. Network issue between microservice and InfluxDB
Impact	The UI will respond but it will not show any PPMP data Several functionality will not work anymore e.g. no PPMP data will be shown
General Behaviour	The lost connection ot Influx will be logged on error PQM will keep the service alive
Readiness Endpoint	http response: 200 PQM can still accept requests to the microservice e.g. from the UI Incoming PPMP data must be handled by the microservice itself by rejecting any incoming PPMP requests
Liveness Endpoint	http response: 200 The microservice will still be alive. The internal state is valid and does not need to get restarted

Scenario

Lost connection to the InfluxDB
Reasons:
- Influx instance crashed, restarted, etc.
- Network issue between microservice and InfluxDB

Impact

The UI will respond but it will not show any PPMP data
Several functionality will not work anymore e.g. no PPMP data will be shown

General Behaviour

The lost connection ot Influx will be logged on error
PQM will keep the service alive

Readiness Endpoint

http response: 200
PQM can still accept requests to the microservice e.g. from the UI
Incoming PPMP data must be handled by the microservice itself by rejecting any incoming PPMP requests

Liveness Endpoint

http response: 200
The microservice will still be alive. The internal state is valid and does not need to get restarted

The following diagram shows how Kubernetes probes pod health verification endpoints to determine the pod state.