Required monitoring

It is important to monitor some log messages and metrics because they are indicators that manual analysis or action is required.

Logs

Alert rules maybe use 'JSONPath' for addressing values in structured logs. This is indicated with a '$'.

Table 1. Required log monitoring
Message	Logger	Alert rule (pseudo code)	Context	Symptoms	Solution
Service crashed	LIFE-CYCLE	(($stackTraces[*].failureType = 'liquibase.exception.LockException') > 0) in one minute	Database migration	Pod does not start up / pod restarts	Assert if the kubernetes deployment, listed in the 'LOCKEDBY' column of table 'PQM_TRN_DATABASECHANGELOGLOCK', is running. If not, delete the entry from the database. If so, wait for the migration to complete.
Health status is [DOWN]. <further details>	LIFE-CYCLE	(($status = 'UNHEALTHY') > 0) in one minute	System availability	Loss of functionality, loss of data, system unavailability	Inspect the further details of the message. This contains a detailed report of the problem.
Synchronisation of master data for tenant [<ID_OF_AFFECTED_TENANT>] failed	com.bosch.bci.trinity.trn.ias.mdm.synchronisation.per.tenant.MdmSyncService	(($service.logger = 'com.bosch.bci.trinity.trn.ias.mdm.synchronisation.per.tenant.MdmSyncService') and ($service.message = Synchronisation of master data for tenant*)) > 0) in one day	Device master data	Devices are missing in PQM	Assign the required MDM roles to the PQM client.

Metrics

Table 2. Required metric monitoring
Metric	Alert rule (pseudo code)	Context	Symptoms	Solution
Length of queue q.pqm.dlq.integration-events	Length of q.pqm.dlq.integration-events > 0	Master data synchronization between PQM and MDM, as well as between PQM and MACMA.	MDM master data like devices and facilities are not visible in PQM.	The system heals itself withing 30 minutes. If this does not succeed, execute the '/master-data/v1/{ownerTenantId}/mdm/recreate' endpoint. See the API documentation for further details.