Startup and availability

MACMA Core fails to start due to locked database

Context: Application does not become available when the database is locked. This error can occur, if the application has been stopped abruptly during database migration, thus a database lock has not removed properly.

Symptoms: 502 Bad Gateway errors, e.g. when trying to create application resources. Repeated restarts in short succession.

Solution: Check the logs for the macma-core-deployment pods. The logs may contain one of the following messages

Waiting for changelog lock....
liquibase.exception.LockException: Could not acquire change log lock
Application run failed

If you don’t expect partial database migrations you can lift the lock, otherwise the database has to be repaired manually. To lift the lock, connect to the database used for MACMA Core and execute to following SQL-command:

SELECT * FROM IAM_DATABASECHANGELOGLOCK

Verify, that the returned result set has one entry and the LOCKED is set to 1.
To release the lock, run the following SQL-command:

UPDATE IAM_DATABASECHANGELOGLOCK SET LOCKED=0, LOCKGRANTED=null, LOCKEDBY=null where ID=1

MACMA Core should recover itself and be available again.

Logs about MACMA being potentially unhealthy, but MACMA is available

Context: Some services monitor availability of dependencies.

Symptoms: Multiple but not all service instances fail to reach MACMA and thus determine that it is unhealthy. But at the same time MACMA is running just fine and not showing any signs of health issues. The other modules typically also log timeouts or task cancellations rather than gateway errors.

Solution: Check the firewall logs, potentially there was a foul component or an attacker in the system, causing the firewall to block certain IPs. These blocked IPs may affect multiple service instances of other modules in the cluster.

Calls to MACMA Core lead to infinite redirects (status code 302)

Context: When opening MACMA Webapp or calling the APIs directly.

Symptoms: Infinite loop of 302 redirect status codes.

Solution: Set JAVA_TLS_DISABLE accordingly when terminating TLS and thus only HTTP calls reach the backend service, otherwise it will redirect the caller to HTTPS for security reasons.