Logging and monitoring

Open telemetry

Our services supports Open Telemetry. The Open Telemetry configuration gets injected by Utility-Toolkit into otel-observability-configmap.

Log format

The logs produce a json record for each log entry. The following sample shows a typical log entry:

Log sample

{"timestamp":"2023-01-18T07:17:29.8873990Z","environment":"nexeed-mmpd-equipment-management","system":"","application":"Bosch.Industry.Mdp.Mmpd.EquipmentManagementService","product":"Bosch.Industry.Mdp.Mmpd.EquipmentManagementService","version":"6.6.23018.1","instance":"0ecef3fe-17aa-4d49-adec-58ce167cc6a9","logger":"Bosch.Industry.Mdp.Mmpd.Core.Persistence.EntityFramework.EFDbAdapter","loglevel":"WARN","thread":".NET ThreadPool Worker","correlationId":"9f0a3d85-55b5-4549-aff1-9c113fa89686","userId":"admin","message":"Query could not be translated for entity Device. Rql:and(eq(recordState, Approved),and(eq(isRetired, false),eq(deviceTypeId, fe105dff-6e23-43ad-a1ac-36647baf7ff5)))","SpanId":"30a1e85de47a9fc2","TraceId":"d8898b2a6b3b4902170b04d60ca39d71","ParentId":"2a172dd8703cf951","ConnectionId":"0HMNOTO7RCJRQ","RequestId":"0HMNOTO7RCJRQ:00000002","RequestPath":"/uiquery-api/v1/7311ea8c-5d48-43fe-acf9-980eedf24b6c/Devices","com.bosch.correlationId":"9f0a3d85-55b5-4549-aff1-9c113fa89686","ActionId":"60a5b763-2bc1-4fbc-b096-adfc3bfe09f8","ActionName":"Bosch.Industry.Mdp.Mmpd.EquipmentManagementService.Controllers.UI.DevicesController.PostWithRql (Bosch.Industry.Mdp.Mmpd.EquipmentManagementService)","{OriginalFormat}":"Query could not be translated for entity Device. Rql:and(eq(recordState, Approved),and(eq(isRetired, false),eq(deviceTypeId, fe105dff-6e23-43ad-a1ac-36647baf7ff5)))"}

Logging configuration

The logs can be configured via the environment settings.

Default Logging level

Logging__LogLevel__Default: Warning

Specific logging levels

Logging__LogLevel__Elastic.Apm: None
Logging__LogLevel__Bosch.Industry.Mdp.Mmpd.Core.Persistence.EntityFramework.EFDbAdapter: Information
Logging__LogLevel__Bosch.Industry.Mdp.Mmpd.Core.Persistence: Warning

The log levels can be configured based on the logger name which appears in the "logger" field in the json.

You can configure a specific log or a group of loggers. For instance in the above example everything that starts with Bosch.Industry.Mdp.Mmpd.Core.Persistence will be on Warning level, but the Bosch.Industry.Mdp.Mmpd.Core.Persistence.EntityFramework.EFDbAdapter will be on Information level

Valid log level values

Trace
Debug
Information
Warning
Error
Critical
None

The configuration values are different than the level that appears in the log output. For instance the information levels is configured as Information but appears as INFO in the log

Specific logging Messages

Insufficient resources allocated to em service

{"logger":"Microsoft.AspNetCore.Server.Kestrel","loglevel":"WARN","message":"As of "01/29/2023 14:29:57 00:00", the heartbeat has been running for "00:00:01.7023439" which is longer than "00:00:01". This could be caused by thread pool starvation."}

{"logger":"Microsoft.EntityFrameworkCore.Query","loglevel":"ERROR","message":"An exception occurred while iterating over the results of a query for context type 'Bosch.Industry.Mdp.Mmpd.EquipmentManagementService.Persistence.EntityFramework.Contexts.EquipmentOracleContext'. Oracle.ManagedDataAccess.Client.OracleException (0x80004005): Connection request timed out   at OracleInternal.ConnectionPool.PoolManager`3.Get..."}

These log entries usually appear when there are too many concurrent users and the allocated CPU is not able to handle the load.

The first entry (heartbeat has been running … thread pool starvation) explicitly signals this situation. Even if it seems not related, the second entry (OracleInternal.ConnectionPool.PoolManager) also appears in this scenario.

MACMA errors

Frequent errors or warnings from MacmaAclProvider similar to the ones below usually show a problem on the MACMA module. A restart of this module usually solves the provlem

{"logger":"Bosch.Foundation.OpenId.Authorization.Macma.MacmaAclProvider","loglevel":"ERROR","message":"Error occured while getting ACL: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. --- System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.  at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) ----.. at Bosch.Foundation.OpenId.Authorization.Macma.MacmaAclProvider.GetAclInternal(String aclAuthenticationToken)"}

{"logger":"Bosch.Foundation.OpenId.Authorization.Macma.MacmaAclProvider","loglevel":"WARN","message":"Could not update ACLs."}