ELK stack
One solution that can collect and centralize logs, traces, and metrics from all the modules deployed in one Nexeed IAS is the ELK Stack (https://www.elastic.co/what-is/elk-stack).
The ELK Stack is a collection of three open source projects: Elasticsearch, Logstash, and Kibana, all developed, managed and maintained by Elastic.
-
Elasticsearch - is an open source, full-text search and analysis engine based on Apache Lucene search engine.
-
Logstash - is a log aggregator that collects data from various input sources, execute different transformations and enhancements and than ship the data to various supported output destinations (in our case Elasticsearch).
-
Kibana - is a visualization layer that works on top of Elasticsearch, providing users with the ability to analyze and visualize the data.
-
Beats - last but not least beats are lightweight agents that are installed on hosts to collect logs, traces and metrics and for forwarding it into the stack.
General considerations
To collect logs and metrics from all Nexeed modules, Elastic beats have to be deployed on each node of the kubernetes cluster. The following beats will be deployed as daemonsets:
-
filebeat
-
metricbeat
Deploying these beats as containers inside kubernetes imply some additional configuration items to be aware of. The full details are in the documentation “run filebeat / metricbeat inside kubernetes” but in short:
-
you need to mount both the host machine’s proc filesystem and the cgroup filesystem inside of the container for metricbeat
-
for filebeat you will need to mount /var/lib/docker/containers and /var/log inside container
ELK Cluster can be self hosted on premise or can be used in cloud as a service from Elastic. Of course the main advantage of using it as a service is that the operational aspects of the infrastructure will be managed by Elastic and you will only need to configure and deploy the beats that will collect and send data.
A helm chart for monitoring with the ELK stack is available and can be used. It supports logging to elastic cloud service and deploy metricbeat, filebeat and heartbeat inside the cluster. To gather additional information from kubernetes API we also deploy kube-state-metrics. Heartbeat is used to monitor periodically a configured list of endpoints and based on this information we can trigger alerts when these endpoints are down. For metricbeat the chart also creates a deployment (running alongside metricbeat daemonset) which will gather some metrics for MSSQL, RabbitMQ and query the state of kubernetes cluster (like events and resource status). To properly configure this chart you will need to set the following global variables during deployment:
| Name | Description |
|---|---|
|
Set to |
|
The elastic cloud ID. |
|
The elastic cloud Auth (username and password separated by colon) |
|
It configures the output used for the chart and can be elastic cloud. The value for this variable should be set to: |
|
The value of this variable is added on each document ingested to elastic in |
As a local variable you should supply the value for deviceportalGlobalHealthCheckPassword which is the password for device portal open search module health check (for user health_check).
This monitoring chart is delivered as is and is not covered by support policy of Nexeed application.