Ansible operator troubleshooting

Our Kubernetes operators are made from Ansible Operator Plugin. It honours the run time environment variables for ansible.

The ansible operators first checks the health status of the module to be managed on startup, and then rely on watching custom resources in the selected namespaces (see CustomResourceDefinitions for which CR it watches) to perform management activities.

From here, we already have a few things to initiate the check: * Check ingress-nginx controller status (Check deployment, service and pod status), assuming it is installed under ingress-nginx namespace: - kubectl get -n ingress-nginx deployment/ingress-nginx-controller -o yaml - kubectl get -n ingress-nginx pod Check the pod status - kubectl get -n ingress-nginx svc/ingress-nginx-controller Check cluster ip and external-ip (on cloud environment) - kubectl logs -n ingress-nginx pod/<pod-name> -f To follow the nginx logs * Check DNS setup of the domain and if applicable, the subdomains. * Check if the managed component (i.e. macma, rabbitmq) is properly up and running * Check admin credentials for the managed component under the serverInstances section of the custom-values.yaml file, i.e. rabbitmq management plugin, mssql admin account credentials etc.

If all above checks passed, you can also check on the following items: * The status section of the Custom Resource, i.e. if status section is missing * Ansible operator pod logs * Increase ansible output verbosity (0 to 5), there are multiple ways of doing this: - adding ANSIBLE_VERBOSITY variable to the ansible operator deployment - adding annotation to the ansible-operator deployment in the aops namespace: kubectl annotate -n aops deployment/<ansible-operator-deployment-name> ansible.sdk.operatorframework.io/verbosity="<0-5>" * If the deployment is a split deployment, please check if your custom-values.yaml file is matching the setup requirement of the SplitDeployments section * If in a split deployment, check your module’s namespace is within the global.modules.ansible-operator.additionalNamespaceToWatch section in the custom-values.yaml file * The watches configmap in the aops namespace, here you may change the safe_logs parameter to "false" for revealing the ansible-operator runtime credentials