Start and shutdown

Deployment/startup process overview

Rework Control services are deployed via Helm as dependencies for the Nexeed umbrella chart.
The Docker containers for Rework Control services are created and started automatically once they have been deployed on a Kubernetes cluster.
All services are deployed as Kubernetes deployments with a minimum replica count of 2 and with max count of 5, by default. In case different values are needed, they can be overridden by setting the corresponding values in the umbrella chart’s custom-values.yaml file. Exampe is given below.

<instance-id>:
  deployments:
    [rework-backend-service|rework-web-service|rework-gateway-service|rework-control-core-backend-service|rework-control-core-web-service]:
        replicaCount: # default replica count
        autoscale:
            minReplicas: # min replica count
            maxReplicas: # max replica count
            targetCPUUtilizationPercentage: # target CPU utilization percentage
            targetMemoryUtilizationPercentage: # target memory utilization percentage
            targetMemoryUsage: # target memory usage (absolute value)

The web service will also register its web interface with the Nexeed Web Portal that has been installed via the umbrella chart.
The services have built-in health checks that show a “healthy” status when it is up and running.
Managing the life cycle (and other aspects) of Rework Control is done by connecting to the cluster where the umbrella chart has been installed using, for example, k9s utility.

Startup dependencies

rework/rework-backend-service: Readiness is controlled by the availability (being up and running) of the modules marked as Required shown in the System Requirements section for the backend service.
rework/rework-web-service: Readiness is controlled by the availability of the modules marked as Required shown in the System Requirements section for the web service.
bci-app/nginx: Readiness is independent of all other services.
rework/rework-control-core-backend-service: Readiness is controlled by the availability (being up and running) of the modules marked as Required shown in the System Requirements section for the backend service.
rework/rework-control-core-web-service: Readiness is controlled by the availability of the modules marked as Required shown in the System Requirements section for the web service.

Shutdown

Killing the services pods will lead them to auto restart to fulfill the replica count requirement. Therefore, to completely stop a pod and prevent it from auto restarting, one can scale down the corresponding deployment to have 0 replicas using the following command:

kubectl scale --replicas=0 -n rework deployment/<deployment-name>

where <deployment-name> is one of rework-backend-service-deployment, rework-web-service-deployment, rework-gateway-service-deployment, rework-control-core-backend-service or rework-control-core-web-service.

Shutdown timeout

When a pod is initiated to be stopped, the system will send a SIGTERM signal to the .NET application, and wait for 30 seconds for it to stop. It will send a SIGKILL in case the shutdown could not finish in time. The .NET application is configured to have 28(which is configurable) seconds of grace period for shutting down, so that the SIGKILL does not happen while the framework would stop itself. The application is logging the STOPPING and STOPPED events, and the latter will contain the exit code as well.

Possible exit codes for Rework Control Core: * Exit code 0: the application could shut itself down during the grace period * Exit code 1: This exit code is used when at least one background process fails to shut down within the expected timeframe. It helps identify cases where resource cleanup did not complete successfully. * Exit code 2: there was an error during shutdown, or the processes could not finish during the grace period

Configuration of shutdown timeout

rework:
  deployments:
    rework-backend-service:
      env:
        HostSettings__ShutdownTimeoutSeconds: '28' # Host shutdown timeout(seconds)
      ...
    rework-web-service:
      env:
        HostSettings__ShutdownTimeoutSeconds: '28' # Host shutdown timeout(seconds)
      ...
    rework-control-core-backend-service:
      env:
        NEXEED_REWORK_CONTROL_CORE_HostSettings__ShutdownTimeoutSeconds: '28'
      ...
    rework-control-core-web-service:
      env:
        NEXEED_REWORK_CONTROL_CORE_WEBSERVICE_HostSettings__ShutdownTimeoutSeconds: '28'
      ...