How to synchronize devices and machines to OpenSearch

Be aware that this How-To requires having running instances of OpenSearch and OpenSearch Dashboards in place. Using the following endpoints allows verifying the health of an OpenSearch deployment:

OpenSearch: https://{url-to-service}/_cluster/health/?level=indices
OpenSearch Dashboards: https://{url-to-service}/app/home#/

Because of performance reasons, device and machine data is queried from OpenSearch. Therefore, it is necessary to synchronize the device and machine data using the corresponding endpoints provided by the Device Master Data Management service and the Device Monitoring service. This is necessary in the following cases:

OpenSearch has been set up in a new environment ( initial sync)
Desaster Recovery
The database and OpenSearch are out of sync ( amount of devices on database != amount of entries on index), further reading: Device Portal data is out of sync

Synchronize master data of devices and machines

POST <device-master-data-mgmt-base-URL>/operation/v1/synchronize/devicesToOpenSearch
or
POST <device-master-data-mgmt-base-URL>/operation/v1/synchronize/machinesToOpenSearch
{
    "itemCountPerBatch": 400,
    "concurrentBatchCount": 5,
    "retryCount": 1,
    "batchOffset": 0,
    "batchCount": 500
}

Synchronize multiple times

Depending on the amount of Devices or Machines in an environment and the chosen parameters, it’s necessary to execute the request multiple times with an adjusted batch offset.

Example: The first synchronization request returns the following response:

{
    "totalItemCount": 200000,
    "totalBatchCount": 1251,
    "updatedItemCount": 200000,
    "failedItemCount": 0
}

⇒ As we chose a batchCount of 500, and there are 1251 batches in sum, we need to send two more requests with an adjusted batchOffset:

{
    "itemCountPerBatch": 400,
    "concurrentBatchCount": 5,
    "retryCount": 1,
    "batchOffset": 0,
    "batchCount": 500
}

{
    "itemCountPerBatch": 400,
    "concurrentBatchCount": 5,
    "retryCount": 1,
    "batchOffset": 1000, # Now performing updates for batches from 1000 to 1251
    "batchCount": 300
}

Amount of concurrency

The value in the sample are "optimized" for the Nexeed production environment. Depending on the environment, it might make sense to increase or decrease concurrency.

In case the request returns an erroneous response, e.g.

{
    "timestamp": 1646721865636,
    "path": "/synchronize/v1/devicesToOpenSearch",,
    "status": 500,
    "error": "Internal Server Error",
    "message": "Retries exhausted: 1/1",
    "requestId": "149aeb12-11"
},

it is necessary to decrease the amount of concurrency.

Hint: As the endpoint is implemented with a "create-or-update" semantic, there’s no need to adjust the batchOffset in the error case.

Synchronize device-machine relations

After the initial synchronization of the master data of devices and machines it is necessary to also synchronize the device-machine relations (machineId property in devices index).

Changes regarding device-machine relations are synchronized automatically. But in case of inconsistencies this endpoint provided by the Device Master Data Management service can be used to correct the device-machine relations.

POST <device-master-data-mgmt-base-URL>/operation/v1/synchronize/deviceMachineRelationsToOpenSearch

Synchronize status data of devices

Changes regarding status data of devices are synchronized automatically. But in case of inconsistencies the following endpoint provided by the Device Monitoring service can be used to correct the status data of devices.

POST <device-monitoring-base-URL>/operation/v1/synchronize/searchService
{
    "deviceCountPerBatch": 400,
    "concurrentBatchCount": 5,
    "limitQueriesPerSecond": 3,
    "limitRequestsPerSecond": 3,
    "createdAfterMs": -1  // means all
}