Data migration & synchronization

Introduction

The Synchronization service enables MDM to support the BMLP hybrid mode with one "new" MDM installation synchronized to one or more "old" MES MMPD installations.

Concepts & terminology

MDM: This is the "new" Master Data Management, which supports either OnPremise or Cloud installation, and is actively developed by the MDM team.

MMPD: This is the "old" MES MMPD hosted OnPremise, which is not actively developed anymore.

IAS: The Nexeed Industrial Application System which uses the MDM "new" services.

BMLP: The hybrid setup which uses the new MDM services together with the Mapping & Synchronization services.

MES: The Manufacturing Execution System which hosts the "old" MMPD services and Oracle database.

Synchronization Service: A Windows Service that ensures data transfer between the MDM and MMPD systems.

Instance: An installation of the Synchronization Service on an MES system. It is connected to the MMPD database and allows it to be synchronized to MDM.

System Id: A string that uniquely identifies a MES MMPD instance in a BMLP hybrid setup. It is defined in each Configuration on the Data Migration tab. It is visible on facilities (SystemId identifier) and on processes and process groups (Origin field).

Overview

The Synchronization service is installed on the MES Windows Server. It is connecting to the MDM Service APIs and RabbitMQ on the BMLP side and directly to the Oracle MMPD database on the MES side.

Features

In the BMLP operation mode MDM supports the following:

Initial data transfer: Existing data from an old MMPD database is imported into an empty MDM installation.
- Re-running the initial transfer:
  - Data objects already transferred: no further changes in the MMPD database can be carried over to MDM
  - New data objects or not already transferred: New data will be automatically detected and transferred to MDM
Synchronization: The changes in the MDM system are propagated to the MMPD database.

One MDM installation can be connected to multiple MMPD databases. As MDM supports more tenants there are multiple scenarios which can be mixed in a BMLP installation:

Each MMPD DB can be mapped to a different MDM tenant
More MMPD DB can be combined into the same MDM tenant

Workflow

The following steps have to be performed in order to setup and operate the MES synchronization in the BMLP scenario:

On MES System

Install Synchronization service package on the MES system via MES installer
Configure Synchronization Service:
1. Connection details to BMLP: MDM service url, client id and secret with permissions for MDM, RabbitMQ connection details
2. Oracle connection string
Start Synchronization service

On IAS Portal open Master Data / Equipment ⇒ Data Migration tab

Check that the MES instance appears on the Instances page and the it is Online (green status)
Use the Configurations tab to define OpConLocation string mapping between MDM and MMPD
Use the Jobs tab to perform initial data transfer
1. Running a job will migrate data from MMPD to MDM according to the selected configuration
2. A job can be re-run / edited / configuration changed until all the desired data is migrated
Visually check data in MDM UI
Enable synchronization mode if required
1. MDM to MMPD Synchronization runs in background

Setup & configuration

Service deployment

The service is provided as an MES package which needs to be installed on the MES Windows server: SyncService_xxxxxx.mespkg

It can be acquired from the official MES distribution or it can be downloaded from the artifacts section of the MDM ADO Build Pipeline:

The mespkg package name includes the build number which can be used to identify the corresponding BMLP release.

After the installation there will be a service installed in the \OPCON folder and visible in the Services window:

System & connectivity requirements

System requirements:

Windows Server OS, 16 GB RAM
.NET Core 8.0+

The Synchronization service needs to have access to the following resources:

Oracle MMPD database
- At least for the first startup the service needs to be able to create tables in the MMPD schema (it will create the SYNC_MAPPING table on startup)
- During normal operation just data read / write permissions are enough
MDM services API
- A client id / client secret with access to all the tenants that will be used in the synchronization
- The client needs to be assigned Equipment Administrator and Synchronization Manager roles in all the used tenants
  - NOTE: The BMLP MDM client does not have this roles by default
RabbitMQ
- The MDM user or another user with access to: x.nexeed.integration, x.mdm.synchronization, q.mdm.synchronization.management
- x.nexeed.integration must be existing before service startup
- x.mdm.synchronization, q.mdm.synchronization.management are automatically created at service startup

Service configuration

The service is configured via the Opcon\Config folder. There are two files that need to be configured:

Opcon.Settings.xml - Base port number (optional: HTTPS Certificates, OIDC (MACMA), RabbitMq)
appsettings.json - Instance configuration, Service urls, HTTPS Certificates, OIDC (MACMA), RabbitMq

Some settings can be configured in any of the files, and others just in a specific file - see the tables below. If the same option is configured in both files, the appsettings.json takes precedence over the OpCon.Settings.xml file.

Instance configuration parameters

The synchronization is performed between a MDM installation and one or more instances of MMPD. For each of these instances, the following set of parameters needs to be defined:

Configuration

Description

Instance Name

This is the name of the MMPD instance. The name should be unique for all instances connected to the same MDM installation. The name will identify the instance on the MDM "Data Migration" Tab in the MDM UI.

TenantId

The MDM tenant where the data will be transferred. The instnace will be registered on this tenant and will apear on the UI only on this tenant.

Connection string

The MMPD Oracle database connection string with the existing data

Synchronization service configuration

The following table provides an overview of the configuration parameters.

Opcon.settings.xml

Setting name

Description

Foundation.BasePort

Base port number. If that port is already in use, the next higher free port number is used. Default is 59710.

Appsettings.json

Setting name

Description

Certificates.HTTPS.*

The SSL certificate to use if running on HTTPS

HttpClientProvider.AllowInvalidServerCertificate

If true it will not validate certificates when connecting to Equipment or Mapping services

SynchronizationConfiguration.EquipmentManagementServiceReference, SynchronizationConfiguration.ProcessServiceReference

The urls to the Equipment and Process services

SynchronizationConfiguration.Instances.<InstanceName>

Add a new property for each new instance. The name will be visible on the MDM UI

SynchronizationConfiguration.Instances.<InstanceName>.MdmTenantId

The MDM tenantId for the instance <InstanceName>

SynchronizationConfiguration.Instances.<InstanceName>.ConnectionString

The connection string to the MMPD Oracle database for the instance <InstanceName>

SynchronizationConfiguration.AllowSynchronization

If true then the syncronization from MDM to MMPD can be enabled for this instance. If false, it cannot be enabled, even if the user chooses to do so on the UI. Use this setting if you want to make sure that only the initial transfer can be performed, and no backward transfer (MDM ⇒ MMPD) is possible.

Messaging.Connector.*

The connection to RabbitMQ

OIDC.*

OIDC.NamedHttpClients.Clients.EquipmentClientService.*

The Url of the MDM Macma authorization endpoint. ClientId, ClientSecret, and Scope (aud:macma_client_id) used to request tokens Because of technical limitations some of these parameters have to be entered twice

Sample configuration files:

OpCon.Settings.xml (recommended)

    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
    <configSections>
        <section name="appSettings" type="System.Configuration.AppSettingsSection, System.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"></section>
    </configSections>
    <appSettings>
        <add key="Foundation.BasePort" value="59710" />
    </appSettings>
    </configuration>

appsettings.json (recommended)

    {
    "HttpClientProvider": {
        "AllowInvalidServerCertificate": true
    },
    "Certificates": {
        "HTTPS": {
        "Source": "Store",
        "StoreLocation": "LocalMachine",
        "StoreName": "My",
        "Subject": "CN=localhost",
        "AllowInvalid": true
        },
    },
    "SynchronizationConfiguration": {
        "EquipmentManagementServiceReference": {
          "Url": "https://space-1.devspace-a.bosch-nexeed.com/mdm/equipment-management/"
        },
        "ProcessServiceReference": {
          "Url": "https://space-1.devspace-a.bosch-nexeed.com/mdm/process/"
        },
        "Instances": {
          "My MES FeP": {
            "ConnectionString": "Data Source=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ATMO1ORA12.de.bosch.com)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=ATMO1ORA12.de.bosch.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=TestO12c)));User Id=user;Password=pass;",
            "MdmTenantId": "7311ea8c-5d48-43fe-acf9-980eedf24b6c"
          },
          "DEVTEST2": {
            "ConnectionString": "Data Source=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ATMO1ORA12.de.bosch.com)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=ATMO1ORA12.de.bosch.com)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=TestO12c)));User Id=DEVTEST2;Password=pass;",
            "MdmTenantId": "12341234-1111-2222-3333-1234567890ab"
          }
        }
    },
    "OIDC": {
        "ServiceUrl": "https://space-1.devspace-a.bosch-nexeed.com/iam",
        "ClientId": "",
        "ClientSecret": "",
        "NamedHttpClients": {
            "Clients": {
                "EquipmentClientService": {
                    "TokenEndpoint": "https://space-1.devspace-a.bosch-nexeed.com/iam/access-management/v1/tenants/7311ea8c-5d48-43fe-acf9-980eedf24b6c/openid-connect/token",
                    "Scopes": [ "aud:" ]
                }
            }
        }
    },
    "MessagingEnabled": "true",
    "Messaging": {
      "Connector": {
        "VirtualHost": "/",
        "Hostname": "space-1.devspace-a.bosch-nexeed.com",
        "Port": 5672,
        "UserName": "mmpd",
        "Password": "themmpdpass",
        "SslEnabled": false,
        "SslVersion": "Tls12",
        "UseFailover": false,
        "IsAutoRecovered": true
      }
    }
    }

OpCon.Settings.xml (all supported settings)

    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
    <configSections>
        <section name="appSettings" type="System.Configuration.AppSettingsSection, System.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"></section>
        <section name="opcon.connections" type="Bosch.OpCon.Configuration.ConnectionStrings.ConnectionsSection, OpCon.Configuration, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null" />
    </configSections>
    <appSettings>
        <add key="Foundation.BasePort" value="59710" />
        <add key="Foundation.SystemId" value="urn:com:bosch:nexeed:systemtype:mmpd" />
        <add key="Common.Certificates.HTTPS.Source" value="Store" />
        <add key="Common.Certificates.HTTPS.StoreLocation" value="LocalMachine" />
        <add key="Common.Certificates.HTTPS.StoreName" value="My" />
        <add key="Common.Certificates.HTTPS.Subject" value="CN=localhost" />
        <add key="Common.Certificates.HTTPS.AllowInvalid" value="true" />
        <add key="Common.HttpClientProvider.AllowInvalidServerCertificate" value="true" />
        <add key="Common.SynchronizationConfiguration.EquipmentManagementServiceReference.Url" value="https://space-1.devspace-a.bosch-nexeed.com/mdm/equipment-management/" />
        <add key="Common.SynchronizationConfiguration.ProcessServiceReference.Url" value="https://space-1.devspace-a.bosch-nexeed.com/mdm/process/" />
        <add key="Common.Messaging.Connector.Hostname" value="space-1.devspace-a.bosch-nexeed.com" />
        <add key="Common.Messaging.Connector.VirtualHost" value="/" />
        <add key="Common.Messaging.Connector.UserName" value="admin" />
        <add key="Common.Messaging.Connector.Password" value="..." />
        <add key="Common.OIDC.ClientID" value="nrv5pf7bvrf1mic3smun4ytm" />
        <add key="Common.OIDC.ClientSecret" value="..." />
        <add key="Common.OIDC.NamedHttpClients.EquipmentClientService.ClientID" value="nrv5pf7bvrf1mic3smun4ytm" />
        <add key="Common.OIDC.NamedHttpClients.EquipmentClientService.ClientSecret" value="..." />
        <add key="Common.OIDC.NamedHttpClients.EquipmentClientService.TokenEndpoint" value="https://space-1.devspace-a.bosch-nexeed.com/iam/access-management/v1/tenants/7311ea8c-5d48-43fe-acf9-980eedf24b6c/openid-connect/token" />
        <add key="Common.OIDC.NamedHttpClients.EquipmentClientService.Scopes0" value="aud:nrv5pf7bvrf1mic3smun4ytm" />
    </appSettings>
    </configuration>

If the IAS deployment is using subdomains then all the IAS urls in the configuration should have the correct subdomain. For example:

https://iam.space-1.devspace-a.bosch-nexeed.com/iam/auth/realms/7311ea8c-5d48-43fe-acf9-980eedf24b6c
https://mdm.space-1.devspace-a.bosch-nexeed.com/mdm/equipment-management/
https://mdm.space-1.devspace-a.bosch-nexeed.com/mdm/process/

Service operation

The Synchronization service can perform two operations:

Initial data transfer from MMPD to MDM: this is started from the Jobs tab on the UI
Data synchronization from MDM to MMPD: this runs in the background only for the configurations where it is enabled. It can be monitored in the Synchronization Logs tab in the UI

Workflow

The synchronization is designed to run in two stages:

An initial data transfer must be performed first (transfer MMPD ⇒ MDM)
- via the Jobs tab
- multiple transfers can be run until all the required data is transferred
Only after the initial transfer has been completed the synchronization can be enabled (transfer MDM ⇒ MMPD)
- Enable according to your usecase - this might not be needed, only the initial transfer might be required
- If needed the synchronization can be enabled / disabled for each configuration
- It can be monitored on the Synchronization log tab

These operations are managed and configured form the IAS Portal on Master Data Management / Data Migration tab. This tab will display all the synchronization service instances that are properly configured and can connect to IAS. Each instance is managed via the Configuration, Jobs, and Synchronization sub-tabs.

Instances

The Instances screen shows all the synchronization services that are active and have registered successfully.

Each instance is shown as one box with the relevant green / orange status:

Green status: synchronization service is configured properly. It can connect to MDM Equipment Management service and also to RabbitMQ
Green status that changes to orange in a few seconds: synchronization service connects to MDM Equipment Management service but not to RabbitMq
- Initial registration at synchronization service startup succeeded ⇒ instance appears in Data Migration screen with green status
- Service does not respond to refresh request ⇒ no RabbitMq connectionv ⇒ status changes to orange
Orange status: instance has been online recently, but not since the last refresh request
- If no successful registration withing 7 days, the instance will disappear from the Instances screen

Refresh: pressing the refresh button will request (via RabbitMq) to all instances to re-register to Equipment Management service. If the synchronization service receives the request it will call the Equipment service via Api call using the configured client id and secret. If both RabbitMQ and Api connection work the status will turn green.

It will take a few seconds for a refresh cycle to take place. The status might turn orange during this time.

A refresh is performed automatically each time the Data Migration tab is activated, just like pressing the Refresh button.

Each instance has its own configurations, jobs and logs. These are managed in the subtabs by selecting the instance card.

Configurations

The configurations are managed in the Configurations tab. Each configuration provides a set of rules to transfer data between MMPD and MDM.

Each configuration has a SystemId which will identify the data source in MDM. It will appear in MDM on each facility as the SystemId identifier.

Multiple configurations can be defined and each configuration can have multiple mapping rules defined. The user is free to organize them as it is more appropriate for each use case, for example:

one configuration for each line or station (or even area … as needed)
one configuration for line 1, another configuration for an area, another one for some stations, and a fourth configuration for all the rest of the lines
one single configuration mapping an entire plant
any mix form the above

For each configuration (SystemId) the application engineer must define a mapping between each LocationId in MMPD and the OpConLocationString in MDM. Only the mapped locations are transferred during the initial transfer and then synchronized.

Each mapping rule defines a set of Location string components that are to be replaced during the transfer.

Sample mapping:

Details

  [
    {
      "MmpdLocationId": {
          "plant": "0000",
          "product": "000",
          "area": "0"
      },
      "MdmLocationId": {
          "plant": "AAAA",
          "product": "DB1",
          "area": "0"
      }
    },
    {
      "MmpdLocationId": {
        "plant": "0000",
        "product": "000",
        "area": "0",
        "line": "0817"
      },
      "MdmLocationId": {
        "plant": "AAAA",
        "product": "DB2",
        "area": "9",
        "line": "0817"
      }
    }
  ]

The above mapping defines the following:

All locations under Plant 0000 / Product 000 / Area 0 in MMPD are transferred to MDM and the Plant / Product / Area are replaced with AAAA DB1 0
- If there would be an Area 1 or Product 010 these are not transferred as they are not included in the mapping
Plant / Product / Area / Line 000000000817 and everything under it from MMPD is transferred to MDM and OpCon is set to start with AAAADDB290817

As shown above when there are more matching definitions the most specific mapping is applied. This allows to define a generic rule for a group of locations and a more specific rule for some location sub-trees.

If there are conflicting rules in the configuration then an error will be displayed when saving.

Only the locations covered by the mapping rules are transferred:

At least a plant from MMPD needs to be mapped to another plant in MDM so that it and all children are transferred and synchronized after that.
There is no rule to say "Transfer all contents of MMPD to MDM". Each plant has to be mapped individually.
An empty mapping will be ignored

The supported names for the Location components: plant, product, area, line, station, stationindex, functionunit, workposition, toolposition.

Synchronization: If set to true, the backward synchronization (MDM → MMPD) will be enabled for the entites mapped in this configuration.

Remember there is also a SynchronizationConfiguration.AllowSynchronization setting at instance level in the appsettings file of the synchronization service (on the MMPD server). That one can be used as a global disable for backward MDM → MMPD synchronization, and the individual settings at configuration level will have no effect.

Jobs

Topology Configuration in IAS system (Equipment Topology UI) should be set to Legacy or Custom template before running the Initial Transfer jobs.

The jobs can be used to perform the initial data transfer from an existing MMPD instance into an empty MDM database or a new / empty tenant. A job progress can be monitored via the logs. Any errors will be shown in the logs and will remain saved in the database.

A job contains the following:

configuration to use: it defines the system id and mappings
parameters: what entities to import

In case of errors a job can be re-run as many times as needed until it completes successfully. There is the option to re-run the same job (the logs will be appended) or a new job can be created and run - the logs will be available in the second job. The result is the same.

If the initial transfer is interrupted then it can be re-run and will continue from where it left off and transfer the remaining data. So in the case of an error you should address the cause and then re-run the job.

Again we offer complete flexibility here and the user can choose how to use the jobs, when to re-run, when to edit and re-run, create another job and delete the failed one, etc.

Running the job again will append the logs and might be more difficult to follow. But if the first attempt failed then it is confortable to just re-run it.
Creating new jobs will keep distinct logs.
Deleting a job will erase also the logs.
More than one job can use the same configuration and even the same parameters - there is no restriction. Manage them as needed, delete old ones to avoid confusions, etc.

In order to avoid confusion we advise to keep one job / configuration, or one job / configuration + different parameters.

The initial data transfer is not designed nor tested with existing data in the target MDM database. It can lead to unpredictable results when transferring (and afterwards synchronizing) new data on top of existing data that does not come from an initial transfer process.

Various operations are possible but using them in an "unnatural" order might have strange results:

Example 1: Run a job, import some data successfully, then edit the configuration and change the System Id, then re-run the import ⇒ The job will not do anything, will appear as successful, but entities will remain with the old system id.

Example 2: Run a job, import some data successfully, then edit the configuration and change the mappings, then re-run the import ⇒ unpredictable results

Under exceptional circumstances the following scenarios can be attempted. These are assumed to be working but not officially required or tested:

Empty the MDM database and rerun the initial data transfer (due to errors or other reasons)
- The SYNC_MAPPING table on the source MMPD DB must also be cleaned or deleted. This table is automatically created and maintained during the initial transfer / sync operation and is not part of the MES schema.
Add new mappings and rerun the initial data transfer
- Supported only with completely separated data sets (eg. add a new mappings for locations that were never transferred before)
- Changing mappings after an initial data transfer will lead to unpredictable results

Synchronization MDM - MMPD

When a change is performed on the MDM side it generates an event in the RabbitMQ. This is picked up automatically by the synchronization service and the corresponding action is performed on the MMPD database. There is no user intervention needed.

Each synchronization action performed by the service will appear in the Synchronization Logs tab.

It is not needed to enable this feature. Enable it only if required.

If it is forbidden to synchronize data back to MMPD then use the option SynchronizationConfiguration.AllowSynchronization to globally disable this feature. This is defined in the MES server / Config folder / Synchronization service appsettings.json and cannot be overridden by using the Data Migration UI.

While the synchronization is running there must be an open connection to the q.mdm.synchronization.management queue in RabbitMQ. This could be checked in the Rabbit UI:

Maintenance and diagnostics

Logging

The synchronization service also log messages to the Windows Application logs

The logging is configured in:

\OpCon\Config\MMPD-SYNCHRONIZATION\OpCon.Diagnostics.Config.xml

The log can be renamed, enabled/disabled and the logging level can be configured. See the following sample:

Event log configuration - OpCon.Diagnostics.Config.xml

   <opcon.diagnostics application="MMPD-MAPPING">
     <eventTracing>
       <add name="EL1" endPoint="net.tcp://localhost:55089/Bosch.OpCon.MES.EventLogger.Service/EventTracingService" isolatedLog="true" logName="MMPD-MAPPING" enabled="true" level="2" activated="true" />
     </eventTracing>
     <messageLogging />
   </opcon.diagnostics>

Metrics

The synchronization service provides a Prometheus /metrics endpoint to monitor service status. For example http://fe0bci-mmpd1.de.bosch.com:59710/metrics

It includes the process statistics, http client statistics and also the number of MDM synchronization messages processed:

 # HELP mdm_sync_total_messages_processed The total number of messages processed.
 # TYPE mdm_sync_total_messages_processed counter
 mdm_sync_total_messages_processed 638
 # HELP mdm_sync_total_messages_received The total number of messages received.
 # TYPE mdm_sync_total_messages_received counter
 mdm_sync_total_messages_received 758