Influx database migration
In the course of the split of the Condition Process Monitoring (CPM) module into Condition Monitoring (CM) and Process Quality (PQ) the existing CPM data in the Influx database needs to be migrated to a new Influx database for each respective module.
This explains the CPM to PQM data migration. For CPM to CM data migration, please check the CM migration docs.
To migrate the PQM Influx data use the following guides:
For more details on how to use the Influx data migration tool see Appendix: influx migration tool
Prerequisites
-
Prepare the new Influx Databases
-
Clarify with the business which data needs to be migrated
-
Should all data be migrated or just the data after a specific date?
-
Should the original data be deleted after the migration?
-
-
Inspect the retention policies in the existing Influx DB and their disk sizes
-
What is a suitable chunk size for migrating the data so that we don’t run out of memory?
-
-
Derive the fitting parameters for the migration tool (see below for details) and create a migration plan according to your previous findings
-
Install Java JDK 21 on the machine on which you want to execute the migration tool
-
Add the needed Certificates in the Java KeyStore (optional if 'disable-ssl' option is enabled).
-
The JAR file can be found in artifactory (use the newest released version):
| Make sure that there is a valid and stable connection between the Source, Target and the host where the Influx Migration Application is running. |
The migration mainly depends on 3 variables: the data size per Influx Point that is to be migrated, the query-chunk-size and the quality of the HTTP connection between the Source, Target and host where the Influx Migration application is running.
Therefor there is no golden value for setting the query-chunk-size; it all depends on how much data there is to migrate and how fast and reliable the HTTP connection is.
|
Please be advised that K9s’s port forwarding feature proved to be very unreliable during our tests.
If you rely on this feature to do the migration, it is a high chance that you will encounter problems.
However, there is a workaround by setting ok-http-logging-interceptor-level to BODY.
This will result in lots of output information, so grep for errors to make sure the migration ran successfully.
We advise to find a better solution and DO NOT rely on this workaround.
|
Migrating processes
Processes will have a very large data size per Influx point as they contain blob data.
Here we need a small query-chunk-size, so the migration process will take longer.
If the migration is done within a trusted network, the --disable-ssl option can be used in order to trust the certificates of the Source and Target. This implies that there will be no need to add the self-signed certificates to the Java TrustStore.
During our test phase, we used the following query to migrate processes (it excludes rp_event, among others, to only migrate rp_processes):
java -jar influx-migration-<release_version>.jar \
--tenant-id=7311ea8c-5d48-43fe-acf9-980eedf24b6c \
--query-chunk-size=2500 \
--source-url=https://si0vmc3101.de.bosch.com:8086 \
--source-db=cpm \
--source-username=cpm \
--source-password=aklnnNBUIf-efekla \
--target-url=http://localhost:8086 \
--target-db=cpm \
--target-username=admin \
--target-password=admin_password \
--skip-continuous-queries=true \
--source-retention-exclusions=autogen,rp_msm_deferred,rp_msm_level1,rp_msm_level2,rp_msm_level3,rp_msm_raw,rp_ppmp_meta_data,rp_eventGeneral notes
-
This tool streams the Influx points into memory and then writes them to the target DB. So make sure that you have good internet connection between the source host, the host where you execute the tool and the target host.
-
This tool does NOT create, delete, or modify rights to a specific database in any way
-
As continuous queries do not belong to a retention policy, they will either all be copied or none (use
--skip-continuous-queries) -
This tool does NOT delete any Continuous Queries whatsoever. The reason for this is that not all data is guaranteed to be moved to the new database, and then the Continuous Query might still be relevant to the remaining data in the original database.
-
One of the stretch goals for this tool was to be able to continue the migration at a later stage (to continue where it was previously stopped). Although this was not specifically implemented, the tool will not duplicate any Data, Measurements, Retention Policies, Continuous Queries or anything else when it is run again with the same parameters.
-
BE CAREFUL when choosing to add the
--delete-sourcetag. The results are permanent and immediate. (It has been set to default=false to prevent this accidentally happening) -
ALWAYS check and double-check the results, especially Retention Policies and Continuous Queries that were created.
-
The tool will output the number of points copied per chunk to give a sense of progress. You can use a count query to know how many points to expect before starting the migration tool
-
If the tool doesn’t output progress for some time (> 30s) and seems stuck, hit
CTRL-Cand decrease the chunk size
with--query-chunk-size
-
-
Migrate processes, see Migrating processes
-
Figure out which retention policies you do not want to migrate and add then use
--source-retention-exclusionsto exclude them -
As PQM doesn’t use/need any continuous queries, they can always be excluded using
--skip-continuous-queries -
If the tool crashes at some time, you can run it again - existing points will just be overridden. If you know what data is missing, you can tweak it with the
--from-timestampand--to-timestampoptions. -
There might be some points that run out of retention when writing them into the database!
Using the migration tool
The provided migration tool can be run via the command line.
Prerequisite: Java 21 installed.
The following parameters needs to be provided and should be specified before running the tool:
Source Database Configuration
| Parameter | Required | Description | Example |
|---|---|---|---|
source-db |
Name of the migration source database |
cpm |
|
source-url |
URL to the migration source Influx DB instance |
https://rngvmc0129.de.bosch.com:8086 |
|
source-username |
Username to read data from source database. Recommendation: use user with admin rights. |
admin |
|
source-password |
Password to the respective username to read data from source database |
influx-db-admin-secret! |
Target Database Configuration
| Parameter | Required | Description | Example |
|---|---|---|---|
target-db |
Name of the migration target database |
pq |
|
target-url |
URL to the migration target Influx DB instance |
https://rngvmc0129.de.bosch.com:8086 |
|
target-username |
Username to write data to target database. Recommendation: use user with admin rights. |
admin |
|
target-password |
Password to the respective username to write data to target database |
influx-db-admin-secret! |
Migration Related Parameters
| Parameter | Required | Description | Default | Example |
|---|---|---|---|---|
tenant-id |
Id of the tenant whose data should be migrated |
- |
7311ea8c-5d48-43fe-acf9-980eedf24b6c |
|
query-chunk-size |
The number of data points to be migrated at the same time. |
10000 |
10000 |
|
source-retention-exclusions |
Comma-separated list of retention policies to be excluded from migration.If not set or empty, will migrate all retention policies |
- |
rp_process, rp_event |
|
skip-continuous-queries |
Whether the continuous queries shall be skipped |
false |
true |
|
from-timestamp |
Only data newer than this UTC ISO timestamp will be migrated |
- |
2022-11-30T11:09:35+01:00 |
|
to-timestamp |
Only data up until this timestamp will be migrated |
The UTC timestamp when the migration tool was started |
2022-11-30T12:09:35+01:00 |
|
delete-source |
Whether the Source data should be deleted or not |
false |
false |
|
disable-ssl |
Whether to disable the SSL certificate checks |
false |
true |
|
ok-http-logging-interceptor-level |
Sets the logging level of the OK HTTP Client. Accepted values: NONE, BASIC, HEADERS or BODY |
NE |
BODY |
|
ok-http-read-timeout |
OK HTTP Client read timeout in seconds |
30 |
30 |
|
ok-http-write-timeout |
OK HTTP Client write timeout in seconds |
30 |
30 |
To get the help of all options:
java -jar influx-migration-<version>.jar --helpExample Run in Command Line:
java -jar influx-migration-0.0.1-SNAPSHOT.jar \
--source-url=https://rngvmc0129.de.bosch.com:8086 \
--source-db=cpm \
--source-username=admin \
--source-password=pool31-admin-influxdb-secret! \
--target-url=https://rngvmc0129.de.bosch.com:8086 \
--target-db=cpm_migration \
--target-username=admin \
--target-password=pool31-admin-influxdb-secret! \
--tenant-id=7311ea8c-5d48-43fe-acf9-980eedf24b6c \
--query-chunk-size=50 \
--source-retention-exclusions=rp_process,rp_msm_raw \
--from-timestamp=2022-11-30T11:09:35+01:00 \
--delete-source=falseWith the above parameters, the tool will use the SSL protocol to establish the connections to both Source and Target and it will create all retention policies of the source database 'cpm' except 'rp_process' and 'rp_msm_raw' on the target database 'cpm_migration'. Next, it will copy all data newer than 11/30/2022 9:35:01 from all retention policies except 'rp_process' and 'rp_msm_raw' from database 'cpm' to database 'cpm_migration' in batches of 50 data points each. It will also create Continuous Queries on 'cpm_migration' that existed on 'cpm' and are related to any but the excluded retention policies, and will NOT delete the original source data