WPS Database Importer icon

WPS Database Importer

Import data from Kingsoft WPS output to MySQL database with field mapping and deduplication
Join our community

Overview

This node imports data from Kingsoft WPS output into a MySQL database table. It supports two import modes: full import, which clears the target table before importing all data, and incremental import, which imports only new records based on a unique key combination. The node processes data in batches, supports field mapping, and can verify the import count to ensure data integrity. It is useful for automating data transfer from WPS exports to MySQL databases, handling large datasets efficiently, and avoiding duplicate entries during incremental updates.

Use Case Examples

  1. Full import of a complete dataset into a MySQL table, clearing existing data first.
  2. Incremental import of new records into a MySQL table based on unique keys to avoid duplicates.
  3. Batch processing large datasets with configurable batch size for performance optimization.

Properties

Name Meaning
Import Mode Selects the import mode: full import clears the target table and imports all data; incremental import imports only new records based on unique keys.
Target Table Name of the target MySQL database table to import data into, supports database.table format.
Batch Size Number of records to import in each batch, configurable between 100 and 10,000.
Import Fields Comma-separated list of fields to import from the source data, specifying which columns to map.
Unique Key Fields Comma-separated list of target field names that form the unique key for incremental import to identify new records.
Options Additional import options including whether to truncate the table before full import, continue on error during batch processing, and verify that the number of imported records matches the source data.

Output

JSON

  • success - Indicates if the import operation was successful.
  • totalRows - Total number of rows in the source data to be imported.
  • importedRows - Number of rows successfully imported into the database.
  • duplicateRows - Number of duplicate rows detected and skipped during incremental import.
  • failedRows - Number of rows that failed to import.
  • message - Summary message detailing the import results including counts and performance.

Dependencies

  • MySQL database connection using credentials with host, port, user, password, and optional database name.

Troubleshooting

  • Ensure the source data has at least a header row and one data row; otherwise, the node throws an error.
  • For incremental import, at least one unique key field must be specified; missing this causes an error.
  • Batch insert failures may occur due to database constraints or connectivity issues; enabling 'Continue on Error' allows processing to continue despite batch failures.
  • Mismatch in import count verification may indicate data issues or partial imports; check logs for detailed warnings.

Discussion