Article body
Full article
HENGSHI SENSE provides data integration and batch synchronization. Incremental extraction, output update strategies, execution plans, and run records help teams organize updates from data ingestion through analytical use. Recovery requires considering these settings together with source changes, target database capabilities, and reconciliation results.
After synchronization is interrupted, immediately rerunning it may introduce duplicates, while clearing the target table may affect downstream analysis. A reliable sequence is to identify the failed stage, establish what was already read and written, and then select an appropriate recovery method. Even after a task reports success, verify that its results meet business calculation requirements.
1. Distinguish read progress from write rules
Hengshi data integration supports input, transformation, and output nodes. Teams can filter, join, merge, and aggregate data before writing it to a target database. Batch synchronization is suited to strategies configured by connection, directory, and table. These entry points manage incremental state differently, so identify the actual project type before attempting recovery.
| Configuration object | Problem it addresses | Recovery considerations |
|---|---|---|
| Incremental fields | Determine which new data to read | Whether fields increase monotonically and where the progress value comes from |
| Output key fields | Identify business records already present in the target | Whether keys are complete and stable and distinguish genuinely separate fact rows |
| Table operations and update methods | Determine preservation, clearing, recreation, insertion, and updating | Effects of repeated execution on existing data and downstream pages |
| Execution plans and records | Organize update order and record results | Failure causes, dependencies, alerts, and review ownership |
2. Use incremental extraction correctly to avoid carrying omissions forward
Incremental fields, or their ordered combination, must increase monotonically. Data integration input nodes maintain the corresponding incremental state; batch synchronization obtains the maximum incremental value from the target table. When no incremental position exists on the first run, a full extraction is still required. Subsequent runs read new data that satisfies the incremental conditions.
Field-based incremental extraction needs particular attention to historical modifications, new records sharing a timestamp, and late-arriving data. If an old order’s amount changes without increasing the incremental field, the next extraction may miss the update. A creation-time field alone also does not automatically handle deletion. Design update timestamps, change tables, supplemental extraction, or full reconciliation according to how the source system changes.
Incremental extraction can reduce repeated reads, but actual savings depend on change volume, query indexes, and source database load; measure them using task records. Reading in batches can limit the amount read at once, but batching does not establish a general resumable-transfer capability across processes or data sources. Confirm the recovery position against the actual configuration and test results.
3. Manage repeated writes with Upsert and protect real data with business keys
In data integration output nodes, Insert appends records to the target table. Upsert updates existing records according to configured key fields and inserts new records. Upsert provides an important configuration basis for repeated execution, provided the key fields correctly identify business facts and the target connection supports the selected strategy.
For example, an order table may be unique by order number, while order details require both order number and line number. Matching detail lines by order number alone can overwrite real records. Check keys for nulls and duplicates first, then verify row counts, amounts, and field values after executing the same batch again.
| Output approach | Suitable tasks | Main checks |
|---|---|---|
| Preserve table + Insert | Append-only new records that will not be read again | Whether repeated execution adds duplicates |
| Preserve table + Upsert | Inserts and updates with reliable business keys | Existing-row updates, new-row inserts, and duplicate-key handling |
| Clear or recreate | Regenerate a complete result | Availability on failure, permissions, indexes, and downstream dependencies |
4. Diagnose failures from execution records before deciding to rerun
Hengshi execution plans support retry counts, task priorities, schedules, prerequisites, and failure alerts. Execution records help teams locate task status. Operations staff should compare these records with source and target database conditions to distinguish temporary network problems or lock waits from field, authorization, and key configuration problems that need correction.
When a source table’s structure changes, batch synchronization responds according to the selected strategy, and some configurations trigger full extraction. A suddenly slow incremental task should therefore also be checked for a switch to full extraction. Beyond a writable target connection, the database account and relevant directory need the required permissions.
Retry after the failure condition has been resolved, setting counts and waiting strategies according to the actual scheduler’s capabilities. Type mismatches and incorrect business keys require correction; repeated execution will not resolve them. When multiple operators or external schedulers share a target table, agree on execution windows and mutual exclusion responsibilities to prevent different tasks from rewriting results concurrently.
5. Example scenario: checking and recovering interrupted order synchronization
The following is a hypothetical solution example. A retailer synchronizes order details into its business analytics database, and the task stops because of a target database lock wait. Operations staff retain the execution records, identify the blocking transaction and data already written, and inspect input incremental state and output table business keys.
For data with reliable change fields, the team determines the supplemental extraction range from actual state. Historical records are recovered through a verified, separate supplemental extraction process, avoiding blind changes to incremental positions. The output uses a tested Upsert configuration. After recovery, the team checks the following results.
| Reconciliation method | What to compare | Considerations |
|---|---|---|
| Records and coverage | Business-key counts, time range, and organizational scope | Do not mechanically require equal raw row counts when filtering or aggregation applies |
| Key metrics | Sales, refunds, and valid order counts under consistent definitions | Fix the period, currency, and business status |
| Representative details | Field values for new, modified, and late-arriving records | Check mappings, types, and null handling |
After reconciliation passes, update downstream analytical data and dashboards through the agreed process. Hengshi connects ingestion, transformation, writing, and analysis into a coherent workflow. Correct incremental rules, reliable business keys, and inspectable execution results help teams recover synchronization in support of trustworthy business analysis.