From 100,000 to 10 Million Rows: Engineering Boundaries for HENGSHI SENSE Exports

Financial reconciliation, business archiving, and data review often require taking analytical results out of a BI platform. HENGSHI SENSE 6.2 changes the export rule for chart detail data: instead of a fixed 100,000-row limit, exports follow the system security policy, supporting up to 10 million rows. Understanding this change requires considering the export object, file format, resource use, and permissions together. This article explains how to deliver an export that can be checked and used in subsequent work.

1 What the 10-million-row limit means

The 6.2 release notes identify the object of this improvement: chart detail data. The administrator’s export-row policy determines how much a user can actually export. An upper limit of 10 million rows does not mean every user receives that allowance by default, nor does it indicate how long a file will take to generate.

Under Settings → Security Management → Security Policy, administrators can control which users may export and the permitted row count. The current manual allows a setting from 1 to 10,000,000 rows and lists product limits of 10 million rows and 2 billion cells per file, with at most 1 million rows per sheet. Rows beyond a sheet’s limit are assigned to subsequent sheets. These limits govern different aspects: row count constrains data volume, while the number of cells also depends on the number of fields.

During implementation, check product support, the deployment’s configuration, and the target account’s permissions separately. Record how many rows the query returns and how many the file actually contains. A permitted volume does not replace validation of completeness and resource capacity.

2 Choose the required result before the export method

The same report can serve different delivery purposes. For transaction-by-transaction reconciliation, finance needs detail for the intended scope; for management review, an aggregate table or formatted document may be more suitable. Define what the recipient needs to do with the file before deciding whether all data is necessary.

Need	What to check first	Acceptance focus
Check the current table	Current-page scope, expansion level, and selected export options	Records on the current page and their displayed meaning
Process data in bulk	Detail or aggregate results and the all-data scope	Row count, field types, and allocation across sheets
Present and read results	Delivery formats such as PDF, PPT, and PNG	Pagination, titles, charts, and watermarks

“Export current page” and “Export all” also offer different formatting capabilities. For tables, the current manual limits cell styles, plain text, filter-condition descriptions, expansion level, and control titles to current-page exports; all-data exports do not support that same set of options. Whether a formatting option is available does not determine whether business filters should apply: verify the intended data scope at the actual export entry point. For bulk processing, prioritize fields and values; for reproducing a reading layout, choose an output suited to that purpose.

3 Streaming reduces the working set; capacity still requires measurement

A large export usually involves querying, reading results, converting formats, writing a file, and downloading it. If the implementation retains all results in memory before generating a file, adding rows and fields increases memory pressure. Streaming aims to advance reading and writing in batches and promptly release processed data.

When designing this pipeline, developers must check whether each stage still accumulates results: how the database driver reads rows, whether conversion caches the entire table, and whether the writer retains all cells. Making only the file writer incremental cannot remove full buffering elsewhere. Temporary files, long text, styles, and concurrent jobs also affect peak resource use.

Therefore, an export assessment should go beyond saying “memory does not grow with row count.” Useful records include field count, data types, query duration, generation duration, peak memory, and temporary disk usage. Measure these in the target environment to determine which volumes suit online delivery. This is a capacity-design and acceptance method, not a performance result for any deployment.

3.1 A small example separates reading, writing, and publication

The following Python 3.8+ standard-library example writes SQLite results to CSV in batches. It explains an export pipeline; it is not HENGSHI’s internal implementation, public API, or Excel-export code. It reads only order IDs and amounts. The database must already contain example_orders, with order_id as a unique integer primary key, order_date as text in YYYY-MM-DD format, and net_amount as monetary text with confirmed precision.

import csv
import os
import sqlite3
from contextlib import closing
from pathlib import Path
from uuid import uuid4


def export_csv(db_path, out_dir, start, end):
    out_dir = Path(out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    final = out_dir / f"export-{uuid4().hex}.csv"
    partial = final.with_suffix(".csv.partial")
    rows = 0
    try:
        with closing(sqlite3.connect(db_path)) as conn:
            conn.execute("BEGIN")
            cursor = conn.execute(
                "SELECT order_id, net_amount FROM example_orders "
                "WHERE order_date >= ? AND order_date < ? "
                "ORDER BY order_id",
                (start, end),
            )
            with partial.open("x", newline="", encoding="utf-8-sig") as file:
                writer = csv.writer(file)
                writer.writerow(["order_id", "net_amount"])
                while batch := cursor.fetchmany(5000):
                    writer.writerows(batch)
                    rows += len(batch)
                file.flush()
                os.fsync(file.fileno())
            conn.rollback()  # End this read transaction
        os.replace(partial, final)
        return {"path": str(final), "rows": rows}
    except Exception:
        partial.unlink(missing_ok=True)
        raise

After preparing the example table, call export_csv("demo.db", "exports", "2026-10-01", "2026-11-01"). The inclusive start and exclusive end avoid including the first day of the next month. Parameter binding separates values from SQL; fixed primary-key ordering helps check sequence; the explicit read transaction gives this SQLite query a consistent read view. Verify transaction isolation and cursor behavior separately for other databases and drivers.

Each loop reads up to 5,000 rows and writes them before fetching the next batch, so the application does not first build a complete results list. The batch size is illustrative, not optimal. Wider rows use more space for the same row count; database sorting, driver buffering, and file conversion can still consume substantial memory or disk. Any stage that accumulates the entire table weakens the benefit of batching.

The temporary and final files share a directory. Only after writing finishes and the file closes does os.replace run, avoiding the publication of an unfinished download. Exception handling removes this process’s temporary file; forced process termination still requires a separate expiry-cleanup mechanism. The example does not implement resumability, job scheduling, or download authentication, and it does not preserve spreadsheet field types. Real delivery must also handle long identifiers, null values, and text that spreadsheet software might interpret as formulas.

3.2 Evaluate field width and concurrency

Even at the same 1 million rows, 20 short fields and 200 fields containing long text can have very different generation costs. For an initial file-size estimate, measure the average encoded bytes per row in a representative sample and multiply by the target row count. Compression, styles, shared strings, and temporary files change actual disk use, so this estimate only helps select test volumes.

Validate concurrency separately. Three simultaneous jobs may compete for database connections, read snapshots, file-writing bandwidth, and temporary disk space. First measure one job’s query and generation durations, then increase concurrency gradually and record peaks and failure stages. If the recipient only needs aggregated results, reducing fields or delivering aggregates is usually more useful than simply raising the export allowance.

4 Validate asynchronous execution and failure recovery separately

Asynchronous processing can move lengthy work out of a single long-running request. Reliable design also needs to define who owns the job, how results are retrieved, how failures are handled, and who cleans temporary files. A browser receiving “submitted” and a backend completing file generation are different states.

For a particular HENGSHI SENSE export entry point, verify completion and download behavior in the deployed version. An asynchronous request alone does not establish that a job survives page closure, logout, or a service restart. Update and subscription schedules under Task Management are also not a universal queue for all interactive exports.

When a Gateway Time-out occurs, retain the export time, entry point, and error, then locate the timeout across the proxy, load balancer, WAF, and server. An interrupted outer request does not establish that the database is slow or file generation failed. Raising one timeout cannot solve every problem in the chain.

4.1 Recover according to the failed stage

An export can be analyzed in the following stages. These are diagnostic categories, not product states with these names. Find the last confirmed completed stage before deciding whether to rerun the query, reducing ineffective retries.

Failed stage	Evidence to retain first	Recovery checks
Query or retrieval	Original error, query conditions, data source, and time	Connections and permissions; source-data changes; whether a comparison batch must be fixed again
File generation	Rows read, temporary-file size, disk and process logs	Available disk; field-conversion errors; isolation of incomplete artifacts
Result publication	Whether file generation finished; registration or storage error	Whether the generated file is reusable; whether the completion marker matches the artifact
Download transfer	Response status, proxy logs, and client download error	Whether the result remains valid; whether downloading again is allowed or regeneration is necessary

Do not repeatedly click Export after a timeout. First check whether the earlier request is still running, avoiding multiple identical jobs. Where retries are supported, determine whether retry continues the original job or retrieves data again. If source data has changed, even a successful second attempt may no longer be comparable on the same basis.

5 Carry permissions and business definitions into file delivery

Viewing and exporting are separate permissions. A user may be allowed to read a dashboard while export-user scope or row-count policies restrict extraction. Administrators should use real business roles to check application data-permission modes, row and column rules, masking requirements, and export policies. Hiding a field in the interface does not replace column-permission checks.

Before exporting, fix the business scope: period, organization, filters, metric definitions, and the last successful data-update time. If source data changes during reconciliation-file generation, agree on a comparison point and, where necessary, use a confirmed data batch. Otherwise, two successful queries may still differ because they ran at different times.

Deliver a scope description, actual row count, and key aggregate values with the file. Sample order IDs, long identifiers, dates, nulls, and monetary precision to avoid changes in meaning when the recipient opens the data in spreadsheet software. Opening the file is only the first check.

6 Rehearse the complete chain with a month-end close

This is an illustrative scenario, not a product benchmark. A finance team plans to reconcile 2.3 million order-detail rows. The owner first fixes the closing period and field list, then confirms that the target account may export that volume. Under the manual’s per-sheet limit, the result spans multiple sheets. Acceptance reviewers should total the records across sheets and inspect orders around each boundary for duplication or omission.

The team then checks total amounts for the same period and filters, samples refunds, nulls, long order IDs, and records across day boundaries, and records query and generation duration. If export fails, locate the failure before retrying to avoid further resource use from duplicate submissions. After checking the data, distribute the file only to the agreed recipients through the agreed channel.

6.1 Retain the acceptance conclusion

The following JSON structure can record acceptance results. It is a fictional manual handover example, not a platform API response or a performance measurement. Sheet counts include data records only, excluding headers; use the actual export file to determine allocation.

{
  "example_only": true,
  "period": {
    "start_inclusive": "2026-10-01",
    "end_exclusive": "2026-11-01"
  },
  "data_batch": "Example: finance-confirmed month-end batch",
  "expected_data_rows": 2300000,
  "actual_data_rows": 2300000,
  "sheet_data_rows": [
    1000000,
    1000000,
    300000
  ],
  "amount_reconciliation": "Totals match for the same batch and filters",
  "boundary_samples": "Orders at sheet boundaries and long identifiers checked",
  "delivery_scope": "Authorized finance reconciliation recipients"
}

In actual use, add the specific dataset, filters, file name, reviewer, and measurements. Row counts can reveal truncation; amount totals can expose value differences; boundary sampling can reveal duplicates or omissions. These checks complement one another; none alone proves completeness.

By connecting chart-detail exports to the security policy, HENGSHI SENSE lets enterprises configure delivery volumes for business needs. Data teams must turn that allowance into reproducible operations: choose the right object, fix definitions, validate capacity, and check file contents. Exported data can then support month-end closing, archiving, and subsequent analysis.