Sub-Second Refresh, Stream-Batch Fusion: Inside Hengshi BI Real-Time Data Analytics Engine

Abstract: Yesterday data viewed today is being replaced by current data viewed now. Whether it is GMV bouncing on an e-commerce promotion real-time dashboard, millisecond-level financial risk control anomaly detection, or real-time manufacturing sensor data monitoring, real-time data analytics has shifted from a nice-to-have to a must-have. This article breaks down the technical内核 of Hengshi BI real-time data analytics engine—from data ingestion to stream-batch fusion computing to sub-second dashboard refresh—providing enterprise technical teams with a reference for real-time BI solution selection.

1. Why Real-Time BI Needs Both Speed and Accuracy

Before discussing technical solutions, clarify a key distinction: the essential difference between real-time and historical data analytics is not about how many seconds apart, but that the computing paradigms are completely different.

Historical data analytics is store-then-compute: data enters the data warehouse, T+1 batch processing runs, reports come out the next day. This model inherently has a data time gap—what happens today is visible earliest tomorrow.

Real-time data analytics is compute-as-it-comes: data is consumed, computed, and displayed the moment it is generated. This requires the compute engine to support stream processing, the storage engine to balance fast writes with fast reads, and the display engine to support incremental updates rather than full refreshes.

Hengshis real-time BI engine solves exactly this compute-as-it-comes problem. Its technical system can be divided into four layers: data ingestion, stream computing, hybrid storage, and real-time rendering.

---

2. Data Ingestion: Getting Real-Time Data to Flow In

2.1 Diversity of Real-Time Data Sources

Real-time analysis data sources are increasingly diverse for enterprises. Message queues (Kafka, Pulsar) carry business system event streams—every user order, payment, and inventory change writes to the message queue as an event. IoT gateways aggregate sensor data reported by production and logistics equipment, with much higher data density than business systems. Database change logs (CDC) capture database INSERT/UPDATE/DELETE operations through tools like Debezium, outputting as real-time streams.

Hengshis data ingestion layer provides standardized access for these three real-time data source types. For Kafka message streams, Hengshi has a built-in Kafka Consumer connector supporting Avro, JSON, Protobuf, and other serialization formats. For database CDC, Hengshi is compatible with Debezium output format, able to real-time subscribe to MySQL Binlog, PostgreSQL WAL, and other change logs.

2.2 Data Quality Gate

Real-time data is characterized by arriving fast with uneven quality. If real-time data is not filtered for quality, dirty data directly pollutes real-time dashboards, leading managers to make decisions based on incorrect data.

Hengshi embeds a lightweight data quality gate at the ingestion layer: schema validation automatically rejects data rows with mismatched field types, range validation rejects obviously anomalous values (such as negative sales amounts or astronomical order quantities), deduplication window detects duplicate data within short time periods, and latency monitoring tracks end-to-end delay from data generation to ingestion. These validations complete within the first few milliseconds of the stream processing pipeline—even discarded data is logged in the data quality log for investigation.

---

3. Stream Computing: From Event Streams to Business Metrics

3.1 Stream-Batch Fusion Computing Model

In traditional solutions, real-time computing uses Flink or Spark Streaming while offline computing uses Spark Batch. The same metric requires two sets of calculation logic: one stream-style for real-time dashboards and one batch-style for T+1 reports. The maintenance cost and definition inconsistency risk of these two sets are both high.

Hengshis solution is stream-batch fusion: the same metric definition is defined only once in the Metrics Platform. The engine layer automatically determines: when this metric is used for real-time dashboards it computes in stream mode, when used for T+1 reports it computes in batch mode. Only one set of calculation logic is maintained, and definitions are naturally unified.

3.2 Key Technical Components

Time window management is the infrastructure for computing metrics like order volume in the last 5 minutes or sales in the past hour. Hengshi engine supports sliding windows and tumbling windows, and handles late-arriving data updates properly: when data arriving 30 seconds late due to network issues reaches the engine, it automatically updates the corresponding time window calculation results and triggers dashboard refresh.

State management is one of the difficulties in stream computing. Metrics like cumulative sales or daily UV that need cross-window accumulated state—Hengshi engine maintains lightweight state snapshots in memory and periodically persists them to disk, ensuring continued computation without losing intermediate state after故障 recovery.

Multi-stream Join is the most prone to performance issues in real-time analysis. Hengshi engine adopts a time-alignment plus dimension-correlation strategy, controlling the Join time window within a business-reasonable range, avoiding infinitely expanding state storage.

3.3 Trade-off Between Accuracy and Throughput

Real-time computing always balances precision and speed. Hengshi provides three computing mode options.

Exact mode guarantees computation results are completely consistent with batch processing, at the cost of higher latency—suitable for financial reconciliation, compliance reporting, and other scenarios where precision is paramount. Approximate mode uses probabilistic data structures like HyperLogLog and Count-Min Sketch, significantly reducing memory usage and computation latency within acceptable error ranges (typically below 2%), suitable for dashboard-level trend monitoring. Adaptive mode is the default—the engine automatically selects exact or approximate algorithms based on data volume and computation complexity, dynamically balancing precision and efficiency.

---

4. Hybrid Storage: Balancing Write Speed and Read Speed

Real-time BI presents contradictory requirements to the storage layer: write fast (potentially tens of thousands or hundreds of thousands of records per second) and read fast (dashboard queries require millisecond-level response). Traditional row storage excels at fast writes, column storage excels at fast reads—no storage engine satisfies both extremes simultaneously.

Hengshis approach is a hybrid storage architecture: hot data is stored in a time-series-optimized in-memory store, responsible for the last few minutes to hours of data, pursuing write speed and low-latency queries. Warm data flushes to a columnar storage engine, responsible for the last few days of data, ensuring sub-second query response while reducing memory costs. Cold data archives to low-cost object storage, supporting historical comparison and trend analysis, with query latency relaxed to seconds.

The query routing layer is transparent to upper layers—whether querying the last 5 minutes or the last 5 years, the query router automatically determines data distribution and selects the appropriate storage layer for query execution, without requiring users or dashboard developers to sense underlying storage transitions.

---

5. Real-Time Rendering: Sub-Second Dashboard Refresh

5.1 Incremental Updates Instead of Full Refreshes

Traditional BI dashboard updates are full refreshes: when data changes, the entire dashboard re-renders. In real-time scenarios, dozens of data points may change per second, and full refresh overhead becomes completely unacceptable.

Hengshis real-time dashboards adopt an incremental update mechanism: the rendering engine only partially redraws chart components affected by changed data. This design keeps dashboards smooth even at refresh rates of dozens of times per second.

5.2 Data Push Protocol

Hengshis real-time dashboards do not rely on browser polling. A WebSocket long connection is established between the browser and Hengshi server—data changes trigger the server to actively push updates. In this mode, the browser does not actively ask is the data different, the server only sends pushes when data actually changes, with zero invalid requests. Additionally, because the server knows which dashboard the browser is currently displaying and what filter conditions are applied, the pushed content is precise to the current views incremental data, wasting no bandwidth.

5.3 Dashboard-Level Refresh Strategies

Different dashboard components have different real-time requirements. The big board transaction total wants refreshing every 2 seconds, category sales trends are fine at 10 seconds, and store rankings might only need updating every 30 seconds.

Hengshi allows dashboard developers to configure refresh strategies per component: key KPI cards refresh at high frequency, charts requiring complex database aggregations refresh at medium frequency, and static ranking charts refresh at low frequency. This differentiated refresh strategy ensures perceived real-time performance for users while significantly reducing backend engine query pressure.

---

6. Practical Scenarios

Scenario 1: E-Commerce Promotion Real-Time Dashboard

During Double 11 or 618 promotions, the operations command center needs a real-time GMV dashboard. The data source is the Kafka message stream from the order system. GMV is the sum of effective order amounts calculated through filtering rules (excluding unpaid, cancelled, and risk-controlled orders), with discount amounts real-time split and attributed by product dimension.

Hengshi engine consumes order events from Kafka, passes through data quality validation and complex event processing, flows through tumbling window aggregation, and finally pushes to the big board via WebSocket. The entire chain from order generation to screen update completes within 3 seconds. When network jitter causes some data to arrive late, the engine automatically performs backfill updates, ensuring statistical data is eventually consistent.

Scenario 2: Manufacturing IoT Real-Time Monitoring

A manufacturing enterprise needs real-time monitoring of hundreds of sensor points across thousands of devices. If temperature or vibration sensor values for any device exceed thresholds, alerts need to trigger within 2 seconds.

In this scenario, speed outweighs precision. Hengshi engine enables approximate computation mode with sliding windows, with each window maintaining aggregation results for only the last 30 seconds. Alerts trigger when a devices vibration values exceed normal ranges for 3 consecutive windows, rather than on single anomalies—avoiding false alarms from sensor instantaneous noise. Alert messages push to on-call engineers via enterprise WeChat, with device ID, anomalous metrics, and a mini trend chart of the last 5 minutes.

---

7. FAQ

Q1: How high are the additional costs of real-time BI?

Main costs come from three parts: message queue storage and bandwidth, stream compute engine computing resources, and hybrid storage I/O overhead. It is recommended to only enable stream processing mode for key metrics that truly need real-time updates, not running all metrics on real-time pipelines. Hengshis one-click switching mechanism lets you flexibly decide which metrics go real-time and which go batch.

Q2: What if real-time and offline data frequently do not match?

This is a classic problem of dual-write solutions: real-time streams and offline batch use two sets of computing logic, so results naturally diverge. Hengshis stream-batch fusion solution fundamentally solves this: the same metric definition, the same computing logic, just different execution modes. If real-time and offline results still have tiny differences, these are usually due to subtle differences in late-arriving data and deduplication strategies, and can be ignored within acceptable error ranges.

Q3: At what scale of real-time data volume do performance bottlenecks appear?

This depends on data granularity, window size, and query complexity. Hengshis publicly published performance metrics in cluster mode: single cluster supports million-level events per second ingestion, sub-second window aggregation, and smooth rendering of dozens of concurrent real-time dashboards. In actual deployment, it is recommended to start with a single node and scale cluster nodes linearly with business growth.

---

Conclusion

Real-time BI is not simply accelerating batch processing to real-time—it is an upgrade to the computing paradigm of the entire data architecture. Through the ingestion layers multi-source unified access, the compute layers stream-batch fusion model, the storage layers hot-cold hybrid architecture, and the rendering layers incremental update mechanism, Hengshi BI real-time engine provides enterprises with a complete solution for millisecond-level closed-loop from data to decisions. As more industries shift real-time analytics from a nice-to-have to a survival necessity, choosing the right underlying technical architecture is far more important than chasing feature highlights.