Article body
Full article
Introduction
BI has become the decision entry point for many enterprises. Executives depend on morning dashboards, while risk teams monitor transactions around the clock. A BI outage removes the information people need to make decisions and can interrupt real-time operations.
High availability (HA) keeps services running through routine failures. Disaster recovery (DR) restores them after a site-level or regional incident. Hengshi BI addresses stability through component redundancy, failover, data backup, and multi-site deployment. This article explains the architecture and its implementation requirements.
1. Defining Targets with SLA, RTO, and RPO
1.1 Three Measurements
Service Level Agreement (SLA) defines the percentage of time a system remains available during an agreed period. Availability of 99.9% permits about 8.76 hours of annual downtime, while 99.99% permits about 52 minutes.
Recovery Time Objective (RTO) defines the target time for restoring service after a failure. An RTO of 15 minutes requires the team to restore core services within that window.
Recovery Point Objective (RPO) defines the acceptable data-loss window. An RPO of five minutes means the recovered state can lag the failure by no more than five minutes.
Higher availability and smaller recovery windows increase deployment complexity and cost. Project teams should set targets according to business-continuity needs. Contractual commitments depend on the agreed deployment design.
1.2 Defining Failure Domains
Resilience design starts by identifying the scope of each failure:
- Process failure: One service process stops, affecting requests handled by that process.
- Node failure: One server stops, affecting every service on that node.
- Site failure: A power or network outage affects all nodes in one data center.
- Regional failure: A large incident affects several sites in one region.
Redundant instances must run in separate failure domains so that one event does not remove every copy.
2. Hengshi BI High-Availability Architecture
2.1 Stateless-Service Redundancy
Ingress services, API gateways, and query routers keep session data in shared storage such as Redis instead of local disks.
Each stateless service runs at least two instances on separate nodes behind a load balancer. If a health check fails, the load balancer sends new requests to healthy instances. Teams add instances to scale horizontally as traffic grows.
2.2 High Availability for Stateful Services
Query engines and metadata services need state replication and leader election.
Query engine
- Uses a primary-replica or multi-replica topology
- Elects a new leader with a protocol such as Raft after the current leader fails
- Routes queries only to healthy nodes
Metadata service
- Uses a distributed key-value store or a relational database with replication, such as PostgreSQL
- Fails over to a standby database after the primary fails
- Gives priority to foundational metadata such as metric definitions and dataset configurations
2.3 Storage Redundancy
OLAP engines
Engines such as StarRocks and Doris store data in shards with several replicas. Other replicas continue serving queries after a node failure, and a background task restores the missing copy from healthy replicas.
Object storage
Charts and exported files can reside in a MinIO cluster or cloud object storage, using storage-level replication to withstand disk and node failures.
2.4 Health Checks and Graceful Removal
Health checks can cover four layers:
- L4 checks the TCP port.
- L7 checks the HTTP API.
- A business check runs a probe query.
- A dependency check inspects the database and OLAP engine.
Before an instance shuts down, the load balancer stops sending new work and waits for in-flight requests to finish. This process reduces interrupted user operations.
3. Multi-Site and Cross-Region Disaster Recovery
3.1 Active-Active Sites
Two low-latency sites carry production traffic and run complete service clusters. OLAP storage, metadata, and object storage replicate data through their corresponding mechanisms.
If one site fails, traffic moves to the other. Either site must have enough capacity to serve the production peak after its peer becomes unavailable.
3.2 Remote Active-Standby Recovery
The primary region handles daily traffic, while a remote recovery region synchronizes data and remains on standby. The replication interval determines the attainable RPO. Service startup and traffic switching determine the RTO.
This topology suits organizations that need cross-region recovery without operating both sites under normal load.
3.3 Backup and Restore
Active-active and standby sites do not replace backups. Teams can combine:
- Weekly full backups
- Daily incremental backups
- Continuous database logs such as WAL for point-in-time recovery
Backups should reside in object storage outside the production region. Teams also need periodic restores into a test environment to verify data integrity and measure the recovery time.
3.4 Failure Exercises
An architecture diagram does not prove availability. In an isolated environment, test teams can terminate service processes, disconnect node networks, and trigger database failover while recording the system response.
Multi-site deployments also need scheduled switchovers. Teams compare the result with their RTO and RPO targets and track every issue found during the exercise.
4. Observability for Resilience
4.1 Core Monitoring Signals
Availability
- Process, port, and API health
- HTTP 5xx rate and request success rate
- End-to-end probes from a user request to chart rendering
Performance
- P95 and P99 query latency
- CPU, memory, I/O, and connection use
- Request queue depth
Capacity
- Storage use
- Database and service connection counts
- Missing data replicas
4.2 Alert Severity and Consolidation
Teams can classify alerts by impact:
- P0: The service is unavailable and needs immediate response.
- P1: A core feature is degraded and needs action within the agreed window.
- P2: A resource is approaching a limit and the on-call team schedules remediation.
The alerting platform should group component alarms caused by the same dependency failure. It can attach dependency context and a response procedure to reduce duplicate notifications.
4.3 Automated Recovery
The platform can handle selected failures without operator action:
- Restart a crashed process with exponential backoff.
- Remove a node after repeated health-check failures.
- Rebuild a missing storage replica from healthy copies.
- Rate-limit low-priority work when traffic exceeds capacity.
5. Implementation Guidance
5.1 Capacity Planning
Redundancy increases cost. Teams need to size each topology against peak traffic and failover targets:
- A single-site deployment keeps headroom above the expected peak.
- An active-active deployment lets either site handle all core traffic after the other fails.
- A recovery site reserves the compute and storage required by the agreed recovery scope.
5.2 Common Problems
Applications are redundant, but dependencies are not
Two application nodes still fail as a service if both connect to one database primary. Load balancing, applications, databases, and storage all need redundancy that matches the target, with replicas in separate failure domains.
The recovery environment is never tested
Data replication can stop without attracting attention. Teams should restore and switch according to an agreed schedule and confirm that recovery data supports production workloads.
RTO and RPO do not match the budget or business need
Near-zero interruption and data loss require synchronous replication, active capacity on both sides, and a mature switching process. Project teams should measure the effect of downtime before selecting an architecture and investment level.
6. Conclusion
Enterprise BI resilience requires an end-to-end design. Stateless services need multiple instances and load balancing, stateful services need replication and election, and the data layer needs redundant copies. Deployments across failure domains add active-active or standby recovery.
Hengshi BI combines these mechanisms with health checks, monitoring, alerting, backups, restores, and failure exercises. Organizations can define SLA, RTO, and RPO from their business-continuity needs and then choose the matching topology and capacity.