Article body
Full article
Introduction
A large enterprise BI environment may contain 200 datasets, 500 metrics, 800 dashboards, and 3,000 reports. Metrics depend on dataset fields, dashboards reference metrics, and reports embed dashboard charts. Together, these assets form a dependency graph.
Renaming one field can break several downstream metrics, including metrics used in executive dashboards. If the team discovers the problem just before a management meeting, engineers have to trace the failure under time pressure.
Data lineage records how data moves from its source to each consumer. It shows the impact of a change before deployment and gives responders an upstream path to inspect after a failure. Hengshi BI provides lineage graphs and impact analysis within its metric-management and dataset systems. This article explains the layered model, graph operations, and engineering use cases.
1. Problems That Data Lineage Solves
1.1 Three Common Risks
Changes without dependency awareness
The data platform team changes a field type, and metrics that depend on that field begin to fail. If the upstream and downstream teams cannot see their shared dependencies, they exchange context only after an incident starts.
Slow incident diagnosis
A KPI on a dashboard drops to zero. The cause may sit in the source system, an ETL job, a metric definition, or a report configuration. Without lineage, responders inspect each layer in sequence.
Unknown impact radius
A dataset that appears unused may feed several core metrics through indirect references. Removing it can break daily reports across multiple departments.
1.2 Business Value
Data lineage turns hidden dependencies into a graph that teams can query:
- Before a change: Identify the metrics, dashboards, and users affected by a field, metric, or dataset update.
- During an incident: Trace an abnormal metric upstream to the earliest failing node.
- Before decommissioning: Confirm that no downstream assets depend on the target.
2. Hengshi BI’s Four-Layer Lineage Model
Hengshi BI models the complete path from a data source to its consumers as a multilayer network.
2.1 Physical Lineage: Data Sources to Datasets
Physical lineage records the sources, tables, and fields read by each dataset.
The platform collects dependencies by:
- Parsing dataset SQL to extract source tables and fields
- Reading field mappings from visual data models
- Recording the topics and partitions consumed by real-time data flows
The graph preserves field-level detail. For example, it records that field X in dataset A comes from field Y in table B instead of storing only a dataset-to-table link.
2.2 Semantic Lineage: Datasets to Metrics
Semantic lineage records the datasets, fields, and calculations behind each metric definition.
The platform collects:
- Source datasets declared in the metric-management platform
- Fields referenced by metric expressions
- Dependencies between derived and base metrics, such as the dependence of net margin on net profit and revenue
Metrics generated with AI also enter the semantic-lineage graph. The platform retains the datasets, fields, and calculation basis used to create each metric so that teams can review it later.
2.3 Consumption Lineage: Metrics to Dashboards and Reports
Consumption lineage records the metrics used by dashboards, reports, charts, subscriptions, and alert rules. Each consumer links to a stable metric ID, allowing teams to find every direct and indirect consumer of a metric.
2.4 User Lineage: Dashboards and Reports to Users and Roles
User lineage combines dashboard permissions with subscription relationships to identify the users and roles that can access content based on a given metric.
If a metric has a quality issue, the team can identify both the affected dashboards and the audience that needs a notification.
3. Storing and Computing the Lineage Graph
3.1 Graph Model
Hengshi BI can store lineage in a graph database or represent it with adjacency tables in a relational database.
The graph contains two basic elements:
- Nodes: Source tables, fields, datasets, metrics, dashboards, reports, and users
- Edges: Physical dependencies, semantic dependencies, consumption references, and user permissions
Each node has a unique identifier and a type label. Each edge records its dependency type and properties such as field mappings or metric references.
3.2 Upstream and Downstream Traversal
Lineage analysis uses two main graph queries.
Upstream tracing
The platform starts from an abnormal metric or chart and follows dependency edges in reverse to find upstream nodes that may have caused the failure. A breadth-first search can perform the traversal, with a depth limit to keep unrelated paths out of the result.
Downstream impact analysis
The platform starts from a field or asset scheduled for change and follows dependencies to every downstream node. Results can be grouped by distance into direct, indirect, and remote impact.
3.3 Incremental Updates and Version Snapshots
Datasets, metrics, and dashboards change over time, so the lineage graph must change with them:
- Source schema events update physical lineage.
- Metric-definition changes recalculate semantic lineage.
- Dashboard saves update consumption lineage.
- Snapshots taken before major changes show dependency differences before and after the update.
4. Engineering Impact Analysis
4.1 Previewing Impact Before a Change
When a user modifies a dataset field, the platform runs downstream impact analysis and returns the affected metrics, dashboards, and users. If three metrics use the field and one of them supports two executive dashboards, the team can proceed, notify users first, or postpone the change.
The report can rank assets by business importance, separating executive, departmental, and personal dashboards so the team addresses the largest impact first.
4.2 Producing a Root-Cause Path
After a dashboard fails, an operator can trace upstream from the chart through its metric, dataset, and source table. The system stops at the first unhealthy node and produces a path such as “dashboard → metric → dataset → deleted source table.”
The lineage graph converts a broad, layer-by-layer investigation into one dependency chain that responders can verify.
4.3 Running a Safety Check Before Decommissioning
Before deleting a dataset or metric, the platform checks downstream dependencies:
- If direct or indirect consumers exist, it blocks deletion and lists the affected assets.
- If no downstream dependency exists, it allows a safe deletion.
Graph construction also needs cycle detection. If A depends on B, B depends on C, and C depends on A, the platform should flag a configuration error before traversal enters a loop.
5. How Lineage Supports Data Governance
5.1 Data-Quality Monitoring
Teams can attach null-rate, range, and consistency rules to core source tables and metrics. When an upstream rule fails, the platform propagates a quality warning through the lineage graph and marks dashboards that use the affected metric.
5.2 Compliance Auditing
Organizations in regulated industries can use lineage to:
- Trace a report value from its source table through each calculation layer
- Identify downstream assets that use a sensitive field and confirm masking policies
- Record who changed a dependency, when it changed, and which assets it affected
5.3 Metric Lifecycle Management
Consumption lineage can show how many dashboards reference a metric and how many users access it. The platform can flag a metric with no recent consumers as a retirement candidate, then run another dependency check before removal. During metric creation, semantic lineage can also surface similar definitions and reduce duplication.
6. Common Problems and Design Guidance
6.1 Lineage Stops at the Table Level
Table-level lineage cannot show whether a change to one field affects a consumer, which creates noisy impact reports. Field mappings distinguish referenced fields from unused fields and produce a narrower result.
6.2 Teams Maintain Lineage by Hand
Manual dependency records become stale as configurations change. A lineage platform should collect relationships from SQL, model configuration, and asset-change events.
6.3 Impact Reports Lack Priority
A low-level table may have hundreds of downstream nodes. The report should combine asset tier, audience size, and business context to put executive dashboards and core metrics first.
7. Conclusion
Teams cannot track every dependency from memory once an enterprise runs hundreds of metrics and thousands of dashboards. Hengshi BI connects data sources to users through physical, semantic, consumption, and user lineage, with automated collection keeping the graph current.
The graph supports change previews, root-cause tracing, and safe decommissioning. It also provides a shared dependency model for data quality, compliance, and metric governance. Teams can see the impact before changing an asset and follow a known path when an incident occurs.