Article body
Full article
Introduction
Traditional BI data exploration is field-driven. You open a dataset, scan dozens of field names, work out what each one means, then drag fields into rows, columns, and filters to build an analysis.
Technical users can manage this workflow. Business users face a steep barrier. A sales director who sees the field ord_amt_excl_tax_amt may not know that it means “order amount excluding tax.” When that director wants to analyze customer value, a table with hundreds of fields offers little guidance on which fields belong in the analysis.
The core issue is that a BI system understands field names but not business semantics. Users speak in business terms, while the system stores technical fields. A translation gap sits between them.
Vector search gives BI systems semantic understanding for the first time. It moves matching from exact field names to approximate matches between business concepts. The latest HENGSHI BI release builds a complete vector-search and semantic-layer system. This article examines the technical path behind it.
1. From Keyword Matching to Semantic Matching
1.1 Limits of Traditional Search
Data discovery in traditional BI relies on keyword matching. A user searches for “sales revenue,” and the system looks through field names and descriptions for text containing those words.
That approach has several blind spots.
Synonyms. A user searches for “revenue,” while the field is named “operating income.” The keywords do not match, so the search finds nothing.
Abbreviations. A user searches for “GMV,” while the field is named gmv_amount. Without case-insensitive and substring matching, the search may miss it.
Semantic proximity. A user wants to analyze customer activity, but no field carries that name. Several fields may jointly express it, including login count, order count, and visit duration. Keyword matching cannot establish that semantic relationship.
1.2 How Vector Search Addresses the Problem
The core of vector search is simple: it converts text into high-dimensional vectors and measures semantic similarity by vector distance.
“Sales revenue” and “operating income” use different words, yet their vectors sit close together in semantic space because the terms appear in similar contexts across a large corpus. Vector search captures that proximity.
The workflow follows four steps:
- Convert field names, field descriptions, metric names, and business terms into vectors with an Embedding model.
- Store the vectors in a vector index such as HNSW.
- Convert the user’s query into a vector.
- Find the nearest Top-K vectors in the index and return their corresponding fields or metrics.
The key change is a move from literal matching to semantic matching. When users ask for “revenue,” the system can find the “operating income” field. When they ask for customer activity, it can recommend related fields such as login count and order frequency.
2. Vector Index Architecture for the HENGSHI BI Semantic Layer
2.1 Semantic-Layer Data Model
The HENGSHI BI semantic layer is not a simple field catalog. It is a multi-layer semantic network.
Layer 1: Fields. Each dataset field forms a basic metadata unit with a name, type, description, and source dataset.
Layer 2: Metrics. Derived metrics are calculated from fields. For example, “monthly sales revenue” equals the monthly grouping of SUM(order amount). Each metric links to its underlying fields and calculation logic.
Layer 3: Concepts. Business concepts provide an abstract expression of business language. For example, customer value is a concept represented jointly by three metrics: spending amount, purchase frequency, and most recent activity time. The concept layer bridges business language and technical implementation.
Layer 4: Relationships. Fields, metrics, and concepts carry semantic relationships. These include synonym relationships such as “revenue” = “operating income”; hierarchical relationships such as East China containing Shanghai and Jiangsu; and derivation relationships such as gross margin being derived from gross profit and revenue.
2.2 Building the Vector Index
HENGSHI vectorizes and indexes every node in the four-layer semantic network.
Field vectorization. The system joins a field name and field description, then uses an Embedding model to generate a 1024-dimensional vector. For the field ord_amt, the descriptive text might read: “Order amount: the total amount paid when a user places an order, excluding shipping and discounts.” Its vector encodes semantics related to orders, amounts, and payments.
Metric vectorization. The system joins the metric name, calculation-standard description, and business explanation. For example, the text for “monthly sales revenue” may read: “Monthly sales revenue: the total amount of paid orders aggregated by calendar month, excluding refunds and canceled orders.” The vector encodes month, sales revenue, paid, and excluding refunds.
Concept vectorization. The system joins a concept name, definition, and list of related metrics. For example, the text for customer value may read: “Customer value: a composite evaluation of a customer’s contribution to the business, based on spending amount, purchase frequency, and most recent activity time.” Its vector encodes customer, value, contribution, spending, frequency, and activity.
Relationship vectorization. HENGSHI does not vectorize relationships separately. Descriptive text encodes them implicitly in node vectors. For example, the description of East China includes Shanghai, Jiangsu, Zhejiang, Anhui, and Fujian, which embeds the hierarchy in the vector.
2.3 Selecting an Embedding Model
The HENGSHI BI semantic layer uses a two-level Embedding strategy.
Level 1: General semantic encoding. BGE-M3 serves as the base model for general semantic encoding. BGE-M3 supports mixed Chinese and English text and performs well on the MTEB benchmark.
Level 2: Domain fine-tuning. HENGSHI fine-tunes BGE-M3 through contrastive learning on its accumulated BI-domain corpus. Training data includes:
- Metric synonym pairs: “revenue” - “operating income” - “Revenue” - “total operating income”
- Field semantic pairs: “order amount” - “actual paid amount” - “amount excluding tax”
- Business concept pairs: “customer activity” - “login frequency” - “visit count”
On the BI semantic test set, the fine-tuned model raises Top-5 recall from 75% to 93%.
3. Four Modes of Semantic Data Exploration
3.1 Mode 1: Semantic Field Search
A user enters “customer spending power” in the dataset interface. Rather than searching field names, the system searches the semantic space of field vectors:
- Customer LTV (lifetime value): nearest semantic match
- Average monthly spend: second-nearest semantic match
- Average order value: third-nearest semantic match
- Total spend in the last 30 days: fourth-nearest semantic match
The system recommends these 4 fields and shows a semantic-similarity percentage for each. The user chooses the field that best matches the intended analysis.
Result. Users do not need exact field names. They only need to describe the concept they want to analyze. This matters especially for wide tables with more than 100 fields, where semantic search finds a target faster than manually scanning dozens of fields.
3.2 Mode 2: Automatic Mapping from Concepts to Metrics
In ChatBI, a user asks: “Help me rank customer value across regions.”
The Agent reasoning engine identifies customer value as a business concept instead of a specific metric. The system retrieves the concept definition and finds that it consists of three metrics:
- Spending amount, weighted at 50%
- Purchase frequency, weighted at 30%
- Days since most recent activity, weighted at 20% and inverted so a more recent activity produces a higher value
The Agent generates the calculation logic: aggregate the three metrics by region, calculate a weighted score, and rank the scores. The user states the analytical intent without needing to know the calculation details.
Result. The translation from a business concept to a technical implementation becomes automatic. Traditional BI requires users to break customer value into explicit metrics and formulas.
3.3 Mode 3: Semantic Recommendations for Related Fields
After a user chooses the sales revenue field for analysis, the system recommends semantically related fields:
- Order volume: highly related because sales revenue = average order value × order volume
- Average order value: highly related because it equals sales revenue / order volume
- Refund amount: moderately related because net sales revenue = sales revenue - refund amount
- Discount rate: moderately related as an attribution dimension that affects sales revenue
The recommendation logic combines field co-occurrence analysis with semantic similarity. Fields that frequently appear in the same analysis and share semantic relevance receive higher recommendation weights.
Result. The system reduces cognitive load. Users do not need to remember which fields belong beside sales revenue in an analysis.
3.4 Mode 4: Cross-Dataset Semantic Relationships
A user asks a question spanning multiple datasets: “What relationship exists between customer purchasing behavior and customer-service tickets?”
The semantic layer retrieves the field sets for purchasing behavior and customer-service tickets. It finds that both datasets contain a customer ID field. Their field names may differ, such as cust_id in one dataset and customer_code in the other, but the semantic layer labels them as synonyms.
The system then recommends linking the datasets through customer ID to support cross-domain analysis.
Result. The largest obstacle in cross-dataset analysis is often not knowing how two tables relate. Semantic-layer synonym labels enable the system to discover possible links.
4. Performance Engineering for Vector Search
4.1 Index Selection: HNSW
The HENGSHI BI semantic layer uses the HNSW (Hierarchical Navigable Small World) algorithm for its vector index for three reasons:
- Fast queries. Query latency in a vector database with millions of vectors can stay below 10ms.
- Incremental insertion. New fields and metrics can be inserted directly without rebuilding the index.
- Tunable recall. The efSearch parameter balances speed and recall.
4.2 Hybrid Retrieval Strategy
Pure vector retrieval is less reliable than keyword search for exact matches. When a user searches for “GMV,” vector retrieval may return “Gross Merchandise Volume” rather than the abbreviation “GMV” itself. HENGSHI uses hybrid retrieval:
- Vector retrieval recalls Top-20 candidates for semantic similarity.
- BM25 keyword retrieval recalls Top-20 candidates for exact matching.
- The system merges both result sets, reranks them with a Cross-Encoder, and keeps the Top-5.
Hybrid retrieval raises recall by 8-12% over vector retrieval alone while retaining exact-match capability.
4.3 Caching and Precomputation
For high-frequency semantic searches, HENGSHI uses two cache layers.
Result cache. The system caches results for identical query text for 1 hour. If several users search the same concept, it performs vector retrieval once.
Embedding cache. The system caches an Embedding vector for query text for 24 hours. Even after a result-cache entry expires, it can reuse the Embedding vector and skip model inference.
5. Semantic-Layer Governance and Maintenance
5.1 Semantic Quality Monitoring
Vector-retrieval quality depends heavily on semantic-layer data quality. HENGSHI provides a semantic-quality monitoring dashboard.
Coverage. This measures the share of searches that match a field or metric. If 30% of searches return no result, the semantic layer has a coverage gap.
Accuracy. This measures the share of recommended results that users select. If the system recommends 5 fields and users select none, the recommendations lack accuracy.
Exposure of infrequently used fields. The dashboard lists fields that have gone a long time without appearing in search results. Their metadata may need clearer descriptions.
5.2 Continuous Semantic-Layer Optimization
The semantic layer needs maintenance as the business evolves, new concepts appear, and old definitions change. HENGSHI uses several optimization mechanisms.
User-feedback loop. A “This is not what I am looking for” control appears next to search results. When users select it, the system records negative feedback for improving the Embedding model and reranking model.
Synonym discovery. The system periodically analyzes search logs for patterns where a user searches for A and selects field B. It extracts A-B synonym pairs for an administrator to review before adding them to the synonym table.
Concept mining. The system analyzes frequent search terms, identifies business terms that lack a concept definition, and prompts administrators to add one.
6. Summary
Vector search gives the BI semantic layer an ability to understand. It moves from exact field-name matching to semantic search over business concepts, so the BI system understands what a user wants to analyze rather than only which field the user selected.
The HENGSHI BI vector-search and semantic-layer system delivers three core benefits:
- Lower barrier to use. Business users describe their intent in natural language, and the system finds relevant fields and metrics.
- Discovery of hidden relationships. Semantic recommendations help users find analytical dimensions they may have missed.
- Concept-level analysis. The system moves from field-driven work to concept-driven work and understands business language.
As BI evolves from a tool for displaying data into a partner for understanding data, the entry point for analysis becomes a natural-language conversation box instead of a field list.