Hengshi RAG × ChatBI: How Retrieval-Augmented Generation Stops Natural-Language Data Queries from Hallucinating

Article body

Full article

18 min read

Introduction

Large language models have a widely known problem: hallucination. When asked a question they do not know, they do not say, “I don’t know.” Instead, they can confidently fabricate a seemingly plausible answer.

In casual conversation, hallucinations may only cause an awkward laugh. In BI, however, their cost can be disastrous. If ChatBI interprets “sales revenue” as “order count,” or “East China” as “Eastern Time Zone,” business decisions can be made on the basis of incorrect data.

Retrieval-Augmented Generation (RAG) is currently the most effective engineering approach for suppressing large language model hallucinations. Its core idea is simple: before an LLM generates an answer, retrieve relevant context from a knowledge base and inject it into the prompt, so that the model answers while looking at source material rather than relying on memory.

The latest version of Hengshi ChatBI introduces a complete RAG enhancement system that injects knowledge across four dimensions: metric semantics, business terminology, historical Q&A, and data schemas. This article examines the technical architecture and engineering practices behind this RAG system.

1. Why Does ChatBI Need RAG?

1.1 Three Problems with Pure LLM Data Queries

Without RAG, asking an LLM to answer natural-language data queries directly leads to three core problems.

Problem 1: Misunderstanding domain terminology

A user asks: “What was ARR last month?”

The LLM may not know that ARR means Annual Recurring Revenue. It may interpret it as an array or as an abbreviation for a field. Different enterprises also calculate ARR differently: some include trial-to-paid conversions, while others do not.

Problem 2: Metric definition drift

A user asks: “What is GMV this month?”

Within an enterprise, GMV may be defined as paid order amounts + unpaid order amounts - refund amounts. However, the LLM may simply sum the “order amount” field, ignoring refund deductions and unpaid-order filtering.

Problem 3: Insufficient schema understanding

A user asks: “What is the repurchase rate of the Top 5 customers in South China?”

The LLM needs to know which field identifies “South China,” which field identifies “customer,” how “repurchase rate” is calculated, and which tables are involved. If this information is not in the prompt context, the LLM can only guess. A correct guess is luck; an incorrect one is an incident.

1.2 RAG’s Approach to the Problem

RAG’s solution is straightforward: do not make the model guess. Give it the clues it needs to answer.

Before generating an answer, the system runs a retrieval process:

Encode the user’s question as a vector.
Retrieve the semantically most relevant document fragments from the knowledge base.
Concatenate the retrieved fragments into the prompt.
Have the LLM generate an answer based on these fragments.

The key shift is from “what the model knows” to “what we give the model.” The model’s parametric knowledge is no longer the only basis, and the external knowledge base becomes a more controllable source of information.

2. Hengshi ChatBI’s Four-Dimensional RAG Architecture

Hengshi ChatBI’s RAG system is not a single retrieval pipeline. It is divided by knowledge type into four independent retrieval dimensions, each with its own indexing strategy and injection method.

2.1 Dimension One: Metric Semantic Knowledge Base

Knowledge source: all metric definitions registered in the Hengshi Metric Management Platform.

Indexed content: each metric is stored as a structured document containing:

Metric name, including Chinese and English names, aliases, and common abbreviations
Calculation definition, including numerator logic, denominator logic, and filter conditions
Related dataset, identifying the dataset from which the metric is calculated
Business description, including business meaning, usage scenarios, and precautions
Synonym mappings, such as “revenue” = “operating revenue” = “Revenue”

Retrieval strategy: when a user’s question contains metric-related words, the system retrieves matching metric definitions from the metric library and injects them into the prompt. Matching uses dual-path recall, combining vector semantic similarity with exact keyword matching. Vector retrieval captures semantic similarity, such as matching “revenue” to “operating revenue,” while keyword matching captures exact abbreviations, such as an exact hit for “ARR.”

Injection format: retrieved metric definitions are injected into the prompt as structured text, explicitly marked as: “The following are definitions of relevant metrics. Calculate strictly according to these definitions.”

2.2 Dimension Two: Business Terminology Dictionary

Knowledge source: an enterprise-defined business terminology table.

Many enterprises have unique business concepts that appear infrequently in the training corpora of general-purpose LLMs. For example:

“Sales per square meter” for retailers, calculated as sales revenue / store area
“Load factor” for logistics enterprises, calculated as actual load / vehicle capacity
“NRR” for SaaS enterprises, meaning Net Revenue Retention

Indexed content: each term contains its name, definition, calculation formula, business context, and associated metrics.

Retrieval strategy: perform terminology entity recognition on the user’s question to detect whether it contains words from the business terminology table. If a term is detected, inject its full definition into the prompt.

Special handling: terminology detection must handle not only exact matches but also colloquial variants. For example, if a user says “store efficiency” rather than “sales per square meter,” the system can still identify it through synonym mapping.

2.3 Dimension Three: Historical Q&A Repository

Knowledge source: interactions in users’ historical Q&A records that are marked “correct.”

When a user asks a question and the AI provides a correct answer, meaning the user clicks “satisfied” or makes no correction, that Q&A is automatically added to the historical Q&A repository. This creates a continuously growing repository of correct-answer samples.

Indexed content: each record contains the original question, the intent understood by the AI, the generated query logic, the final result, and user feedback.

Retrieval strategy: when a new question arrives, retrieve semantically similar questions from the historical Q&A repository. If a historical record with high similarity is found, with similarity > 0.9, directly reuse its historical query logic and skip the LLM generation stage. This is both fast and accurate.

Injection strategy: if similarity is between 0.7 and 0.9, inject the historical Q&A as a reference case to guide the LLM to make appropriate adjustments based on historical logic. If similarity is below 0.7, do not inject it, to avoid misdirection.

2.4 Dimension Four: Data Schema Knowledge Base

Knowledge source: metadata for all datasets in Hengshi BI.

Indexed content: schema information for each dataset, including field names, field types, field meaning descriptions, relationships between fields, and data distribution characteristics such as enumeration value lists and value ranges.

Retrieval strategy: based on entities mentioned in the user’s question, such as “customer,” “order,” and “product,” retrieve schemas for datasets that contain those entities. If the question involves multiple entities, retrieve all related datasets and inject descriptions of their relationships.

Injection format: schema information is injected into the prompt as a concise table-structure description containing field name, type, meaning, and foreign-key relationships. For wide tables with more than 50 fields, inject only the subset of fields relevant to the current question to prevent the prompt from becoming too long.

3. Engineering Details of the RAG Pipeline

3.1 Three Stages: Retrieval, Reranking, and Injection

Hengshi ChatBI’s RAG pipeline is not as simple as “retrieve once and inject.” It uses three processing stages.

Stage 1: Multi-path recall

Retrieve across the four dimensions in parallel, with each dimension independently returning Top-K candidate results.
Vector retrieval uses Hengshi’s self-built embedding service, fine-tuned from the BGE-M3 model.
Keyword retrieval uses the BM25 algorithm as a supplement for exact matching.
Merge and deduplicate the results from both paths.

Stage 2: Reranking

Use a Cross-Encoder model to rerank the merged candidate results.
The reranking model considers deep semantic matching between the question and document, rather than relying only on vector similarity.
After reranking, take Top-N, typically N=5-10, as the final injected content.

Stage 3: Context assembly

Trim content against the prompt length budget. Total injected content does not exceed 4000 tokens.
Keep higher-ranking results first, and truncate lower-ranking results.
Label each injected item with its source type, metric definition, terminology, historical Q&A, or schema, to help the LLM distinguish information layers.

3.2 Embedding Model Selection and Fine-Tuning

General-purpose embedding models perform poorly in BI scenarios. They do not understand that “GMV” and “transaction amount” are synonyms, or that “South China” includes Guangdong, Guangxi, and Hainan.

Hengshi fine-tunes the BGE-M3 base model for the domain:

Training data: construct positive sample pairs from the Metric Management Platform and business terminology table, with “revenue” and “operating revenue” as a positive sample pair, and construct negative samples from random pairings.
Fine-tuning method: Contrastive Learning, which brings positive sample pairs closer and pushes negative sample pairs farther apart.
Result: after fine-tuning, the Top-5 recall rate of the embedding model on the internal test set increased from 72% to 91%.

3.3 Incremental Indexing and Real-Time Updates

The RAG knowledge base is not static. New metrics are continually registered, new terms are continually added, and historical Q&A continually accumulates. Hengshi’s incremental indexing mechanism includes:

Metric and terminology changes trigger real-time index updates, with latency < 5 seconds.
Historical Q&A is indexed in batches every day at midnight.
Schema changes, including added fields and changed types, listen for dataset metadata events and automatically trigger index updates.
The vector index uses the HNSW algorithm, supports efficient incremental insertion, and does not require a full rebuild.

4. Measuring the Effect of RAG Enhancement

4.1 Evaluation Metrics

RAG effectiveness cannot be measured only by whether an AI answer is correct. Retrieval and generation must be evaluated separately.

Retrieval metrics:

Recall@K: whether relevant documents are retrieved within Top-K
Precision@K: how many retrieved documents are truly relevant
MRR, Mean Reciprocal Rank: the rank position of relevant documents

Generation metrics:

Answer accuracy: the degree to which the AI’s final answer matches the standard answer
Hallucination rate: the proportion of information in the AI’s answer that conflicts with the knowledge base
Refusal rate: the proportion of cases in which the AI correctly says “I don’t know” when knowledge is insufficient

4.2 Measured Results Comparison

Hengshi conducted an A/B test on real data from 20 enterprise customers: RAG off versus RAG on.

Metric	RAG Off	RAG On	Improvement
Answer accuracy	68%	89%	+21pp
Metric definition error rate	22%	4%	-18pp
Schema misunderstanding rate	15%	3%	-12pp
Average response latency	1.2s	1.8s	+0.6s

Key finding: RAG delivered a 21-percentage-point increase in accuracy at the cost of an additional 0.6 seconds of retrieval latency. For BI scenarios, this latency trade-off is worthwhile. Accuracy matters more than speed.

4.3 The Flywheel Effect of Continuous Optimization

RAG systems have a natural flywheel effect: more user adoption → a richer historical Q&A repository → higher retrieval quality → higher accuracy → more user adoption.

Hengshi provides customers with a “RAG health dashboard” that shows:

The number of new historical Q&As added each week
Historical Q&A reuse rate, meaning how many new questions match historical records
Coverage of each knowledge-base dimension, such as the percentage of user questions covered by the metric library
Hallucination-rate trends

When the historical Q&A reuse rate exceeds 30%, the RAG system has entered a positive cycle. Most questions can then benefit from historical experience.

5. Common Pitfalls When Implementing RAG

Pitfall 1: Poor Knowledge-Base Quality, Garbage In and Garbage Out

Directly pour ungoverned metric definitions into the RAG knowledge base. Fields with the same name have different meanings in different tables, metric definition documents contradict one another, and terminology explanations are ambiguous. RAG retrieves “knowledge,” but that knowledge itself is wrong.

Avoidance: govern the knowledge base before launching RAG. Business teams must confirm metric definitions, terminology tables must be reviewed, and historical Q&A must be manually reviewed before entering the repository.

Pitfall 2: An Overlong Prompt Scatters the LLM’s Attention

Put all 20 retrieved pieces of knowledge into the prompt, making its total length exceed 8000 tokens. The LLM may exhibit “middle forgetting” in long contexts, ignoring information placed in the middle of the prompt.

Avoidance: control the total amount of injected content. Hengshi keeps total injected content below 4000 tokens and truncates it after sorting by relevance. It is better to inject less than to inject noise.

Pitfall 3: Using Only Vector Retrieval and No Keyword Retrieval

Pure vector retrieval does not match exact abbreviations, such as ARR, NRR, and ROI, well. These abbreviations may be semantically distant from their full phrases in vector space.

Avoidance: use dual-path recall, running vector retrieval and keyword retrieval in parallel and merging the results. Vectors handle semantic similarity, while keywords handle exact matching.

Pitfall 4: Ignoring Negative Injection in Retrieval Results

Retrieved knowledge is not always useful. Sometimes a retrieved metric definition is unrelated to the user’s question, but the LLM still refers to it and is misled.

Avoidance: explicitly mark the prompt with: “The following may be relevant reference information. Please determine whether it applies to the current question.” Give the LLM permission to ignore irrelevant information.

6. Conclusion

RAG is not merely a nice-to-have for ChatBI. It is essential. ChatBI without RAG is a “smart guessing machine.” It guesses correctly most of the time, but occasional mistakes carry an unacceptable cost in BI scenarios.

The core value of Hengshi ChatBI’s four-dimensional RAG architecture is that it turns the model’s implicit knowledge into the enterprise’s explicit knowledge:

Metric definitions no longer exist only in people’s minds. They reside in a retrievable knowledge base.
Business terminology no longer depends on a new employee’s learning curve. It is injected into the AI context immediately.
Historical Q&A is no longer consumed once. It becomes an asset that continuously increases in value.

When every correct answer from a BI system becomes the foundation for the next correct answer, the system is no longer merely a tool. It becomes an analytical partner that can grow.