Article body
Full article
RAG solves an important problem: a language model cannot learn a company’s private warehouse schemas, field names, and metric definitions during general training. At query time, the system retrieves relevant schema and business knowledge and supplies that context to the model. Retrieval improves the chance of selecting the right objects, but it cannot guarantee a correct analytical answer.
An enterprise query can fail before retrieval, during retrieval, at generation, or after execution. Trustworthy ChatBI turns the complete chain into observable and testable control points.
What RAG Solves and What It Does Not
A typical flow understands the question, retrieves knowledge, assembles context, generates a query, executes it, and explains the result. RAG avoids filling the model context with hundreds of tables and thousands of fields. It also lets teams update knowledge after a schema change without retraining the model.
RAG does not automatically resolve these conditions:
- the knowledge base contains both approved definitions and obsolete drafts;
- the most similar dataset is outside the current user’s permissions;
- “revenue” lacks a tax, currency, organization, or time definition;
- the SQL is valid, but its aggregation grain or magnitude is wrong;
- a user correction remains in chat history instead of updating a governed asset.
Vector retrieval is one part of trustworthy analytics, not its final guarantee.
Guardrail One: Retrieve Governed Assets
The knowledge base should contain maintained dataset documentation, metric definitions, synonyms, lineage, and example queries. Each knowledge unit needs a version, owner, business domain, effective period, and permission labels. Schema and metric changes should update dependent indexes, while obsolete versions leave the default retrieval set.
Embedding every DDL statement, chat transcript, and historical document mixes drafts, duplicate definitions, and retired fields. Knowledge quality must be governed before it enters the vector store. The model should not be expected to repair it during inference.
Guardrail Two: Combine Semantic Retrieval with Structural Filters
Vector similarity finds concepts with related meaning, but it does not inherently respect tenants, permissions, data grain, or version boundaries. A safe retrieval flow first restricts the candidate set with structured policy, then performs semantic retrieval and reranking.
| Filter | Purpose |
|---|---|
| Tenant and user policy | Exclude invisible datasets, fields, and metrics |
| Business domain and current application | Reduce false matches across domains |
| Data version and effective period | Avoid obsolete schemas and definitions |
| Metric grain and compatible dimensions | Prevent invalid aggregation and joins |
| Language and terminology mapping | Map Chinese and English questions to the same object |
Permission filtering belongs in the retrieval service. Masking the answer after generation is too late.
Guardrail Three: Disambiguate Definitions Before Generation
If the agent finds several revenue metrics, it should show candidate definitions or narrow the set using the current application and user context. It must ask when time, organization, currency, tax treatment, or aggregation grain changes the answer.
Disambiguation does not require a technical form. The system can surface only the decisions that matter, such as order date versus payment date or gross versus net revenue. One short confirmation can prevent an entire report from being rebuilt.
Guardrail Four: Validate the Executed Result
A syntactically valid query can still produce a wrong business result. Different checks belong at different stages:
| Stage | Example checks |
|---|---|
| Before execution | Read-only policy, table and field allowlists, query cost, tenant filters |
| After execution | Empty result, magnitude, time range, aggregation grain, outliers |
| Before explanation | Evidence supports the conclusion; definitions and filters are disclosed |
| High-risk metrics | Cross-check against certified reports or published metric results |
An anomalous result should enter correction or human review instead of being converted into a confident statement. The execution identity, query text, and validation outcome must remain bound to the same task record.
Guardrail Five: Feed Corrections into Governance and Audit
A user correction should not remain a chat memory. The team must classify the failure as retrieval, missing semantic definition, query generation, or data quality, then update the knowledge base, metric platform, example library, or test set.
Audit records should include the question, retrieved knowledge, metric version, generated query, execution identity, validation result, and final output. These records support an offline evaluation set that measures retrieval recall, definition accuracy, execution success, and validation interception.
Choose Models After the Engineering Controls
A larger model may improve language understanding and complex reasoning, but it cannot repair inconsistent metric definitions. Build governed assets, permission-aware retrieval, disambiguation, and result validation before comparing model cost, latency, and deployment options.
Private deployments also need versioned embedding models, vector-index backups, offline update procedures, and recovery plans. Models can change; knowledge versions and evaluation baselines must remain. Trustworthy ChatBI is a property of the complete system.
Further reading: HENGSHI SENSE Product and Technology White Paper.