Article body
Full article
Abstract: Enterprise data Q&A needs verification in its specific business context: are values correct, are definitions consistent, can the role access the relevant data, and do conclusions go beyond the evidence? Using HENGSHI SENSE metrics, datasets, knowledge management and Data Agent, teams can establish repeatable question sets and business benchmarks. This article explains how to design samples, distinguish scoring denominators, analyze errors, and feed evaluation results back into model and data-asset maintenance.
1 Define “correct” before calculating accuracy
“The ten product categories with the highest first-quarter sales in East China” appears simple, but it involves at least the sales definition, regional dimension, quarter boundaries, refund rules, sorting and permissions. Comparing only the final total can miss an incorrect definition. Checking only whether SQL or a chart was generated does not establish that the business answer is usable. Evaluation begins by writing the conditions accepted by the business into a verifiable record.
HENGSHI metric management provides a foundation. Business owners confirm metric meanings, data teams maintain calculation logic in models and HQL metrics, and charts and Q&A work from these definitions. Benchmarks can come from manually verified details, metric results and reports. Comparisons must fix the same period, granularity, role and data version. Comparing yesterday’s dashboard with today’s refreshed answer does not reveal the source of a discrepancy.
Correctness also includes how a question is handled. A question lacking information may need clarification first; unauthorized data should remain within access boundaries; and conclusions unsupported by current data need an explanation of the gap. Scoring all these samples by whether a number is returned rewards guessing and can misclassify appropriate clarification as failure. Decide the category of each question before it enters the test set.
2 Build a layered question set that reflects real use
| Question type | Example | Evaluation focus |
|---|---|---|
| Common operational query | This month’s East China net sales and store ranking | Consistent metric, period, filters, sorting and values |
| Combined analysis | Compare sales growth and refund changes by category | Consistent scope across metrics and explanations supported by data |
| Ambiguity or missing conditions | How is our profit doing? | Obtain necessary definitions first, or state the assumptions used |
| Permissions and organizational boundaries | A regional manager requests another region’s details | Returned objects, row scope and sensitive columns match authorization |
| Questions unsupported by current data | Predict next quarter’s profit using only an order table | Explain data and capability gaps without inventing a conclusion |
Samples can come from frequent business questions, historical anomalies and new scenarios. Include everyday single-metric queries as well as error-prone conditions such as month and year boundaries, empty results, tied rankings, refunds and duplicate joins. Start with one operational topic and have business, data and technical staff label the samples together. As the set grows, retain a stable benchmark subset and identify new scenarios separately.
Each sample should record at least the question, role, context, dataset or topic, metric definition, time range, expected result and judgment rules. For explanations of causes, list the data facts that must be checked and the conclusions that cannot be inferred directly, while allowing different wording and reasonable query paths. For multi-turn questions, also fix preceding context; otherwise, different conversation histories mean the comparison is no longer testing the same task.
Separate tuning samples from samples reserved for acceptance. If every knowledge and prompt adjustment targets the same failed questions, improved scores may only show greater familiarity with those phrasings. Retaining questions that were not used for tuning, and adding synonymous wording and adjacent dates, helps check whether improvements transfer to real use.
3 Measure business correctness, boundary handling and delivery efficiency separately
For directly answerable questions, a strict pass rule is advisable: the metric, period, scope, calculation and key explanations must all meet expectations. Agree in advance on monetary precision, rounding, time zones and null handling. Identical wording is unnecessary, while fluent language cannot compensate for incorrect numbers or definitions. Partially correct answers can be recorded separately but should not enter the strict pass count.
For clarification samples, assess whether necessary conditions were obtained. For permission samples, inspect the data actually visible. For out-of-scope samples, check whether the gap is accurately explained. Report the sample count and pass count for each category before deciding whether to aggregate. For security samples in particular, finding no unauthorized access in one round only means no issue was found within that coverage; it cannot establish safety for every phrasing and integration.
Record efficiency alongside correctness, including time to first visible response, time to complete analysis, successes and failures. With enough samples, report the median and P95. Count model calls, tokens or runtime resource costs only when reliable records are available. Also retain sample size, test window and deployment conditions so that one fast demonstration is not mistaken for stable performance.
4 A fictional evaluation record: why an overall score cannot replace category results
The following fictional evaluation record illustrates scoring only and does not represent measured HENGSHI product performance. A retail team labels 100 questions in advance: 60 directly answerable, 20 requiring clarification, 10 checking permission boundaries, and 10 outside the currently published data scope. Every question has one primary category to avoid counting it in multiple denominators.
| Pre-labeled category | Samples | Correctly handled | Result for this round |
|---|---|---|---|
| Directly answerable | 60 | 54 | 54 ÷ 60 = 90% strict business-answer pass rate |
| Clarification required | 20 | 18 | 18 ÷ 20 = 90% correct handling of necessary conditions |
| Permission boundaries | 10 | 10 | All 10 met the expected authorized scope in this round |
| Outside current scope | 10 | 9 | 9 ÷ 10 = 90% correct explanation of gaps |
| Total | 100 | 91 | 91 ÷ 100 = 91% overall correct-handling rate |
The 91% here is not “numerical answer accuracy.” It is an overall correct-handling rate that includes clarification and boundary handling. Of the 60 directly answerable questions, suppose four have definition or numerical errors and two fail during execution: only 54 still pass strictly. Permission checks should also specify datasets, roles, row scope and sensitive columns, rather than merely checking whether an answer says “no permission.”
These results support different corrections: trace numerical errors to metrics and data; address clarification failures through terminology and question guidance; investigate connections and call environments for execution failures; and revisit knowledge scope and integration delivery agreements when the system insists on answering unsupported questions. Aggregate scores help show trends, while category results explain what to improve next.
5 Use HENGSHI data assets to carry evaluation improvements
HENGSHI SENSE dataset knowledge management can maintain descriptions and business meanings and trigger intelligent learning; people need to inspect and adjust the results. For questions that consistently select the wrong object, supplement dataset purposes, field descriptions and synonyms. For complex calculations, prioritize reusable metrics and explain statistical conditions in their descriptions. The corrected objects can then continue serving dashboards, self-service analysis and Q&A.
Permission issues require returning to actual objects and usage patterns. Rules apply at different points depending on whether an application uses author, dataset-author or user data permissions. Retest using ordinary business roles and boundary roles, checking actual results in both Q&A and embedded entry points. Added related objects, organizational changes or sensitive-column changes should bring the corresponding samples into the change’s verification scope.
For model changes, record the model service and version, prompt version, data scope, conversation-history conditions and relevant call settings. HENGSHI provides general model configuration that project teams can use to fix experimental conditions. Change one main factor at a time and compare with the same question set. If both the model and metric definitions change, validate them separately rather than attributing all gains from corrected data to the model.
6 Place regression checks where actual changes occur
Metric updates, field renaming, dataset changes, model replacement and permission changes can all affect existing answers. Teams can select regression samples by dependency scope and then spot-check other frequent questions. Retain at least before-and-after results, the reasons for differences and human conclusions. A successful new version should rest on correct business definitions and expected permissions.
A project can incorporate this evaluation approach into its own acceptance process or development pipeline, but methodological advice should not be presented as a fully automated scoring service included by default in the platform. Teams should also organize and verify production feedback. A follow-up question may signal deeper analysis or dissatisfaction, and an absence of follow-up cannot automatically count as correctness. New questions belong in future samples only after review.
HENGSHI provides a foundation for connecting business questions to governed metrics, datasets, knowledge and analytics pages. Establishing benchmarks from these objects and continuously recording results, errors and changes turns answer quality from a demonstration impression into reviewable business evidence. Enterprises can compare individual model adjustments and also observe the long-term benefits of data governance and analytics-asset maintenance.