Data Agent Evaluation and Benchmarking: HENGSHI's Quality Measurement and Regression Testing Framework for Analytics Agents

Article body

Full article

15 min read

Introduction

“Our Data Agent is live.” This sentence is becoming more common inside enterprises. The next questions, however, are often left unanswered: Is it actually good? How do we measure it? Is it better or worse than last month? If we change the model, did it improve or regress?

This is a seriously underestimated engineering problem. In the LLM era, BI Agent behavior is probabilistic. It changes with model versions and drifts with prompt tuning. Without a scientific evaluation system, enterprises operate Agents blindly: the Agent can fly, but nobody knows whether it is stable or heading in the right direction.

HENGSHI Data Agent builds an evaluation framework covering four dimensions: accuracy, efficiency, cost, and safety. It also includes reusable benchmark datasets and regression testing mechanisms. This article explains the system that makes Agents measurable, comparable, and continuously optimizable.

1. Why Is Agent Evaluation Harder Than Traditional Software Testing?

1.1 Traditional Evaluation Breaks Down

Traditional software evaluation relies on assertions: input A should produce output B. If it does not, the test fails. The result is deterministic: right or wrong.

Agent evaluation deals with a fuzzy space:

The same question may be solved through different correct paths.
“Correct answer” itself must be defined: the number, metric definition, and interpretation all have to be right.
Results are probabilistic, so a single run cannot represent stable performance; repeated sampling and statistics are needed.

1.2 Three Costs of Missing Evaluation

Blind iteration: after changing a model or prompt, you do not know whether the system improved or degraded. You can only rely on intuition.
Unclear vendor or model selection: when facing GPT-4, Claude, DeepSeek, local models, and other choices, you do not know which performs better for your business.
Production incidents: without regression testing, a model update can silently reduce accuracy for a class of questions, and the team only discovers it after users complain.

2. A Four-Dimensional Evaluation Framework

HENGSHI evaluates Data Agent performance across four dimensions, each with quantitative metrics.

2.1 Accuracy — Is the Answer Correct?

Accuracy is the core, and it can be further decomposed:

Task Success Rate: the percentage of benchmark cases where the Agent ultimately gives a correct and usable answer. This is the most important north-star metric.
Step Accuracy: in multi-step tasks, whether each tool choice and generated parameter is correct. Step accuracy helps pinpoint where the failure occurred.
Definition Accuracy: whether the metric definition, time window, and filters match the user’s intent and business rules. This is heavily weighted in industries such as finance.
Hallucination Rate: the percentage of output that cannot be derived from tool results or is fabricated. The target should approach zero.

2.2 Efficiency — Is the Answer Fast Enough?

Average step count: the average number of ReAct loops needed to complete a task. Too many steps indicate inefficient planning or reflection.
End-to-end latency: total time from question to final answer, including LLM reasoning, tool execution, and reflection overhead.
First response time: the delay before the user senses that the Agent has started working, which strongly affects experience.

2.3 Cost — Is the Answer Economical?

Average token consumption: input plus output tokens per interaction, the direct cost of model calls.
Tool invocation cost: some tools, external APIs, or compute resources have their own cost and should be included.
Cost-to-accuracy ratio: pursuing accuracy alone can raise cost sharply. HENGSHI focuses on the accuracy improvement gained per unit of cost to guide model selection and strategy optimization.

2.4 Safety — Is the Answer Compliant?

Permission overreach rate: whether the Agent attempts to access data it is not authorized to see. The ideal is zero overreach and zero missed blocks.
Appropriate refusal rate: whether the Agent properly refuses out-of-bound or high-risk questions instead of answering anyway.
Sensitive information leakage rate: whether answers accidentally expose another user’s or region’s data.

3. Benchmark Dataset Design

3.1 Where Do Test Cases Come From?

HENGSHI’s benchmark datasets are not invented from scratch. They are distilled from real business:

Historical trace mining: extract frequent and representative questions and tasks from anonymized production traces.
Scenario template expansion: manually construct cases around financial, retail, manufacturing, and other industry templates to cover different subtypes.
Adversarial examples: deliberately create traps that are easy to get wrong, such as ambiguous phrasing, long-tail dimensions, and cross-source definition conflicts, to stress-test robustness.

3.2 Layered Test Structure

The benchmark is layered by difficulty and type, making it easier to locate capability gaps:

L1 single-step query: one sentence directly leads to an answer, such as “yesterday’s total sales revenue.” Tests basic understanding and execution.
L2 multi-step analysis: requires planning multiple steps, such as “top 5 products in East China and their year-over-year growth.” Tests planning and tool orchestration.
L3 complex reasoning: involves attribution, comparison, or hypothesis testing, such as “why did yield drop?” Tests reflection and validation.
L4 adversarial robustness: ambiguity, omission, and traps. Tests disambiguation and refusal boundaries.

3.3 Annotation and Scoring Rubric

Each test case includes a standard answer and a scoring rubric:

The standard answer contains the correct metric, dimension, definition, and expected conclusion.
The rubric defines how to judge fully correct, partially correct, and incorrect answers. For example, an answer with the right number but the wrong definition is considered partially correct with points deducted, rather than treated as a binary failure.

4. Regression Testing: Making Every Change Verifiable

4.1 Why Regression Testing Is Necessary

Any change to a Data Agent’s components, including model version, prompt, semantic layer, or toolset, can affect overall behavior. Regression testing ensures that changing one part does not silently break another.

4.2 Regression Testing Workflow

HENGSHI embeds regression testing into Agent iteration:

Change trigger: when a model, prompt, or semantic layer changes, regression testing is triggered automatically.
Batch run: the Agent reruns against the benchmark dataset and collects the four-dimensional metrics.
Difference comparison: compare with the previous baseline and produce reports such as “accuracy +2.1%, latency -8%, cost +5%, hallucination rate 0%.”
Gate decision: apply quality gates, such as “task success rate must not drop by more than 1% from baseline” and “hallucination rate must not increase.” If the gate fails, release is blocked and optimization resumes.

4.3 Integration with CI/CD

In previous CLI and CI/CD articles, we explained how HENGSHI makes BI development pipeline-driven. Agent regression testing follows the same principle. Every MR that changes prompts or the semantic layer can trigger the regression suite automatically, moving quality assurance left into development instead of waiting for users to report issues after release.

5. Online Evaluation: Continuous Measurement in Production

5.1 Limits of Offline Evaluation

Benchmarks are offline and controlled, but real user questions are diverse and often outside benchmark coverage. A high offline score does not guarantee production reliability.

5.2 Online Evaluation Signals

HENGSHI collects three types of production signals as a supplement to offline evaluation:

Explicit feedback: likes, dislikes, and correction behavior such as “no, I meant…” Negative feedback and corrections are strong signals and feed directly into quality measurement.
Implicit signals: whether the user repeatedly asks the same question, which may indicate a weak first answer, or whether the user continues to drill down, which may indicate the first answer was useful.
Exception interception rate: the number of unauthorized access attempts blocked by the sandbox, reflecting the online safety level.

5.3 Feedback Loop into Optimization

Online signals flow back into the benchmark dataset. High-frequency questions that users downvote are converted into standard test cases, preventing similar issues from recurring. This creates a continuous loop: production issue → offline test case → iterative optimization → regression validation → release.

6. Quantitative Basis for Model Selection

6.1 No More Choosing by Reputation

With an evaluation framework, model selection becomes a quantitative experiment:

Run GPT-4, Claude, DeepSeek, and local models on the same benchmark.
Compare the four-dimensional metrics and plot accuracy-cost tradeoffs.
Choose based on business priorities: accuracy, cost, or whether data is allowed to leave the domain.

In practice, a model that performs strongly on general benchmarks may perform weakly on a financial definition task in your business. That is the value of evaluation: test it on your business, not on someone else’s leaderboard.

6.2 Evidence for Hybrid Routing

HENGSHI supports multi-model routing: cheaper models for simple tasks, stronger models for complex ones. How should the routing boundary be set? The evaluation framework provides the answer. Test each model’s accuracy and cost across L1-L4 tasks, then build routing rules based on data rather than intuition.

7. Implementation Recommendations

7.1 Start with a Small Dataset

There is no need to build a huge benchmark from day one. Start with 50-100 real high-frequency questions, run the evaluate-compare-regress loop, then expand the dataset as the business grows.

7.2 Look at Metrics by Layer

Do not rely only on task success rate. An Agent with high success rate and high hallucination rate is more dangerous than one with slightly lower success rate and zero hallucination. The four dimensions should be considered together, with weights adjusted by industry. Finance emphasizes definitions; retail emphasizes efficiency.

7.3 Keep Evaluation Cost Under Control

Full regression consumes LLM calls. HENGSHI uses layered sampling: daily iterations run a fast L1-L2 subset, while major versions run the full L1-L4 suite. This balances feedback speed and evaluation cost.

8. Technical Comparison

Dimension	No Evaluation	Manual Sampling	Systematic Evaluation (HENGSHI)
Accuracy measurement	Intuition	Subjective sampling	Quantified benchmark
Change impact	Unknown	Discovered after the fact	Regression gate blocks release
Model selection	Reputation	Trial and error	Data-driven comparison
Continuous optimization	Random	Passive	Feedback-loop driven

9. FAQ

Q1: How much effort does it take to build a benchmark dataset?

The starting cost is low. Extracting 50-100 high-frequency questions from real traces and annotating standard answers can take half a day to one day. The key is to establish the loop first, then expand continuously.

Q2: Can evaluation results be used in external messaging?

Yes, and they should be. The four-dimensional metrics generated by HENGSHI’s framework are objective evidence of product capability and are more persuasive than vague claims that “we are intelligent.”

Q3: Do small teams need such a heavy evaluation system?

“Heavy” is relative. Small teams can use a lightweight version: one test set, one comparison run, and manual review of differences. That is already enough to avoid flying blind. The value of the system is that it exists; it does not need to be complete from day one.

10. Conclusion

Launching a Data Agent is not the finish line. It is the starting point for being measurable, optimizable, and provable. Through a four-dimensional evaluation framework + layered benchmark dataset + regression testing gates + online feedback loop, HENGSHI turns Agent quality from guesswork into engineering.

The core insight is: what cannot be measured cannot be optimized; what cannot be regression-tested cannot be iterated safely; what cannot be proven cannot be trusted. When an enterprise can objectively answer whether its Agent is good, whether it is better than last month, and where it regressed, the Data Agent becomes a manageable and accountable digital asset.

With this article, we complete a six-article arc from ChatBI’s ability to understand, converse, and stay trustworthy to the Data Agent’s ability to correct, execute, and be measured. It extends the 55 HENGSHI technical articles and adds the latest view of next-generation intelligent analytics.