Article body
Full article
When an operations leader asks, “What is the renewal rate for customers whose contracts expire this quarter, and how far is it from the same period last year?”, the team can easily treat it as a request for a number. It is actually a business definition that must be confirmed: who counts as an expiring customer, who belongs in the denominator, whether renewal is confirmed by contract, payment, or service activation, whether a quarter follows calendar months or a fiscal year, and how merged accounts are deduplicated. Until the definition is settled, even a chart that returns a number is hard for the business to verify.
HENGSHI ChatBI can help users analyze data in conversation, but answer quality still depends on whether datasets, fields, metrics, and business knowledge are well prepared. Using renewal rate as an example, this article explains how a team can turn a frequent business question into maintainable semantic preparation and revalidate it with a question set after changes. The regression question set and acceptance steps described here are team-governance practices, not built-in automated testing features of the product.
1. Write “renewal rate” as an auditable business question
The business owner, data owner, and metric maintainer should first agree on a definition that can be restated. Suppose the team agrees to calculate by customer: the denominator is formal customers with contracts expiring in the second quarter, excluding trial, internal, and test contracts; the numerator is customers among them whose renewals are confirmed within 30 days after expiry, with early renewals included; if the same customer has multiple expiries in the quarter, count the customer once and use that customer’s first expiry date in the quarter to define the observation window. Set the data cutoff to July 31 so that the grace periods for every cohort expiring in the quarter have ended before calculating the complete quarterly result. When the denominator is zero, mark the result as having no calculable sample; it must not be interpreted as a zero renewal rate. This is only an example. Each enterprise needs to replace it with its own contract rules and customer master data.
| Semantic item to confirm | Specific decision for renewal rate | Evidence retained for acceptance |
|---|---|---|
| Unit of analysis | Count by customer, contract, or account; whether multiple accounts of a group customer are consolidated | Customer primary key, consolidation rules, and deduplication key |
| Numerator and denominator | Whether the denominator is limited to expiring customers; whether renewal is confirmed by signing, payment, or activation date | Business policy, status fields, and grace period |
| Time attribution | Attribute by expiry date, renewal date, or fiscal year; how backfilled data is handled | Time field, fiscal calendar, and data cutoff date |
| Identity and scope | Whether to exclude trial, internal, test contracts, and regions outside the user’s permissions | Filter conditions, user attributes, and permission scope |
The team can write the confirmed question as: “By contract expiry date, calculate the renewal rate for formal East China customers whose contracts expired in the second quarter, using data through July 31, 2026 and the exclusion rules, customer deduplication, and 30-day observation window above.” That sentence fixes the metric, time, scope, and filters together. When business users say “renewal proportion” or “existing-customer retention,” the team can use it to determine whether those phrases map to the same metric. If they cannot, the questioner should choose the definition.
2. Carry the confirmed definition through metric and dataset knowledge
HENGSHI metric management supports aggregation calculations defined for datasets or models, along with aliases, descriptions, groups, and display formats. A metric’s calculated value changes with the dimensions and filters used. Therefore, “renewal rate” cannot be only a name: its description should also state the numerator, denominator, time attribution, deduplication, and exclusions, so that business users can understand why a regional chart and an industry chart show different values.
The Data Agent guide explains that teams can use dataset knowledge management to describe, in natural language, a dataset’s purpose, implicit filter rules, synonyms, mappings between specialized terms and fields, and mappings among metrics. A team can record the relationship of “renewal payment,” “contract renewal,” and “renewal” to the governed metric in knowledge management, map “this fiscal year” to the agreed calendar, and retain business explanations in field or metric descriptions. These provide retrieval and interpretation cues to the model; the business owner must still confirm the rules themselves.
Teams can begin by maintaining a small number of frequent, disputed metrics, then expand coverage. Add descriptions for fields such as “contract expiry date,” “renewal confirmation date,” and “unique customer identifier,” and hide temporary fields that should not participate in Q&A, so similarly named fields are not exposed at the same time. If the team needs to configure industry terminology, resource-search rules, or synonym handling further, it can refer to the relevant configuration in Data Agent tuning. Keep question-set revalidation records before and after configuration; prompt adjustments must not be treated as approval of a metric definition.
3. Make clarification and permissions part of asking questions
“Which region has the lower renewal rate?” may also be missing a time range, customer segment, or comparison baseline. The team can prepare a clarification pattern for these common ambiguities: ask for the specific periods for the current and previous periods; confirm whether a region means customer location or sales organization; and confirm whether the comparison is against a target, the same period last year, or the whole company. When a business term has more than one approved definition, the question interface or usage guidance should let the user choose; the model should not decide on its own.
Permissions change the visible scope and can also change the number. Data Agent documentation explains that the Agent retrieves and analyzes data within the range a user is permitted to access. Acceptance reviewers need to verify the result with accounts that have different regional, departmental, or role permissions: can the same governed metric be calculated within each legitimate scope, and can restricted objects be prevented from appearing through totals, details, or follow-up questions? The team should put permission validation and metric validation in the same release check.
Official guidance also notes that large-model behavior is uncertain: the same input is not guaranteed to generate exactly the same response. Teams cannot promise that every conversation will use identical wording, follow-up paths, or presentation. Acceptance should focus on what can be audited: whether the model reaches an approved metric and scope, whether the result can be reconciled to a manual calculation or governed chart, and whether ambiguous questions receive explicit clarification.
4. Maintain a question set rather than saving only a few successful prompts
Leaving a screenshot of one successful conversation in a group chat cannot cover the effect of later synonyms, changes to a metric description, or replacement of an upstream field. A team can maintain a small regression question set and associate every question with a metric version, dataset version, reviewer, and comparison result. The team can maintain this acceptance record with its own tools and processes.
| Question type | Example question | Acceptance focus |
|---|---|---|
| Standard definition | Renewal rate of formal East China customers in Q2 2026 | Reaches the approved metric, time attribution, and customer-deduplication rule |
| Synonymous phrasing | Existing-customer renewal-payment proportion in East China | Maps to the same governed metric, or asks the user to confirm the difference |
| Time ambiguity | Renewal performance this fiscal year | The fiscal-year definition, cutoff date, observation window, and year-over-year baseline are visible |
| Permission boundary | Renewal rate for my region | Uses only the regions and customer scope visible to the current account |
| Incomplete information | Is renewal going well? | Requests a time period, comparison baseline, or business scope |
At every revalidation, the reviewer records the original question, account role, current data cutoff date, expected definition, the dataset or metric actually reached, and the manual comparison result. If the output cannot be reviewed, the team should first distinguish among a missing definition, unclear synonym mapping, a changed permission scope, or lagging data updates, then have the corresponding owner address it. These question records also turn new business phrasing into candidates for the next round of semantic preparation.
5. Revalidate after changes to give the business a traceable answer
Changes to metric definitions, dataset knowledge, field names, upstream schemas, refresh plans, or role permissions can all affect answers. HENGSHI dataset-update documentation explains that dataset configuration needs updating when the definition of a source table changes; when data imported into the HENGSHI engine changes, the dataset also needs updating to keep analysis current. After those actions, the team should sample and reaccept the affected question set instead of checking only whether a page opens.
The change record should at least state the owner, reason for change, effective time, affected metrics, and revalidation conclusion. If renewal rate changes from “deduplicated by customer” to “calculated by contract,” the business owner should approve the new definition, the data owner should update the metric and knowledge description, and the reviewer should use a historical slice to explain the numerical difference. For questions that cannot be answered, the team should record the missing fields, unapproved definitions, or permission restrictions and return them for business confirmation.
ChatBI creates value when business users can consistently ask for data that can be verified. That requires a team to maintain metric definitions, semantic descriptions, permission scopes, and regression questions. Establish high-frequency metrics such as renewal rate first, then extend the practice to churn, repeat purchase, and customer value, so the Q&A experience has a stable business foundation.