Article body
Full article
Introduction
Traditional BI monitors systems: CPU utilization, query latency, and error rates. These metrics measure infrastructure health.
AI Agents change the monitoring target from systems to Agent behavior. An Agent can use little CPU, run fast queries, and return no errors while still producing a wrong analysis. Traditional monitoring cannot capture this quiet failure.
AI Agent observability was one of the most active topics in AI engineering in 2025 and 2026. It addresses a core question: after an Agent completes multi-step reasoning, calls several tools, and produces a natural-language answer, how do you determine whether its answer is sound? If it fails, which step caused the problem?
HENGSHI BI provides an Agent observability framework that covers the full reasoning path, from Trace capture, Token monitoring, and tool-call audits to confidence assessment. This article examines its technical architecture.
1. Why AI Agents Need Dedicated Observability
1.1 Blind Spots in Traditional Monitoring
Traditional BI monitoring focuses on system-level metrics:
- Did the query return an HTTP 200 response?
- Did the query finish within its threshold, under 3 seconds?
- Are server resources healthy, with CPU below 80%?
AI Agents introduce a different failure mode: the system works while the result is wrong.
An Agent may choose the right tool, use valid parameters, execute its query, and receive real data. It can still misinterpret the data, select the wrong aggregation dimension, or omit a critical filter. The monitoring dashboard can remain green while the user receives a wrong answer.
1.2 Three Layers of Agent Observability
Agent observability needs to cover three layers:
L1: Infrastructure. This layer extends traditional monitoring with Agent runtime CPU, memory, and GPU utilization, API latency, and network I/O. It provides a necessary baseline.
L2: Reasoning behavior. This Agent-specific layer tracks reasoning steps, tool-selection sequences, parameter quality, intermediate observations, and termination decisions. It exposes quiet failures.
L3: Result quality. This layer evaluates the final output through answer accuracy, user satisfaction, hallucination rate, and refusal rate. It measures the value delivered by an Agent.
HENGSHI BI’s observability framework covers all three layers, with the main investment in L2 and L3 because they address monitoring needs unique to the Agent era.
2. HENGSHI Agent Observability Architecture
2.1 Trace Capture: Visualizing the Full Reasoning Path
A Trace is the central concept in Agent observability. It records the complete execution path of an Agent interaction from start to finish.
HENGSHI BI extends OpenTelemetry’s Span model for its Trace data model. Each Trace includes the following elements:
Trace root node: records the user’s original question, session ID, user ID, start time, and total duration.
Reasoning Span: each ReAct loop creates a reasoning Span containing:
- Thought text, or the LLM’s reasoning content
- Action call, including the tool name and parameters
- Observation result, a summary of the tool response
- Span duration, covering LLM reasoning and tool execution for the loop
Tool execution Span: each tool call creates a child Span containing:
- Tool name, input parameters, and output
- Execution duration, success status, and error details where applicable
- Resource consumption, including query rows and returned data volume
LLM call Span: each LLM inference creates a child Span containing:
- Model name, Prompt length, and Completion length
- Token consumption, including input and output Tokens
- Inference latency and temperature setting
- Finish Reason: normal completion, length truncation, or content filtering
These Spans form a tree that reconstructs the Agent’s reasoning path in full.
2.2 Trace Storage and Queries
Trace data volumes can grow fast. One Agent interaction can produce 5 to 10 Spans, and the Prompts and result data in each Span can range from several KB to tens of KB.
HENGSHI uses the following storage policy:
Hot data, 7 days: stored in Elasticsearch for real-time queries and aggregation analysis. Administrators can filter the Trace search interface by user, time, tool name, success rate, and other dimensions.
Cold data, 90 days: archived to object storage and partitioned by date for long-term trend analysis and model tuning.
Sampling policy: the default records all Traces. High-traffic deployments can configure sampling rates: sample 10% of successful interactions and 100% of failed interactions because failures offer more diagnostic value.
2.3 Real-Time Monitoring Dashboards
HENGSHI provides real-time Agent observability dashboards with the following core views:
Overview dashboard:
- Total Agent interactions today, success rate, and average latency
- Total Token consumption and trend by hour
- Usage share and accuracy comparison for each model
Reasoning-path dashboard:
- Average reasoning-step distribution: 2 or fewer steps, 3 to 5 steps, and 6 or more steps
- Tool-call popularity ranking, showing the most-used tools
- Tool-failure ranking, showing tools that fail most often
Anomaly dashboard:
- Real-time alert stream for reasoning timeouts, consecutive tool failures, and hallucination-check hits
- Hallucination-rate trends by model and time
- Low-confidence answer list for human review
3. Confidence Assessment: Determining Whether an Agent Answer Is Correct
3.1 Why Confidence Is Needed
Traditional monitoring can show whether a system meets operational expectations. It cannot show whether an answer is correct. The core challenge in Agent observability is to evaluate the reliability of an Agent response without relying on human review.
HENGSHI assigns each Agent response a confidence score from 0 to 100 to help determine whether the answer is trustworthy.
3.2 Combining Multiple Confidence Signals
Confidence is a weighted combination of several signals:
Signal 1: Tool execution success rate. Confidence falls if several Agent tool calls fail or need retries. A clean execution path offers stronger support than one with repeated retries.
Signal 2: Data completeness. Confidence falls if the data behind an Agent response has gaps, such as an empty query result or data excluded because of insufficient permissions. Incomplete data cannot support a reliable answer.
Signal 3: Reasoning consistency. Does the Agent’s final conclusion match its intermediate reasoning? A separate LLM compares the Thoughts and Final Answer. If the conclusion says sales declined while the intermediate evidence shows sales increased, the consistency score falls and the system lowers confidence.
Signal 4: Knowledge-base citation quality. In RAG scenarios, did the Agent cite knowledge fragments with high relevance? A large semantic distance between retrieved knowledge and the question suggests the Agent may have forced an answer, so the system lowers confidence.
Signal 5: Historical pattern matching. Confidence rises if the current question has a strong resemblance to historical question-and-answer pairs with high satisfaction and the Agent follows a similar reasoning path.
3.3 Low-Confidence Handling
If confidence drops below the threshold, 70 by default, the system uses the following handling strategies:
Strategy 1: Warning label. The system adds a notice such as, “This answer has low confidence. Please review it by hand,” so the user can handle it with care.
Strategy 2: Proactive clarification. The Agent adds a clarifying question before answering: “Does ‘last month’ mean the calendar month or the fiscal month? The choice affects the calculation.” The clarification removes ambiguity and improves confidence in the next response.
Strategy 3: Human handoff. Questions involving material decisions go to a human analyst for review when confidence is below 40. The system attaches the complete reasoning Trace so the analyst can assess the case with full context.
4. Token Cost Monitoring and Optimization
4.1 Visualizing Token Consumption
LLM cost scales with Token consumption. Agent observability therefore needs Token-level monitoring:
By model: models carry different unit costs. GPT-4 costs more than 20 times as much as DeepSeek. The dashboard shows each model’s share of Token consumption and cost.
By function: which functions consume the most Tokens? Complex analysis with many reasoning steps, such as multi-round ReAct loops, can consume the most; simple queries consume the least.
By user or tenant: which tenants and users show elevated Token consumption? Is there unreasonable misuse?
4.2 Cost Optimization Methods
Based on Token monitoring data, HENGSHI supports the following optimization methods:
Small-model routing: route simple queries, such as single-turn questions and parameter extraction, to lower-cost models such as DeepSeek-V3. Route complex reasoning, such as multi-step attribution analysis, to stronger models such as GPT-4. Model routing can reduce overall Token costs by 40% to 60%.
Context compression: as described in the ReAct article, context compression across multi-round loops can reduce Prompt Token consumption in each round.
Cache-hit optimization: Embedding caches and search-result caches for high-frequency questions reduce duplicate reasoning. Each 10-percentage-point gain in cache-hit rate reduces Token cost by about 5%.
Prompt streamlining: review Prompt templates on a schedule and remove redundant examples and instructions. A Prompt template shortened by 20% can produce substantial savings in high-traffic environments.
5. Hallucination Detection and Quality Gates
5.1 Hallucination Detection Methods
Hallucinations present the most dangerous problem in Agent answers: an Agent provides information that appears plausible but is wrong. HENGSHI uses three detection methods:
Method 1: Factual consistency checks. The system breaks the Agent’s final response into individual factual statements and compares each with the raw data returned by tools. It marks a statement as a suspected hallucination if the raw data does not support its numbers, entities, or relationships.
Method 2: Knowledge-base consistency checks. In RAG scenarios, the system checks whether the answer aligns with the provided knowledge fragments. If an answer states that “gross margin = gross profit / revenue” while the knowledge base defines “gross margin = (revenue - cost) / revenue,” the system marks the answer as inconsistent.
Method 3: Numeric plausibility checks. The system checks key figures in an answer for plausibility. For example, a 500% month-over-month sales increase without a major promotion may need review.
5.2 Quality Gates
The hallucination-check result functions as a quality gate before the Agent returns an answer to the user:
- No hallucination-check hits: return the response.
- Low-severity hit, where the wording lacks precision but the conclusion is correct: return the response with a notice.
- High-severity hit, where the conclusion is wrong or carries serious risk of misleading the user: block the response and trigger the low-confidence handling strategy.
Teams target a quality-gate interception rate of 2% to 3%. A higher rate indicates an Agent quality issue; a lower rate indicates that the gate allows too much through.
6. Feedback Loops and Continuous Tuning
6.1 Collecting User Feedback
Observability ends with using data to improve the Agent. HENGSHI’s feedback loop includes:
Explicit feedback: after each Agent response, users can submit a thumbs-up or thumbs-down rating. A negative rating triggers root-cause analysis. The system analyzes the response Trace to determine whether the issue came from tool selection, parameter extraction, or reasoning logic.
Implicit feedback: user behavior also provides feedback signals. If a user rephrases and asks the question again, the prior answer did not meet the need. If the user copies a number from the answer to verify it elsewhere, that suggests high trust. If the user closes the conversation, the answer may have disappointed them.
Correction behavior: if a user changes an Agent-generated chart configuration or analysis conclusion by hand, the system records the before-and-after difference to improve the Agent’s default behavior.
6.2 Tuning Loop
Feedback data supports continuous Agent tuning:
Prompt optimization: recurring error patterns prompt updates to Prompt templates. If an Agent often calculates month-over-month growth as year-over-year growth, the Prompt can add an explicit instruction: “Month-over-month = current period / prior period - 1; year-over-year = current period / same period last year - 1.”
Tool-description optimization: repeated misuse of a tool calls for a revised description with applicable and inapplicable scenarios.
Model-routing optimization: if a category of questions has low accuracy on a small model, change routing rules so those questions use a stronger model by default.
Knowledge-base optimization: in RAG scenarios, metric definitions with repeated hallucination-check hits start a governance process for the metric semantic layer.
6.3 Verifying Tuning Results
HENGSHI verifies each tuning change through A/B testing:
- Run the new and old Prompt templates at the same time, with a 50% / 50% traffic split.
- Run for 1 week and compare accuracy, hallucination rate, and Token consumption between the two groups.
- Roll out a change to all traffic after it surpasses the old version under the significance test. Roll it back if it falls short.
This disciplined A/B process avoids tuning changes that reduce quality. AI system tuning is measurable and verifiable engineering practice.
7. Conclusion
AI Agent observability extends traditional monitoring into a new engineering discipline. Its purpose is to make unpredictable Agent behavior observable, measurable, and optimizable.
HENGSHI BI’s Agent observability framework provides three core capabilities:
- End-to-end Trace capture: reconstructs the Agent’s reasoning path from Thought through Action to Observation.
- Automated confidence assessment: combines multiple signals to judge answer reliability, then warns the user or transfers the case to a human reviewer when confidence is low.
- Feedback-driven tuning: links user feedback, error attribution, Prompt optimization, and A/B validation into a continuous improvement loop.
A BI system that can trace, assess, and improve each AI answer gives teams production-grade AI infrastructure.