A New Multimodal BI Paradigm: HENGSHI Chart Understanding and Natural-Language Visualization

Introduction

For the past decade, BI has followed a text-driven model: users describe a requirement in words, and the system presents the result in charts. Text enters the system, charts leave it, and structured data processing sits in between.

People also understand data through visuals. A sales-trend chart can show a business user an unusual spike in Q2 at a glance. A CFO can spot a deteriorating gross margin in a financial statement. Until now, only people could read charts in this way. BI systems could not.

Mature Multimodal Large Language Models, or Multimodal LLMs, give BI systems visual understanding for the first time. They can read charts, understand screenshots, and extract data from images. Text-to-Visualization is advancing at the same time.

The latest version of HENGSHI BI introduces multimodal BI capabilities. It links chart understanding and natural-language visualization to create a bidirectional multimodal analytics loop. This article examines the technical system behind that loop.

---

1. The Two Directions of Multimodal BI

1.1 Direction One: Chart Understanding, or Chart-to-Insight

Chart understanding lets AI read a chart and extract insights from it.

Typical scenarios include:

• A user sends a screenshot from a competitor’s industry report and asks, “How large is the gap between us and our competitors in this chart?”
• A user sends an Agent a screenshot of last week’s monthly operations dashboard and asks, “Which metric in this chart deserves the most attention?”
• A user receives an email containing a chart, forwards the image to an Agent, and asks, “Is this data trend normal?”

In these scenarios, the input is an image rather than text. Conventional ChatBI handles text input only and cannot interpret images. Multimodal BI lets an Agent understand image content directly.

1.2 Direction Two: Natural-Language Visualization, or Text-to-Visualization

Natural-language visualization lets users describe an intent in natural language and have the system generate a chart.

The concept predates multimodal BI. Traditional BI provides drag-and-drop chart building and natural-language chart generation. Multimodal LLMs make the process more capable:

• A user says, “Show me the sales trends for each product line in East China over the past six months,” and the system generates a line chart.
• A user says, “Turn this chart into a heatmap and show it by week,” and the system resolves “this chart” to the chart from the preceding turn before changing its format.
• A user says, “Add a year-over-year comparison module to this dashboard,” and the system uses the conversation context to extend the existing dashboard.

The first direction expands multimodal input. Users can send images as well as type. The second direction expands multimodal output. Users can refine charts through an ongoing conversation instead of treating chart generation as a one-time action.

1.3 A Bidirectional Loop

Together, the two directions form a bidirectional multimodal BI loop:

Forward flow, Text → Chart: natural language → semantic parsing → data query → chart generation → multimodal rendering

Reverse flow, Chart → Insight: chart image → visual understanding → structured-data extraction → insight generation → natural-language answer

The loop makes both the entry and exit points of data analysis multimodal. Users can ask in many forms and receive results in many forms.

---

2. Technical Implementation of Chart Understanding

2.1 Image Input Processing

When a user uploads a chart image, the HENGSHI Multimodal BI engine follows these steps:

Step 1: Image preprocessing. The engine improves image quality through resolution normalization, noise reduction, and table-line detection for table screenshots. It splits long images with axes into slices to avoid overloading a single image with information.

Step 2: Chart-type recognition. A vision model identifies line charts, bar charts, pie charts, scatter plots, heatmaps, radar charts, and mixed documents containing charts, such as PowerPoint files and PDF reports. Each type uses its own parsing strategy.

Step 3: Element extraction. The engine extracts structured chart elements:

• Axes: X-axis labels, Y-axis scales, and units
• Data series: the color, name, and exact data-point value for each line
• Legends: mappings that define the meaning of each series
• Titles and annotations: chart titles, data-source notes, and anomaly annotations

Step 4: Data reconstruction. The engine reconstructs visual elements as a structured data table. For example, it restores a line chart as a two-dimensional table of date × metric value. AI needs this structured data to perform calculations.

Step 5: Insight generation. AI performs trend analysis, anomaly detection, and comparative analysis on the reconstructed table, then produces insights in natural language.

2.2 Accuracy Controls for Visual Understanding and Data Extraction

Data-extraction accuracy poses the main challenge in chart understanding. Any number that AI reads incorrectly from an image corrupts the analysis that follows.

HENGSHI applies the following accuracy controls:

OCR plus visual cross-validation. The engine uses OCR to extract numeric labels from a chart, then asks a vision model to validate them. If OCR reads “12,563” and the vision model estimates a Y value of about 12,500 from the point’s position, the engine confirms the value. If the results conflict, the engine marks it as a low-confidence value and alerts the user.

Axis-mapping validation. For unlabelled data points, where a chart shows a point without a numeric label, the engine infers the value from the axis scale. It checks whether the inferred value falls in a sensible range. A Y value above the maximum axis tick signals an error in the mapping logic and triggers another attempt.

Special handling for table screenshots. For a table screenshot rather than a chart, HENGSHI uses a dedicated table-structure recognition model, Table Transformer, to identify rows and columns and extract cell content. This produces higher accuracy than general OCR.

Confidence labels. The engine assigns a confidence score to each extracted value. Analysis uses high-confidence values directly and identifies low-confidence values as estimates, so users do not mistake them for exact data.

2.3 Context Management for Multimodal Conversations

Chart understanding supports more than one-off image analysis. It also supports multi-turn conversations. A typical interaction looks like this:

• Turn 1: The user uploads an image and asks, “What does the trend in this chart look like?” The Agent interprets the image and summarizes the trend.
• Turn 2: The user asks, “What caused that unusual Q2 spike?” The Agent needs to retain the image content and resolve the Q2 spike to the right data point.
• Turn 3: The user asks, “How does it compare with the same period last year?” The Agent links the image context from turn 1 with the topic from turn 2.

The context-management design includes:

• The image-understanding result, including the reconstructed data table and extracted elements, remains in the session as persistent context.
• Coreference resolution maps later references such as “this chart” and “that anomalous point” to stored context.
• The Agent can run a second, more detailed analysis of the original image if the discussion requires exact values instead of visual estimates.

---

3. Technical Implementation of Natural-Language Visualization

3.1 The Path from Text to Chart

When a user describes a visualization need in natural language, the HENGSHI Multimodal BI engine generates a chart through this path:

Step 1: Intent parsing. An LLM parses the user’s natural-language request and extracts visualization elements:

• Chart-type intent, such as trend → line chart, comparison → bar chart, and share → pie chart
• Metric intent, such as “sales” and “profit margin”
• Dimension intent, such as “by region” and “by product line”
• Time intent, such as “the past six months” and “Q1”
• Format intent, such as “heatmap” and “stacked bars”

Step 2: Semantic mapping. The engine maps the intent to specific entities in the BI semantic layer. It reuses vector-retrieval and semantic-layer capabilities to map “sales” to a metric definition and “East China” to a standard region code.

Step 3: Query generation. The engine generates a data query from the mapping result through the NL2Metrics approach and retrieves the data needed for the visualization.

Step 4: Chart-configuration generation. A multimodal LLM selects the chart type, color scheme, axis settings, and annotation positions from the data characteristics and analytic intent. It chooses from data-distribution features: a clear time trend leads to a line chart, fewer than five categories lead to a pie chart, and a complex combination of dimensions leads to a heatmap.

Step 5: Multimodal rendering. The engine renders the chart configuration as an interactive visualization that supports drill-down, linkage, and filtering.

3.2 Iterative Visualization Optimization

Natural-language visualization in multimodal BI supports iterative optimization through conversation:

Format adjustments. A user can say, “Switch to a bar chart,” “Use more vivid colors,” or “Start the Y-axis at zero.” The Agent applies each instruction as an incremental change to the existing chart configuration instead of building the chart again.

Content additions. A user can say, “Add a line for the same period last year.” The Agent adds last year’s data to the current chart query and extends the data series.

Perspective changes. A user can say, “Show it by week,” or “Change it to month-over-month.” The Agent changes the time granularity and calculation definition, then rebuilds the chart.

Intelligent suggestions. The Agent can suggest visualization improvements: “The data has 12 categories, which makes a pie chart difficult to distinguish. Use a bar chart instead.” Or: “The data shows clear seasonality. Add a year-over-year comparison line.”

3.3 Accuracy Controls for Chart Generation

Text-to-chart generation also presents accuracy risks. An unsuitable chart type can mislead users, and poor colors can obscure the data that matters.

HENGSHI uses the following controls:

Chart-type constraints. Each metric-and-dimension combination has a recommended allowlist of chart types. A single-metric time series uses a line chart, and a multi-metric time series uses a multi-axis line chart. The system does not produce a clearly unsuitable chart type.

Misleading-chart detection. After the system generates a chart, it checks for misleading designs, such as a truncated Y-axis that distorts a comparison, colors that exclude red-green color-blind users, or high percentages based on near-zero denominators. The system corrects an issue or alerts the user.

Data-to-chart consistency checks. The system verifies the rendered chart against the source data and checks whether each value shown on the chart equals the query result. This prevents an error where the data is correct but the chart is wrong.

---

4. Integrated Multimodal BI Scenarios

4.1 Scenario One: Rapid Interpretation of a Competitor Report

A market analyst receives an industry-report PDF from a competitor and needs to extract key data before comparing it with the company’s own data.

Multimodal BI process:

1. The analyst uploads the PDF report, and the system uses a document-understanding model to extract every chart and table.
2. The analyst asks, “What market-size growth rate does the report cite?” The Agent finds the relevant chart and extracts “23%.”
3. The analyst says, “Compare that with our actual growth rate.” The Agent queries company data and calculates an actual growth rate of “18%.”
4. The Agent generates a comparison visualization, industry at 23% versus the company at 18%, and identifies the source of the gap: the competitor entered a new market segment.

The analyst does not need to copy data from the PDF by hand. Multimodal BI extracts and structures it.

4.2 Scenario Two: Analyze a Screenshot

An operations director sees a real-time dashboard screenshot from a colleague on a mobile phone. One metric shows an abnormal color, red, but the director does not know its value or trend.

Multimodal BI process:

1. The director sends the screenshot to an Agent and asks, “What is happening with this red metric?”
2. The Agent identifies the chart in the screenshot and extracts the metric name, “conversion rate,” and its current value, “2.1%.”
3. The Agent connects to the complete data in the system and presents the conversion rate’s historical trend and period-over-period change.
4. The Agent attributes the anomaly: “The conversion rate fell from 3.4% last week to 2.1%, mainly because the new landing page has a higher bounce rate.”

A screenshot becomes an entry point for analysis, without returning to a PC.

4.3 Scenario Three: Conversational Dashboard Building

A BI analyst needs to build a monitoring dashboard for a new business line but does not know which charts to include.

Multimodal BI process:

1. The analyst says, “Build a monitoring dashboard for the community group-buying business line. Focus on GMV, group-leader activity, and fulfillment timeliness.”
2. The Agent generates an initial dashboard with three KPI cards, a GMV trend line chart, a group-leader activity distribution bar chart, and a fulfillment-timeliness heatmap.
3. The analyst says, “Break down fulfillment timeliness by city.” The Agent changes the heatmap to use city as a dimension.
4. The analyst says, “Add an anomaly-alert module.” The Agent adds an alert-status panel.
5. The analyst says, “Use the company’s brand blue across the dashboard.” The Agent applies the brand theme.

Dashboard building changes from dragging 20 components to speaking five sentences.

---

5. Engineering Challenges in Multimodal BI

5.1 Visual-Understanding Cost

Multimodal LLM inference costs more than text-only model inference. An image consumes far more tokens than text. A chart screenshot can consume thousands of visual tokens, equivalent to several hundred words of text.

Optimization methods:

• Image compression: The engine optimizes image resolution before upload. Chart understanding does not require 4K resolution, which reduces visual-token use.
• Preprocessing extraction: Before the engine passes material to a vision model, it uses lightweight OCR and chart-structure recognition to extract initial information. It sends only concise text-based information and cropped key areas to the LLM, cutting token use.
• On-demand activation: A visual model is not required for each turn. If the system has already understood an image in a preceding turn, later turns reuse the extracted structured data instead of reading the image again.

5.2 Accuracy Limits of Visual Understanding

Multimodal understanding does not reach 100% accuracy. The following cases can cause errors:

• Low-resolution screenshots blur numeric values.
• Dense charts with more than 30 lines can confuse data series.
• Non-standard charts, such as stylized infographics and 3D bar charts, can defeat structure recognition.

Response strategies:

• Confidence labels: The engine labels each visually extracted value with a confidence score and asks the user to confirm low-confidence values.
• Fallback: If image-understanding confidence falls too low, the system warns, “Image understanding may be inaccurate. Please provide the source data or describe the point you want to examine.”
• User correction: A user can say, “That is wrong. The value should be 12,500, not 12,000.” The system updates its context with the corrected value.

5.3 Privacy and Compliance

Multimodal BI processes user-uploaded images. Those images can contain sensitive information, including undisclosed financial data, competitor secrets, and internal screenshots.

Compliance design:

• The system processes images in an isolated environment and excludes them from general training data.
• Images can expire and be deleted from session context, for example after 24 hours.
• Enterprises can configure a policy that prohibits image uploads and turns off multimodal input.
• The system records every image-processing action in audit logs for security-team review.

---

6. Summary

Multimodal BI marks the next paradigm shift in BI. It removes the barrier between text and charts, expanding the analysis entry point beyond the search box and the presentation beyond static charts.

HENGSHI BI delivers two core multimodal capabilities:

• Chart understanding: AI reads charts in images, extracts structured data, and generates insights. Input expands from typing to sending images.
• Natural-language visualization: Users generate and refine charts through conversation. Output evolves from a one-time chart to an ongoing dialog.

A BI system that can read charts and create charts from plain language lowers the barrier to analysis to an unprecedented level. Users can send a screenshot without knowing the field name.

---

This is article 49 in the HENGSHI BI Technical Series. For the NL2Metrics technical approach behind HENGSHI ChatBI and the paradigm shift of Agentic BI, see our related topic articles.