Do Text-to-Vis Benchmarks Test Real Use of Visualisations?

Overview

• This paper investigates whether current text-to-visualization benchmarks accurately reflect how visualizations are used in the real world.
• The authors collect a new dataset of visualization usage from real-world sources and compare it to existing benchmarks.
• They find that benchmarks focus more on generating visualizations from textual descriptions, rather than the actual usage of visualizations.

Plain English Explanation

The paper looks at whether the benchmarks used to test text-to-visualization models really capture how people use visualizations in the real world. The authors gathered a new dataset of how people actually interact with and use visualizations from various online sources. They then compared this to the types of tasks and scenarios covered by existing benchmarks.

The key finding is that the current benchmarks tend to focus more on generating visualizations from textual descriptions, rather than the actual usage of visualizations. In the real world, people often use visualizations to understand data, draw insights, and support decision-making. The benchmarks don't necessarily reflect these real-world uses of visualizations.

Technical Explanation

The paper begins by surveying related work on text-to-visualization benchmarks, such as ViSEval and ChartBench. It notes that these benchmarks primarily evaluate the ability to generate visualizations from textual descriptions, rather than assessing the actual usage of visualizations.

To investigate this gap, the authors collected a new dataset of visualization usage from real-world sources like online articles, tutorials, and social media. They analyzed the types of tasks, interactions, and visualizations present in this dataset.

The results show that the real-world usage of visualizations is more diverse and oriented towards tasks like data exploration, insight generation, and decision support - rather than just generating visualizations from text. The authors argue that future benchmarks should better reflect these real-world use cases for visualizations.

Critical Analysis

The paper raises important points about the limitations of current text-to-visualization benchmarks. By highlighting the gap between benchmark tasks and real-world visualization usage, it suggests that these benchmarks may not accurately assess the capabilities of models for practical applications.

However, the paper does not delve deeply into the specific challenges or tradeoffs involved in designing more representative benchmarks. It also does not discuss potential biases or limitations in the authors' own data collection process.

Further research could explore ways to bridge the gap between benchmarks and real-world usage, such as incorporating more diverse task types, interaction patterns, and visualization types into benchmark design. Longitudinal studies on how people use visualizations in different contexts could also provide valuable insights.

Conclusion

This paper demonstrates that current text-to-visualization benchmarks do not fully capture the real-world usage of visualizations. By collecting a new dataset of visualization usage and comparing it to existing benchmarks, the authors highlight the need for more representative and task-oriented evaluation of visualization-related technologies.

The findings suggest that future research and development in this area should focus on supporting the actual ways people interact with and use visualizations, rather than just generating visualizations from text. This shift could lead to more impactful and practical advancements in the field of data visualization.