(Translated by https://www.hiragana.jp/)
Reverse-Engineering Visualizations: Recovering Visual Encodings from Chart Images: Computer Graphics Forum: Vol 36, No 3

article

Reverse-Engineering Visualizations: Recovering Visual Encodings from Chart Images

Authors:

Jeffrey HeerAuthors Info & Claims

Computer Graphics Forum, Volume 36, Issue 3

Pages 353 - 363

https://doi.org/10.1111/cgf.13193

Published: 01 June 2017 Publication History

Abstract

We investigate how to automatically recover visual encodings from a chart image, primarily using inferred text elements. We contribute an end-to-end pipeline which takes a bitmap image as input and returns a visual encoding specification as output. We present a text analysis pipeline which detects text elements in a chart, classifies their role e.g., chart title, x-axis label, y-axis title, etc., and recovers the text content using optical character recognition. We also train a Convolutional Neural Network for mark type classification. Using the identified text elements and graphical mark type, we can then infer the encoding specification of an input chart image. We evaluate our techniques on three chart corpora: a set of automatically labeled charts generated using Vega, charts from the Quartz news website, and charts extracted from academic papers. We demonstrate accurate automatic inference of text elements, mark types, and chart specifications across a variety of input chart types.

References

[1]

<label>{BOH11}¿¿</label> Bostock M., Ogievetsky V., Heer J.: D3: Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics Volume 17, Issue 12 2011, pp.2301-2309. 2

Digital Library

[2]

<label>{BS15}¿¿</label> Böschen F., Scherp A.: Multi-oriented text extraction from information graphics. In Proceedings of the 2015 ACM Symposium on Document Engineering 2015, pp. pp.35-38. 2

Digital Library

[3]

<label>{CCA15}¿¿</label> Chen Z., Cafarella M., Adar E.: DiagramFlyer: A search engine for data-driven diagrams. In Proceedings of the 24th International Conference on World Wide Web 2015, pp. pp.183-186. 2, 10

Digital Library

[4]

<label>{CD16}¿¿</label> Clark C., Divvala S.: PDFFigures 2.0: Mining figures from research papers. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries 2016, pp. pp.143-152. 3

Digital Library

[5]

<label>{CV95}¿¿</label> Cortes C., Vapnik V.: Support-vector networks. Machine Learning Volume 20, Issue 3 1995, pp.273-297. 6

Digital Library

[6]

<label>{CWG16}¿¿</label> Choudhury S.R., Wang S., Giles C.L.: Scalable algorithms for scholarly figure mining and semantics. In Proceedings of the International Workshop on Semantic Big Data 2016, pp. pp.1:1-1:6. 2, 9

Digital Library

[7]

<label>{HA14}¿¿</label> Harper J., Agrawala M.: Deconstructing and restyling D3 visualizations. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology 2014, pp. pp.253-262. 2

Digital Library

[8]

<label>{HLYW13}¿¿</label> Huang W., Lin Z., Yang J., Wang J.: Text localization in natural images using stroke feature transform and text covariance descriptors. In Proceedings of the 2013 IEEE International Conference on Computer Vision 2013, pp. pp.1241-1248. 2

Digital Library

[9]

<label>{HT07}¿¿</label> Huang W., Tan C.L.: A system for understanding imaged infographics and its applications. In Proceedings of the 2007 ACM Symposium on Document Engineering 2007, pp. pp.9-18. 2

Digital Library

[10]

<label>{JKS*17}¿¿</label> Jung D., Kim W., Song H., Hwang J.-I., Lee B., Kim B., Seo J.: ChartSense: Interactive data extraction from chart images. In ACM Human Factors in Computing Systems CHI 2017. 2, 7, 10

Digital Library

[11]

<label>{JRW*07}¿¿</label> Jayant C., Renzelmann M., Wen D., Krisnandi S., Ladner R., Comden D.: Automated tactile graphics translation: In the field. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility 2007, pp. pp.75-82. 2

Digital Library

[12]

<label>{JSD*14}¿¿</label> Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., Darrell T.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 2014. 7

[13]

<label>{KA12}¿¿</label> Kong N., Agrawala M.: Graphical overlays: Using layered elements to aid chart reading. IEEE Transactions on Visualization and Computer Graphics Volume 18, Issue 12 2012, pp.2631-2638. 2

Digital Library

[14]

<label>{Kru56}¿¿</label> Kruskal J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society Volume 7, Issue 1 1956, pp.48-50. 5

[15]

<label>{KSH12}¿¿</label> Krizhevsky A., Sutskever I., Hinton G.E.: ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems Volume 25. 2012, pp. pp.1097-1105. 7

Digital Library

[16]

<label>{LH15}¿¿</label> Lee P., Howe B.: Dismantling composite visualizations in the scientific literature. In 4th International Conference on Pattern Recognition Applications and Methods 2015. 3

Digital Library

[17]

<label>{Luc05}¿¿</label> Lucas S.M.: ICDAR 2005 text locating competition results. In Eighth International Conference on Document Analysis and Recognition 2005, vol. Volume 1, pp. pp.80-84. 5

Digital Library

[18]

<label>{mic}¿¿</label> Microsoft Project Oxford. "https://www.projectoxford.ai/vision". 2, 5, 6

[19]

<label>{MMR*01}¿¿</label> Müller K.-R., Mika S., Rätsch G., Tsuda S., Schölkopf B.: An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks Volume 12, Issue 2 2001, pp.181-202. 6

Digital Library

[20]

<label>{MNV16}¿¿</label> Méndez G.G., Nacenta M.A., Vandenheste S.: iVoLVER: Interactive visual language for visualization extraction and reconstruction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems 2016, pp. pp.4073-4085. 2

Digital Library

[21]

<label>{Mor17}¿¿</label> Moritz D.: Text detection in screen images with a convolutional neural network. The Journal of Open Source Software 2017. "https://github.com/domoritz/label_generator". 4

[22]

<label>{NM16}¿¿</label> Neumann L., Matas J.: Real-time lexicon-free scene text localization and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence Volume 38, Issue 9 2016, pp.1872-1885. 2

[23]

<label>{Ots79}¿¿</label> Otsu N.: A Threshold Selection Method from Gray-level Histograms. IEEE Transactions on Systems, Man and Cybernetics Volume 9, Issue 1 1979, pp.62-66. 4

[24]

<label>{RCWG16}¿¿</label> Ray Choudhury S., Wang S., Giles C.L.: Curve separation for line graphs in scholarly documents. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries 2016, pp. pp.277-278. 2

Digital Library

[25]

<label>{Red16}¿¿</label> Redmon J.: Darknet: Open source neural networks in c. "http://pjreddie.com/darknet/", pp.2013-2016. 4

[26]

<label>{SDF16}¿¿</label> Siegel N., Divvala S., Farhadi A.: FigureSeer: Parsing result-figures in research papers. In Proceedings of the European Conference on Computer Vision 2016, pp. pp.664-680. 2, 7, 8, 9, 10

[27]

<label>{SKC*11}¿¿</label> Savva M., Kong N., Chhajta A., Fei-Fei L., Agrawala M., Heer J.: ReVision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology 2011, pp. pp.393-402. 2, 7, 10

Digital Library

[28]

<label>{SLJ*15}¿¿</label> Szegedy C., Liu W., Jia Y., Sermanet P., Reed S., Anguelov D., Erhan D., Vanhoucke V., Rabinovich A.: Going deeper with convolutions. In Computer Vision and Pattern Recognition 2015, pp. pp.1-9. 7

[29]

<label>{Smi07}¿¿</label> Smith R.: An overview of the tesseract ocr engine. In Proceedings of the Ninth International Conference on Document Analysis and Recognition 2007, vol. Volume 2, pp. pp.629-633. 2, 5

Digital Library

[30]

<label>{SMWH17}¿¿</label> Satyanarayan A., Moritz D., Wongsuphasawat K., Heer J.: Vega-Lite: A grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics Volume 23, Issue 1 2017, pp.341-350. 1

Digital Library

[31]

<label>{SRHH16}¿¿</label> Satyanarayan A., Russell R., Hoffswell J., Heer J.: Reactive Vega: A streaming dataflow architecture for declarative interactive visualization. IEEE Transactions on Visualization and Computer Graphics Volume 22, Issue 1 2016, pp.659-668. 2, 3

[32]

<label>{STH02}¿¿</label> Stolte C., Tang D., Hanrahan P.: Polaris: A system for query, analysis, and visualization of multidimensional relational databases. IEEE Transactions on Visualization and Computer Graphics Volume 8, Issue 1 2002, pp.52-65. 1

Digital Library

[33]

<label>{WMA*16}¿¿</label> Wongsuphasawat K., Moritz D., Anand A., Mackinlay J., Howe B., Heer J.: Voyager: Exploratory analysis via faceted browsing of visualization recommendations. IEEE Transactions on Visualization and Computer Graphics Volume 22, Issue 1 2016, pp.649-658. 3

Cited By

Liu CZhang YWu CLi CYuan X(2024)A Spatial Constraint Model for Manipulating Static VisualizationsACM Transactions on Interactive Intelligent Systems10.1145/365764214:2(1-29)Online publication date: 11-Apr-2024
https://dl.acm.org/doi/10.1145/3657642
Huang JChen HYu FLu W(2024)From Detection to Application: Recent Advances in Understanding Scientific Tables and FiguresACM Computing Surveys10.1145/365728556:10(1-39)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3657285
Ying LWu ALi HDeng ZLan JWu JWang YQu HDeng DWu Y(2024)VAID: Indexing View Designs in Visual Analytics SystemProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642237(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642237
Show More Cited By

Recommendations

Reverse-engineering bar charts using neural networks
Abstract
Reverse-engineering bar charts extract textual and numeric information from the visual representations of bar charts to support application scenarios that require the underlying information. In this paper, we propose a neural network-based method ...
Rotate or Wrap? Interactive Visualisations of Cyclical Data on Cylindrical or Toroidal Topologies
In this paper, we report on a study of visual representations for cyclical data and the effect of interactively <italic>wrapping</italic> a bar chart ‘around its boundaries’. Compared to linear bar chart, polar (or radial) visualisations ...
Sample-oriented task-driven visualizations: allowing users to make better, more confident decisions
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We often use datasets that reflect samples, but many visualization tools treat data as full populations. Uncertain visualizations are good at representing data distributions emerging from samples, but are more limited in allowing users to carry out ...

Comments

Information & Contributors

Information

Published In

cover image Computer Graphics Forum

Computer Graphics Forum Volume 36, Issue 3

June 2017

639 pages

ISSN:0167-7055

EISSN:1467-8659

Issue’s Table of Contents

Publisher

The Eurographs Association & John Wiley & Sons, Ltd.

Chichester, United Kingdom

Publication History

Published: 01 June 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu CZhang YWu CLi CYuan X(2024)A Spatial Constraint Model for Manipulating Static VisualizationsACM Transactions on Interactive Intelligent Systems10.1145/365764214:2(1-29)Online publication date: 11-Apr-2024
https://dl.acm.org/doi/10.1145/3657642
Huang JChen HYu FLu W(2024)From Detection to Application: Recent Advances in Understanding Scientific Tables and FiguresACM Computing Surveys10.1145/365728556:10(1-39)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3657285
Ying LWu ALi HDeng ZLan JWu JWang YQu HDeng DWu Y(2024)VAID: Indexing View Designs in Visual Analytics SystemProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642237(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642237
Chen CLee BWang YChang YLiu Z(2024)Mystique: Deconstructing SVG Charts for Layout ReuseIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332735430:1(447-457)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327354
Snyder LHeer J(2024)DIVI: Dynamically Interactive VisualizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332717230:1(403-413)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327172
Lo LCao YYang LQu H(2024)Why Change My Design: Explaining Poorly Constructed Visualization Designs with Explorable ExplanationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332715530:1(955-964)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327155
Kim DChoi SKim JSetlur VAgrawala M(2024)EC: A Tool for Guiding Chart and Caption EmphasisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332715030:1(120-130)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327150
Liu CGuo YYuan X(2024)AutoTitle: An Interactive Title Generator for VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329024130:8(5276-5288)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3290241
Thiyam JSingh SBora P(2024)Chart classification: a survey and benchmarking of different state-of-the-art methodsInternational Journal on Document Analysis and Recognition10.1007/s10032-023-00443-w27:1(19-44)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10032-023-00443-w
Kim HLell NScherp A(2024)Text Role Classification in Scientific Charts Using Multimodal TransformersNatural Language Processing and Information Systems10.1007/978-3-031-70239-6_4(47-61)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-70239-6_4
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents