eVisualization of Big Data (2013-2015)
Netherlands eScience Center (NLeSC) https://www.esciencecenter.nl.
- Prof.dr. Edwin Valentijn (https://www.rug.nl/staff/e.a.valentijn), Kapteyn Astronomical Institute, PI
- Prof. Dr. Jos B.T.M. Roerdink, co-PI
- Prof dr.Sterks, C.G.M., (RUG Center for Information Technology), co-applicant
- Dr. Hugo Buddelmeijer (https://www.rug.nl/staff/h.buddelmeijer), postdoc
- Dr. David Williams (http://www.cs.rug.nl/svcg/People/DavidWilliams), postdoc
- Parisa Noorishad, eScience engineer (NLeSC)
Project goal and results
Visualization highlights the difficulties of dealing with big data: our visualization devices stay in the mega/gigabyte regime, while our data grows to peta/exabytes. The problem of reducing the data about 10 orders of magnitude is especially difficult for interactive exploratory visualization. On the one hand it is a priori unknown which data will be relevant for the exploration, but on the other hand it needs to be present quickly when it is needed. Big data visualization is even more difficult in an environment where the data is rapidly changing. Raw data is continuously gathered, calibration is updated, algorithms enhanced and insights improved. The data changes faster than it can be processed. ‘The’ dataset therefore does not exist.
We addressed the big data visualization problems through full end-to-end data pulling technologies, ultimately linking the visualization all the way to the raw data. The display end of the data flow is the best place to determine which data is necessary for the visualization at hand. This specification of data can either be done manually by zooming in on a region of interest, or automatically, by estimating which data holds the most information, for example to train a classifier. We developed new technology to extend this backward chaining into the visualization domain. The techniques were demonstrated by connecting the Orange visualization software to the Astro-WISE information system and we designed best-practices for other implementers. The software was applied to medical (GLIMPS) and astronomical (KIDS) use-cases and preliminary investigation showed their usefulness in IJkdijk and Lifelines. The results have been published in peer reviewed papers and have been presented in international conferences through talks and posters. The code of our implementations is through public or private repositories.
- Hugo Buddelmeijer, Parisa Noorishad, David Williams, Milena Ivanova, Jos B. T. M. Roerdink, and Edwin A. Valentijn. Analyzing Living Surveys: Visualization beyond the Data Release. In ADASS XXIV (Astronomical Data Analysis Software & Systems), 5-9 October, Calgary. 2014. Abstract.
- David Williams, Deborah Mudali, Hugo Buddelmeijer, Parisa Noorishad, Sanne Meles , Remco Renken, Nico Leenders, Edwin Valentijn, and Jos Roerdink. Visualization of Decision Tree State for the Classification of Parkinson’s Disease. Journal of Biomedical Engineering and Medical Imaging, 3(3):25, 2016.
- E. A., Valentijn, K., Begeman, A., Belikov, D. R., Boxhoorn, J. Brinchmann, H., Holties, K. H., Kuijken, G., Verdoes Kleijn, W-J. Vriend, J. B. T. M. Roerdink, M. A. Swertz, and G. J. W. van Dijk. Target and (Astro-)WISE technologies - Data federations and its applications. In Proceedings IAU Symposium No 135 Astroinformatics. Dec1, Dec1 2016. arXiv http://arxiv.org/abs/1612.05996.
- Roerdink, J.B.T.M.: "Astronomy as a test bed for visual analytics“, Visualization Experts Day, CWI, 2013/04/03.
- Valentijn, E.A.: "The Astro-WISE Information System“, SciOps Conference, Madrid, 2013/09/12
- Valentijn, E.A.: "Probing Big Data for Answers using Data about Data“, CHEP 2013, Amsterdam, 2013/10/16
- Valentijn, E.A.: "Data Federations“, invited keynote, ICT open, Eindhoven, 2013/11/28
- Williams, D.P.: "A General Toolkit for "GPUtilisation" in SME Applications“, ICT Open, 2013/11/27
- Roerdink, J.B.T.M.: "Brain Patterns: from EEG Coherence Networks to Prediction of Neurodegenerative Diseases“, University of Bergen, Norway, 2013/12/13.
- Roerdink, J.B.T.M.: "Visualisation and Visual Analytics“, 7th International PHISCC Workshop, ASTRON, Dwingeloo, 2014/03/17
- Valentijn, E.A.: "Connecting Big Data to the Users“, invited talk, Data Driven Visualization Symposium, 2014/06/02
- Buddelmeijer, H.: "Analyzing Living Surveys - Visualization beyond the Data Release“, ADASS, Calgary, 2014/10/07
- Buddelmeijer, H.: "Query-Driven Visualization over SAMP“, IVOA InterOP, Banff, 2014/10/12
- Valentijn, E.A.: "Target Data Federations“, eScience Symposium, Almere, 2014/11/02
- Noorishad, P.: "Efficient visualization solutions to turn Big Data into insight“, Target Conference, Groningen, 2014/11/04
- Buddelmeijer, H.: "Collaborative Exploration of Big Data in Information Theaters“, Target Conference, Groningen, 2014/11/04
- Valentijn, E.A.: "Target Data Federations“, Target Conference, Groningen, 2014/11/05
- Roerdink, J.B.T.M.: "Brain Patterns: Prediction of Neurodegenerative Diseases“, TU Delft, 2014/12/12
- Roerdink, J.B.T.M.: "Understanding Big and Complex Data: How Visualisation Can Help“, Spotlight Colloquium FMNS, 2015/05/11
- Valentijn, E.A.: "Data Federation“, invited talk, DSSC Symposium, Groningen, 2015/07/07
- Buddelmeijer, H.: "Big Data Visualisation Needs Living Data“, Holland Webweek, Groningen, 2015/09/18
- Roerdink, J.B.T.M.: "Information Visualisation in Neuroscience“, ‘The Information Universe’ Conference, Groningen, 2015/10/08
- Valentijn, E.A.: "Datafederations: AstroWise and its applications“, Sci Ops, Garching, 2015/11/27
- "Query-Driven Visualization - Bridging the gaps between Processing, Archiving and Analyzing“, SciOps, Madrid, 2013
- "The Pulling Paradigm: The Future of Visualization, Preservation and Collaboration“, ESA PV, Frascati, 2013
- "Visualization of Big Data“, eScience Symposium, 2014
- Decision Tree Visualization: https://iwi232.iwinet.rug.nl/svn/svcg/projects/david/GLIMPS (contact Jos Roerdink)
- Query-Driven Visualization in Orange: https://github.com/hugobuddel/orange3 (BSD licence, contact Hugo Buddelmeijer)
- Astro-WISE: http://www.astro-wise.org (contact Edwin Valentijn)