New Media for Scientific Data Visualization by Andrea Effe Rao

Andrea Fasolo Rao

New MediA for scientific data visualization

New Media for scientific data visualization Andrea Fasolo Rao 279245 Supervisor: Marco Ferrari A.A 2014/2015

Sessione di Laurea Marzo 2016 UniversitĂ Iuav di Venezia Dipartimento di Progettazione e pianificazione in ambienti complessi Corso di Laurea Magistrale in Design del Prodotto e della Comunicazione Visiva Curriculum Comunicazioni visive e multimediali

Abstract Italiano

Questa tesi affronta il tema dell’interpretazione e della visualizzazione dei dati scientifici, su come si è sviluppata negli ultimi decenni, il suo legame con la tecnologia informatica e la posizione nei riguardi dei principi del design dell’informazione più tradizionale, oltre all’enorme influenza che il fenomeno Open Data sta avendo sul mondo della visualizzazione in generale. Verranno analizzati inoltre vari progetti basati sulla visualizzazione dati, ma che la affrontano da punti di vista diversi: quello puramente scientifico, quello del design e quello artistico. Quest’ultimo tipo di progetti potrebbe essere fondamentale per l’evoluzione della visualizzazione scientifica e di dati in generale: mettendo al primo posto valori estetici piuttosto che informativi esplora nuove modalità di approccio ai dati precedentemente non prese in considerazione Inoltre, l’utilizzo di strumenti considerati innovativi (video, new media, interattività, suono) può dare nuova vita alla divulgazione scientifica e alla visualizzazione di dati. La complessità e la vastità dei fenomeni e dei dati coinvolti, applicando algoritmi di “traduzione” in immagini e suoni, può contribuire a costruire esperienze di valore sia divulgativo che estetico. Sfruttando questo valore estetico si può avvicinare il pubblico a tematiche apparentemente complesse ma affascinanti, che normalmente passano per forme di divulgazione più tradizionali che, negli anni a venire, rischiano di perdere “appeal” agli occhi di un pubblico sempre più esigente, in termini di entertainment e “spettacolarità”.

New Media for Scientific Data Visualization

This thesis deals with the interpretation and visualization of scientific data, on how it has developed in recent decades, its connection with computer technology and its position regarding the principles of the more traditional information design, and also the huge influence of the Open Data phenomenon is having on the visualization world in general. Various data visualization projects will be analyzed, that approaches it from different angles: the purely scientifically one, the design one and the art one. This last kind of project could be fundamental for the scientific, and data too, visualization evolution: with its primary focus on aesthetic values rather than informative it explores new ways to approach data; in addiction the use of innovative instruments (video, new media, interactivity, sound) can breathe new life in science popularization and to data visualization. The complexity and breadth of the involved phenomena and data, together with appropriate “translating” algorithms to sounds and images, can create informative, popularizing and aesthetic experiences. Leveraging this aesthetic value, it can be possible to bring the public closer to apparently complex but fascinating themes, since the more traditional popularization and communication means will lose of “appeal” to an always more demanding and difficult to astound public, that is becoming used to always more spectacular and entertaining means of communication.

Abstract English

Contents

1. Introduction

Why to investigate Scientific Visualization

2. Scientific Visualization

What is Scientific Visualization

Historic Scientific Visualization

Scientific Visualization, computers and computer graphics

3. Open Data

Beginning of the Open Data era

Early Open Data examples

The birth of public open data

The future: Linked Data

4. Art, Science and Design

Separation of Science and Aesthetics

Evaluating attractiveness

The role of art

New Media for Scientific Data Visualization

5. Case Studies

Case studies selection

Ryoji Ikeda, Supersymmetry

Nathalie Miebach, Weather scores

Alex H. Parker, Worlds

Aaron Koblin, Nik Hafermaas, Dan Goods, eCLOUD

Soo-in Yang, David Benjamin, Living Light

Lorenzo Pagliei, Les Invisibles

JeongHo Park, LHC CERN Data Audio Generator

Jan Willem Tulp, Goldilocks

Kurzgesagt, In a Nutshell

Nasa Scientific Visualization Studio

Nasa Hyperwall Visualizations

6. A Design-centered evolution

Design as the unifying force

Usage of new media

Coverage of contemporary and popular science topics

Usage of datasets and real-time data sources

Keeping the data understandable

Application of design reasoning and verifiable methodology

7. Project: Sounds from the Sun

Sound and Psychoacoustics

100

Application to data

102

The Solar wind and its consequences

103

Solar wind observation

104

Sound rendition of solar wind data

106

Technical aspects

108

Testing process

110

Resources

114

Conclusions

118

Thanks

121

Colophon

122

Introduction

Why to investigate scientific visualization

One of the most rapidly growing field of design in the last decade has surely been data visualization. This happened, and is still happening, for a variety of reasons, including the always-increasing availability of any sort of dataset, the rise of visual journalism, but also the incredible computational power that we can have inside the few kilograms of a nowadays laptop computer, when compared to a 20, or even 10, years ago workstation. Thanks to these few conditions, we have the possibility, power and will to create progressively more useful and appealing visualizations that can fit in different aspects of our lives. Parallel to design-driven and “artistic” data visualizations, in the last three decades another field of representation was born and growing exponentially, both in usage and importance: Scientific visualization, or SciViz. This particular kind of computer graphics is traditionally aimed at rendering three-dimensional phenomena of various kind, with an emphasis on the “realistic” representation of features like volumes or surfaces, depending on the aim of the piece. This kind of artifacts, however, are usually developed in highly specific scientific or academic environments, focusing on very different parameters from the artworks we are more used to see, aiming at a very small group of experts from a specific field and often created without the design knowledge involved in the generally more accessible works of data visualization. However, things are rapidly changing: in the last years the open data revolution caused a great number of research centers and agencies to make their datasets available to the public. For the first time in history each one of us have the possibility to work on the very same numbers analyzed by the top scientists and researchers of NASA or CERN, just to mention two of many. This created a lot of interest in various design and art communities, and made pos-

New Media for Scientific Data Visualization

sible some extremely interesting works of â&#x20AC;&#x153;data renditionâ&#x20AC;?, ranging from static visualization to real-time immersive interactive installations, covering the vast arc that goes from design to art. This could turn out to be a great possibility for Scientific visualization: it has the chance to learn from the great mass of projects involving open data. It could incorporate, for example, a more conscious and consistent design process, to improve usability and readability for the specialists, but also to look at the more experimental and aesthetics-oriented works of art, that could be made into powerful popularization instruments for the general public.

Introduction

Scientific Visualization

What is Scientific Visualization

1: T.M. Rhyne, North Carolina State University, “Information and Scientific Visualization: Separate but Equal or Happy Together at Last”, 2011

2: The Ohio State University, “A Critical History of Computer Graphics and Animation” https://design.osvu.edu/carlson/history/ lesson18.html

3: T. Delmarcelle, L. Hesselink, “Visualizing secondorder tensor fields with hyperstreamlines”, Computer Graphics and Applications, IEEE, 1993

If Scientific visualization is approached, the first thing that should be made clear is if and how it is different from what we usually call “data visualization”: is it a subfield, a completely different thing or something in between? Traditionally, Scientific visualization has always been considered a group of techniques to transform some kind of science-based data into visual information through a reproducible process, with the fundamental difference from data visualization that the data is intrinsically spatial and physically based. In other words, in Scientific visualization the spatial coordinates are part of the data, usually being between the independent variables; while in information and data visualization designers often deal with abstract data, so they can freely decide how to manage the two or three spatial dimensions.1 The same consideration is for time: in Scientific visualization it’s often given and intrinsic to the data points, while in data visualization it’s another parameter that the designer can choose to manipulate. This difference can be further clarified when the most common visualization techniques used in SciViz are taken into account, considering data of different nature:2 bi-dimensional dataset usually represents scalar or vector fields; in the case of scalar data (for ex., temperatures) color mapping, wireframe plots, scatter plots or contour lines can be used, while vector fields (like winds or magnetic fields) take advantage of streamlines or line integral convolution; For tri-dimensional data the subdivision between scalar and vector fields still adapts: in case of scalar values, volume rendering or isosurfaces can be used, while vector data uses streamlines, line integral convolution, particle tracing and topological methods. To visualize tensor fields instead, which are more complex, a technique denominated “hyperstreamlines” were developed.3

New Media for Scientific Data Visualization

Example of line integral convolution applied visualizing heat movements in a hot water pot

Example of mesh visualization of a 3D function

Example of hyperstreamlines applied to tensor field visualization

Scientific Visualization

Historic Scientific Visualization

4: J.C. Maxwell, “Letter to James Thomson”, The Scientific Letters and Papers of James Clerk Maxwell, volume 3, p. 230-231

For how SciViz is defined and for its mechanics, it is tightly bounded to computers: it requires to elaborate and translate to images always increasing amounts of data, or to run complex simulations that may take up to years of calculations on huge supercomputers. Therefore, it may be difficult to imagine Scientific visualization before the digital revolution that happened in the ’80. In fact, usually we consider SciViz to be born in the early ’80, and becoming prominent at the end of the same decade, after a 1987 report released by SIGGRAPH for the National Science Foundation, “Visualization in Scientific Computing”. That report was crucial for the history of Scientific visualization, bringing to the newborn field a lot of attentions, that concretized themselves in new funding, workshops, conferences and publications in the following years. Before the NSF paper there are anyway some remarkable examples of visualizations, developed beginning from the ’60, and there are even more impressing examples of “pre-digital” Scientific visualization: in fact, the century from 1850 to 1950 gave a great push to the developing of various data representation techniques. The most important example, but also quite singular, is considered to be the Maxwell’s thermodynamic surface, a clay sculpture created in 1874 by the mathematician James Clerk Maxwell. He is considered the father of electromagnetism, but he was active also in other physics and mathematics fields. His surface was, in fact, a three-dimensional abstract plot of the thermodynamic states of a fictitious substance, similar to water, putting in relation its volume, entropy and energy on the x, y and z axes, with isothermal and isopiestics lines traced on the surface. With Maxwell’s own words, it allowed “the principal features of known substances [to] be represented on a convenient scale”4 : usually, in thermodynamics, when it is required to analyze the state of any fluid

New Media for Scientific Data Visualization

and to extract a parameter like pressure or temperature, engineers and scientist have to rely on a number of different bi-dimensional diagrams, each one plotting just two parameters at a time, like pressure and volume, or temperature and entropy, and so on. Those diagrams can be considered, in fact, just various bi-dimensional projections of a Maxwellâ&#x20AC;&#x2122;s thermodynamic surface. While the practicality of this artifact is low, due to the obvious difficult to extrapolate or read single values from it, it offers an extreme degree of synthesis, allowing a novel and encompassing view on various phenomena of fluid thermodynamics, offering so a major insight value over the traditional bidimensional graphs. Going back two decades, another remarkable example must be taken into consideration, this time in the field of public health. During 1854 there was a cholera outbreak in London, in the Soho district: it was a particularly unfortunate area of the city, since

Maxwell thermodynamic surface Cavendish Laboratory, Cambridge, England.

Example of Pression / Entropy thermodynamic diagram for water

Example of Entropy / Enthalpy thermodynamic diagram

New Media for Scientific Data Visualization

the sewer system had still to reach it. This wasn’t, anyway, a big concern to the population: at the time it was a wide spread idea that infectious diseases spread through some kind of miasma, or “bad air”, therefore most people had no idea of the importance of proper sanitary services. What really changed the situation, saving many other people from the infection and bringing a new understanding about hygiene, was the work of John Snow, a physician: he started tracing on a map all the cases of cholera he was hearing about from the population, and thus noticing that most cases were localized in a precise area. Further investigations brought him to conclude that most of the infected had been drinking water from a particular street pump. His research led to the closing of that pump, that later was discovered to be polluted by a nearby cesspit, and created in the population the awareness that some diseases are carried by infected water, thus anticipating Louis Pasteur’s germ theory

Snow’s original cholera spreading map

Scientific Visualization

Florence Nightingale’s “Diagram of the causes of mortality in the army in the East” 5: Steven Johnson, The Ghost Map: The Story of London’s Most Terrifying Epidemic – and How it Changed Science, Cities and the Modern World, Riverhead Books, 2006

by 7 years.5 It’s an incredible milestone for scientific visualization: it was one of the first time, and a proof, that visualizing a phenomena on a map in a useful way can cast a new light on events that may seem unconnected between them. Another notable example that are considered precursors of SciViz is the “Diagram of the causes of mortality in the army in the East” by Florance Nightingale, from 1873. She was as a British nurse manager during the Crimean War, founder of the first secular nursing school in London and also a proficient writer, with various publications to spread medical knowledge. She also worked on and popularized the graphical presentation of statistical data, leading to a breakthrough in scientific visualization. During her career she used to produce various diagrams, with a designed that she created and called “coxcombs”, or nowadays polar area diagrams, that she often used to present statistical data and observations to people who may had difficult to understand traditional statistical reports, for example for her reports to the members of the Parliament on the state of the medical care in the Crimean War. One of these diagrams visualized the seasonal mortality by cause of her patients, noting that a big number of deaths was linked to the bad state of British army camps in India, particularly: bad drainage, contaminated water, overcrowding and poor ventilation. So, she started a campaign to popularize the idea

New Media for Scientific Data Visualization

that it was necessary to improve the sanitary conditions in the country to increase the life quality and expectance for both the Indian population and the British army. With support from the Royal Sanitary Commission, she convinced the minister responsible to require every house owner to connect its building to the main drainage system, greatly enhancing sanitary conditions. As in Snowâ&#x20AC;&#x2122;s example, also this case shows how a proper and well-design way to represent abstract data can help to establish connections that are otherwise difficult to be noted, and to spread knowledge on problems that are difficult to understand for whoever may be a stranger to a particular field of knowledge.

Scientific Visualization

Scientific Visualization, computers and computer graphics

Composition of frames from “A two gyro gravity gradient attitude control system”

6: Steven Johnson, The Ghost Map: The Story of London’s Most Terrifying Epidemic – and How it Changed Science, Cities and the Modern World, Riverhead Books, 2006

One of the most remarkable example of early computer-generated scientific visualizations, and also the first computer graphics film, is “A two gyro gravity gradient attitude control system”. Dated 1963, it comes from the Bell Laboratories, at the time owned by AT&T. It was created by Edward E. Zajac, an engineer working on various long-range, trans-oceanic and planetary communications projects. At the beginning of the sixties Zajac was working on Bell’s communication satellites program, involving a new kind of satellite equipped with two gyroscopes, that would have solved some stability and control problems that emerged with the previous satellite designs. While the mathematics required to prove its theory are thoughtfully explained in its paper “A Two-Gyro, Gravity-Gradient Satellite Attitude Control System”, (J.A. Lewis, E.E. Zajac, The Bell system technical journal, 1964), he felt that the data, the graphs and the static images in that dissertation were not really enough to explain the complex rotations that its theorized satellite would have. So, together with his colleague Frank Sinden, they started to work on an innovative visualization technique: a perspective video simulation, programmed in FORTRAN and finalized using a custom-made software called ORBIT, designed for this very purpose. The realization process is to be considered interesting, for today’s standards: the original computations were fed via punch cards to a IBM 7090 supercomputer, and then printed directly to microfilm with a General Dynamics Electronics Stromberg-Carlson 4020 microfilm recorder. While the animation looks clear but extremely elementary, each minute of the resulting 16 FPS-video rendering took up to eight minutes of computation on what was one of the most performing supercomputer at the time.6 In those early days of computing resources were incredibly strict, thus the aesthetics of most visual-

New Media for Scientific Data Visualization

izations were tied just to black and white, since color would have consumed significantly more memory. The first example of the use of color came nevertheless in 1969, by Dr. Louis Frank, who was working with the Space Plasma Physics Research group of the University of Iowa on the data collected by the LEPEDEA instrument of a satellite.7 He plotted time on the x axis, while the energy spectra in on the y axis and the different color hues represents the amount of particle hits per second, effectively summarizing three dimensions of data in two-dimensional plot. Going back again to animation, in 1977 the Computer Graphics Research Group (CGRG) of Ohio State University, with Bob Reynolds and Wayne Carl-

IBM 7090 supercomputer

7: http://www-pi.physics.uiowa.edu/~frank/

Dr. Frankâ&#x20AC;&#x2122;s time / energy graph

Scientific Visualization

The Mice Galaxies

Still frame from â&#x20AC;&#x153;Interacting Galaxiesâ&#x20AC;?

New Media for Scientific Data Visualization

son, created a milestone visualization of interacting galaxies that was also featured on the Carl Sagan Cosmos TV show.8 The importance of this work lies in the fact that it was a display of a very complex numerical simulation, that actually was repeated many times with different parameters to show how different types of interaction can create very differently shaped galaxies, explaining for example the tidal tails formations we observe in the Antennae Galaxies and in the Mice.9 The simulation was programmed in FORTRAN by Dr. Alan Toomre, and even if quiet simple for the number of particles simulated in the system, proved how this kind of simulation were already a valuable research tool in this particular field: in fact, when dealing with this kind of dynamics, simulations are the only way to visualize the evolution of galaxies, since observation can show us only “stills” of galaxies in various stage of their life, considering the time involved in these processes. For example, his simulations led Toomre to suggest that elliptical galaxies are remnants of the major mergers of spiral galaxies,10 which became a widely accepted idea from the scientific community. Visualization and animation proved its value also with another historic example, this time in the field of programming. One of the classic problems posed to computer science students is the implementation of sorting algorithms, which are various ways to put in order a big series of randomly – distributed

8: http://excelsior.biosci.ohio-state.edu/~carlson/ history/ACCAD-overview/overview-old.html

9: http://excelsior.biosci.ohio-state.edu/~carlson/ history/ACCAD-overview/overview-old.html

10: http://excelsior.biosci.ohio-state. edu/~carlson/history/ACCAD-overview/ overview-old.html

Still frames from “Sorting out Sorting”

Scientific Visualization

Still frames from “Sorting out Sorting”

11: R. Baecker, “Sorting out Sorting: A Case Study of Software Visualization for Teaching Computer Science”, Software Visualization: Programming as a Multimedia Experience, MIT Press, 1998.

12: Bruce McCormick, Thomas DeFanti, Maxine Brown, Visualization in Scientific Computing, p. vii

numbers. Mathematics and researchers created in the years a lot of different sorting algorithms, reaching an incredible level of efficiency, but also involving very complex mathematics processes that make very different to imagine how an algorithm actually works and to estimate its efficiency. With this idea in mind, Ronald Baecker from the University of Toronto created in 1980, and presented at Siggraph in 1981, “Sorting out Sorting”, the first computer-generated process-visualization animation that showed the application of different algorithms in real-time, making their inner workings evident, with clear comparisons on time and efficiency.11 The video was a great success, and it’s still used today to illustrate that kind of processes. The fact that really pushed forward the at-thetime young Scientific Visualization field was the publication of a report compiled for the National Science Foundation from a SIGGRAPH panel in 1987, called “Visualization in Scientific Computing”: first of all it acknowledged and defined “officially” Scientific Visualization, poetically describing it as “a tool for applying computer to science, [that] offers a way to see the unseen”,12 and then proceeded to explain the role of visualization in science, which processes are involved, how it should expand and even mentioned the already underrated importance of data, especially from high-volume sources. The paper identifies Visualization as a multidisciplinary field, that at that time involved computer graphics, image processing, computer vision, computer-aided design, signal processing and user interface studies, but also recognizes the need to expand that, including engineering and scientists dedicated to develop new visualization sys-

New Media for Scientific Data Visualization

Still frame from the last section of “Sorting out Sorting”

tems, artists, for their education in visual communication and inventive about visual representations, cognitive scientist and psychologists to design better interfaces and try new concepts and steer research in new directions.13 Throughout the paper it’s underlined many times how visualization in general is fundamental to human beings, and that a lot of different kinds of information are better communicate in a visual way: for example, when dealing with big amounts of data that the brain cannot process in numerical form, or for structures like DNA, molecular models or other kinds of simulation. An interesting fact is that the authors note the need for interaction in Scientific visualization: at the time most elaborations run on supercomputers via batch processing, so the user could only start the process and wait for the result, without knowing what the calculator was doing. They theorized Interactive visual computing, where scientist could “communicate with data by manipulating its visual representation during processing. The more sophisticated process of navigation allows scientists to steer, or dynamically modify, computations while they are occurring. These processes are invaluable tools for scientific discovery.”14 This change should be implemented developing and giving high-quality visualization tools directly to the researchers, via a proposed federallyfunded initiative. The first step should be to abandon the supercomputing model, in favor of “workstations, minicomputers and image computers […] that should be placed on the desks of each and every researcher” with specialized graphics processors, that could be much more efficient, with each of these machines costing between 5,000 $ and 100,000 $. Along with

13: Bruce McCormick, Thomas DeFanti, Maxine Brown, Visualization in Scientific Computing, p. 11

14: Bruce McCormick, Thomas DeFanti, Maxine Brown, Visualization in Scientific Computing, p. 5

Scientific Visualization

this new hardware, research centers should provide dedicated software to develop visualizations, easy enough to use to not require the user to know programming or have particular graphical knowledge. The SIGGRAPH report apparently got its point, causing the interest of the research world, attracting funding agencies and triggering conferences, workshops and various publications on the topic. Most importantly, various software developers proposed a variety of specific and generic visualization software environments, for example IBM Data Explorer, Wavefront Advanced Visualizer and SGI IRIS Explorer. Most of them did not use code or textual instruction, but rather developed a dataflow paradigm that did not require programming or graphics knowledge: the user had to connect modules between them, each one performing a specific operation. It was however possible to create custom modules with traditional programming languages like C or FORTRAN.

Screenshot from the Wavefront Advanced Visualizer

New Media for Scientific Data Visualization

Screenshot from apE from the Ohio Supercomputer Graphics Project at Ohio State

Screenshot from the IBM Data Explorer software

Scientific Visualization

Open data

beginning of The Open Data era

Robert King Merton

A phenomenon that is slowly influencing and changing various aspect of our lives, even if most of us are still not really aware of it, is Open Data: with its idealistic push and novel way of seeing information we can consider it to be the most recent revolution in the history of knowledge. For centuries data, particularly governmental, was considered something to keep away and even dangerous for the public security, until 2008, when the Obama administration realized that opening up their data to the public could bring to more participation and trust in the government. But if we consider the attitude towards the openness of scientific data and results, it has always been more positive than in the political world, with various researchers theorizing that every scientific result should be as more widespread and accessible as possible. Even with this disposition, until the 1990 decade there wasn’t really any active projects to publish or share scientific data. The specific term “Open data” was first mentioned in May 1995, in a proposal for the Congress of the World Meteorological Organization, concerning the disclosure of geophysical and environmental data. From the same paper we can read: “Atmosphere data transcends borders, they promote open exchange of information between countries, a prerequisite for the analysis of this global phenomena”. While this seems almost obvious nowadays, at the time even the sharing of “naturally global” atmospheric and weather data was not really taken into consideration. But the concepts behind Open data are way older than that, always considering the scientific world: from a theoretical point of view, in 1942 Robert King Merton, considered one of the fathers of sociology, wrote: ““Communism,” in the nontechnical and extended sense of common ownership of goods, is a second integral element of the scientific ethos. The substantive findings of science are a product of social collaboration

New Media for Scientific Data Visualization

and are assigned to the community. They constitute a common heritage in which the equity of the individual producer is severely limited. […] Property rights in science are whittled down to a bare minimum by the rationale of the scientific ethic. The scientist’s claim to “his” intellectual “property” is limited to that of recognition and esteem which, if the institution functions with a modicum of efficiency, is roughly commensurate with the significance of the increments brought to the common fund of knowledge. […] The institutional conception of science as part of the public domain is linked with the imperative for the communication of findings. Secrecy is the antithesis of this norm; full and open communication its enactment. […] The communism of the scientific ethos is incompatible with the definition of technology as “private property” in a capitalistic economy”.15 While it does not clearly mention Open Data in the way we consider it nowadays, he is endorsing it way ahead of its time, and even touching the theme of intellectual property, a topic that will be very dear to the Open source movement in the last decades of the century, that was also a great promoter of the Open Data ideals. Another important theoretical endorser of Open Data was the American political economist and Nobel prize Elinor Ostrom, that in 2009 stated that information commons are similar to public goods, because their use by one do not impede the use by other. Instead, information commons can be considered of a new kind: their use does not deplete the common stock, but rather it enriches it.16

Elinor Ostrom 15: Robert King Merton, “The Normative Structure of Science”, The Sociology of Science: Theoretical and Empirical Investigations

16: Charlotte Hess, Elinor Ostrom, “IDEAS, ARTIFACTS, AND FACILITIES: INFORMATION AS A COMMON-POOL RESOURCE”, 2003

Open Data

Early open data examples

Human Genome Project logo

One of the first notable example of the applications of some Open Data principles is from the 1950 decade: the International Council of Scientific Unions created the World Data Center system, in preparation for the International Geophysical Year of 1957/58. Several World Data Centers were established, to minimize the risk of data loss and maximize data accessibility. However, since there was no Internet, that was more like a distributed storage system, rather than an active data sharing platform. The first really Open Data-oriented project was the Human Genome Project, which is still the biggest collaborative biological project. Its planning started in 1984, the development begun in 1990 and was completed in 2003. It was carried out by twenty universities in United States, United Kingdom, Japan, France, Germany and China, under coordination and funding from the US government, with the ultimate goal of mapping the whole human genome, sequencing multiple variations for each gene.17

17: Kathering Harmon, â&#x20AC;&#x153;Genome sequencing for the Rest of Usâ&#x20AC;?, Scientific American

International Geophysical Year 1957/58 celebrating stamp

New Media for Scientific Data Visualization

Going back to consider Governative Open Data, the most important event on this side was the Sebastopol meeting: a reunion of thirty activists and thinkers near San Francisco, aiming to define the concepts of Open public Data, hoping to make it adopt to the US presidential candidates. Between them there were various important figures: Tim O’Reilly, most known for popularizing the concepts of open source and of Web 2.0; Lawrence Lessing, a professor of Law at Stanford Univeristy and founder of the Creative Commons licenses; Aaron Swartz, hacktivist and inventor of RSS, sadly known for his suicide in 2013 after being found guilty of wire fraud for trying to download and share a great number of academic journal articles. The Sebastopol meeting defined that public data should be considered a common property, just like scientific ideas, and it should be of interest to everyone for it to be shared and used. This concept did not come out of nothing, but it is strongly based on the concepts of open source software: openness, participation and collaboration. Coming from that environment most of the participants were aware of how good this model is working for free software development: programmers collaborate and are invited to publicly share their code, both to let others learn from them, and to create a collective exvvpertise. There are not really hierarchical rules, and the work between developers is based on a peer-to-peer collaboration model that promotes competence and reputation. Synthetizing, Tim O’Reilly said that “we must apply the principles of open source and its working methods to public affairs”. The meeting achieved even more than what they were expecting: in 2009 the newly elected president of the US Barack Obama signed three presidential memoranda about open government and Open Data, taking the open source culture at the heart of public actions, stating that “Openness will strengthen our democracy”. Precisely, he defined that the Govern-

The birth of public open data

Aaron Swartz at a Creative Commons event, 2008

Open Data

18: Barack Obama, â&#x20AC;&#x153;Transparency and Open Government - Memorandum for the Heads of Executive Departments and Agenciesâ&#x20AC;?, https:// www.whitehouse.gov/the_press_office/ TransparencyandOpenGovernment

Open data scores per country, evalued from the Global Open Data Index - http://index.okfn.org/

ment should be transparent, to promote accountability and because information is a national asset; participatory, to enhance public engagement, soliciting public input, thus enhancing the government effectiveness and improving its decision-making; collaborative, using innovating tools to create collaboration between Government, public and private sectors.18 This had a huge impact also for scientific data, because it triggered the opening of NASA and other governmental agencies datasets to the public. The situation of Open Data in governments is positively evolving and many states worldwide are moving towards policies similar to the one used in the US, even if the cultural changes required for it may be difficult to accept in some context. The benefits are anyway vastly superior that any possible concern, both in politics, allowing better transparency and citizenship participation, and economics terms, having created a whole new branch of possible activities based on data. What is incredibly relevant about Open Data for the Scientific Visualization field is that making scientific datasets available for the public created a great interest in designers and artist: being able to access that amount of data made possible to create entirely new data-driven concepts and projects both in art and design, that are creating fascinating ways to make use of those numbers, often crossing with fields like new media, interactive and installation art. Anyway, Open Data is not spreading as fast as it could, and it is facing some issues: the most important

New Media for Scientific Data Visualization

being the lack of interoperability and the heterogeneous demand. The first problem can be summarized with the lack of a universal Open Data standard: it is stated that data needs to be machine-readable, but not how, and so dozens of different standards and formats are used around the world. This does not just regard the file formats themselves, but also column subdivision and labeling, precision in number representation, and many more similar details that a developer must face when approaching different datasets. Open Data theorists themselves, after their great initial success and the general social acceptance of their ideals, also are slightly changing their points of view: for example, Lawrence Lessing dissociated himself from the ideas of radical transparency of the movement. Even Beth Noveck, that took part in the creation of the Open Data policies of the Obama administration, is not sure anymore that Open Data is being enough to improve the governance of public affairs. Other participants of the Sebastopol meeting even ended up supporting WikiLeaks. These minor difficulties do not mean that Open Data should be abandoned: they just mean that that was a first step, and we must proceed towards always better models of data sharing. Tim Berners-Lee is proposing, for example, Linked Data.

Beth Noveck

Open Data

The future: Linked Data 19: https://www.w3.org/History/1989/proposal. html

20: https://joinup.ec.europa.eu/community/ semic/news/understanding-linked-dataexample Tim Berners-Lee

Tim Berners-Lee is an incredibly important figure, that influenced the very way we communicate and perceive information nowadays: in March 1989, while working at CERN he defined and prosed a new information management system that became, in the November of the same year, the World Wide Web that we all know, successfully implementing the first HTTP communication.19 Currently he is director of the World Wide Web Consortium (W3C) and founder of the World Wide Web Foundation. Starting from 2009, he worked with the British government to support their Open Data project, that concretized itself in the data.gov.uk portal. Tim Berners-Lee research nowadays is focusing toward the future of Open Data: he is proposing the “Linked Data” standard, as a mean to solve the problems that traditional Open Data are facing. Linked data must respond to four simple rules: 1- Use Uniform Resource Identifiers (URIs) as names for things, e.g. http://dbpedia.org/resource/Brussels can be used for referring to the city of Brussels. 2- Use HTTP URIs, so that people can look up those names. 3- When someone looks up a URI, provide useful information, using the standards (i.e. RDF, SPARQL). 4- Include links to other URIs, so more things can be discovered, e.g. from http://dbpedia.org/resource/ Brussels a link is available to http://dbpedia.org/resource/Belgium.20 To make clear and “rate” how much Linked Open Data is considered good and useful, a rating system has been developed: 1 Star: The Data is available on the Web with an Open license. 2 Stars: Data is in a machine-readable format, like an Excel file instead of a scanned image. 3 Stars: Data is in a non-propertary format, like .csv instead of Excel. 4 Stars: Data uses a W3C open standard to iden-

New Media for Scientific Data Visualization

tify things, making the data easily linkable. 5 Stars: Data is linked to others dataset to provide context.21 This system is addressing mainly governmental agencies and any other entity that can produce and spread Open and Linked Data, to make clear which is considered to be the best and most useful way to share their data, focusing on shareability and possibility to link different contents. The most notable example of Linked Data nowadays is DBpedia22: it is a project born in 2007 from the Universities of Berlin and Leipzig and in collaboration with OpenLink Software, aiming at extracting all the structured content from Wikipedia, to make it available on the web and semantically searchable. This last aspect is a major difference over more â&#x20AC;&#x153;traditionalâ&#x20AC;? forms of Open Data: semantic research allows to query for associative and contextual information, both explicitly and implicitly, allowing to provide a single, precise piece of information, but also answer to open questions through techniques like pattern matching and digital reasoning. With this kind of technology, the value of information grows exponentially when adding new connections, al-

21: https://www.w3.org/DesignIssues/ LinkedData.html

22: https://www.ted.com/talks/ tim_berners_lee_on_the_next_web/ transcript?language=en#t-485180

DBpedia mapping in 2014

Open Data

23: https://www.ted.com/talks/ tim_berners_lee_on_the_next_web/ transcript?language=en#t-485180

24: http://www.geonames.org/

25: http://umbel.org/resources/news/umbel-v1-20-released/

lowing each dataset to be interconnected, effectively building a much more effective network of knowledge than what is possible today. To give an example, as of February 2016, the English DBpedia describes 5.9 million resources, with 4.3 million abstract, 425 thousand geo coordinates and 1.45 million depictions. 4 million resources are part of a consistent ontology and consists of 2.06 million persons, 682 thousand places, 376 thousand creative works, 278 thousand species and 5 thousand diseases. There are 128 languages in which DBpedia is localized, for a total amount of 38.3 million entries. All of this data is automatically selected, extracted, categorized and linked together by automatically scraping Wikipedia.23 Another remarkable example of Linked Data is represented by GeoNames, a geographical database containing 10 million geographical names, corresponding to more than 7.5 million unique features24; and UMBEL (Upper Mapping and Binding Exchange Layer), a knowledge graph of 35 thousands concepts that can be used in information science to relate information from different sources. UMBEL in particular makes heavily rely on the semantic research and interoperability for concept search, concept definitions and ontology consistency checking.25

New Media for Scientific Data Visualization

Open Data

art, science and design

separation of science and aesthetics

Observing both the historic examples and the more contemporary case studies taken into account from scientific visualization, it can be noted that they focus mostly on solving functional requirements, often overlooking on concepts like aesthetics or user experience. Also, they are often created by and aimed to experts in a particular scientific sector, requiring some level of familiarity and knowledge with the topic of the visualization. On the opposite side, the increasing public availability of datasets, APIs, and other forms of Open Data is creating a kind of “parallel universe” of artistic projects involving visualization, centered mostly on the aesthetic value, reversing the more traditional utilitarian approach, and attracting much attention from a new and wider kind of public, that maybe never really approached science and are just generally interested in art and culture: in this way, suddenly, they are experiencing a work of art but also dealing with subatomic particles, pulsars and weather data. “Come for the art, stay for the science”, it might be said. A cause of the low consideration of aesthetics in SciViz resides in the perceived impossibility to evaluate and rate objectively the attractiveness of any visualization, privileging different parameters, specificially utility and soundness. These three principles are considered the elementary requirements of design, and were identified for the first time in 25 BC, by the Roman architect Vitruvius in his book “De Architectura”. Relating these principles to information visualization: Utility is associated with functionality, effectiveness and efficiency, and can be considered the optimization of a visualization, presenting in the clearest way as much information as possible within the given constraints. It is often the main focus of academic research, since it can be evaluated by comparison of different methods and concepts. Soundness is a quality related to the algorithm

New Media for Scientific Data Visualization

used to generate a visualization, mainly to its reliability and robustness; but also takes a wider sense, including how efficient the code is, if it can take different kind of data, or tolerate errors. In an academic environment it is expected that the algorithm behind a visualization is clearly explained, and must fulfill some elementary stability and reliability requirements. It can be evaluated both with mathematical and empirical methods. Attractiveness refers to the aesthetic quality of a visualization, relating primarily to the appeal or beauty, but can be extended to consider other aspects like originality, novelty, and can also encompass the whole user experience matter. The problem with this principle is that, at least traditionally, there is no recognized way to objectively evaluate or it, therefore academic research in scientific visualization does not have any expectation in that regard26. This kind of reasoning has a fundamental flaw anyway: while it is obvious that “beauty” cannot be measured, it is also true that various studies proved that, when dealing for example with computer interfaces, the aesthetic quality can support utility, improving the user’s performances and satisfaction27.

26: Andrew Vande Moere, Helen Purchase, “On the role of design in information visualization”

27: D. Norman, “Emotion & Design: Attractive Things Work Better.” interactions 2002

Art, Science and Desig

Evaluating attractiveness 28: M. Kurosu and K. Kashimura, “Apparent Usability vs. Inherent Usability: Experimental Analysis on the Determinants of the Apparent Usability”. Conference on Human factors in Computing Systems (CHI’95)

29: N. Tractinsky, D. Ikar, “What is Beautiful is Usable”, Interacting with Computers

30: S. Bateman, R. Mandryk, C Gutwin, “Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts”. Conference on Human Factors in Computing Systems (CHI’10)

31: A. Lau and A. Vande Moere, “Towards a Model of Information Aesthetic Visualization.“ International Conference on Information Visualisation (IV’07)

32: David Chek Ling Ngo, Lian Seng Teo, John G. Byrne, “Modelling interface aesthetics”

33: Oliver Conolly, Bashar Haydar, “Aesthetic Principles”

It has been proved that the perceived usability of an interface is related to the subjective aesthetic judgment by the user28; another study revealed that users tend to spend more time using a software with a more appealing interface and are also more forgiving in case of errors or bugs29.Another study empirically demonstrated that “visual embellishments” considered non-utilitarian, like metaphors, positively influence long term recall in infographic charts30. Transferring the same reasoning to visualization, a highly valuable aesthetic applied to data may compel the user to engage with it, giving a higher value to the information presented and enabling a better communication. When it comes to evaluate attractiveness for visualization, some theoretical efforts to identify some kind of criteria have been made, most notably from Lau and Vande Moere. They identified three possible characteristics that determine the engagement of audiences on a visualization: design quality, generally including visual style and user experience; data focus as the core of the visualization is to communicate meaning instead of simple facts or trends; user interaction, that involves the aspects related to fluidity, engagement and social collaboration31. Another effort is from David Chek Ling Ngo, that proposed 14 different metrics to quantify the aesthetic layout of an interface: balance, equilibrium, symmetry, sequence, cohesion, unity, proportion, simplicity, density, regularity, economy, homogeneity, rhythm, order and complexity; backed by mathematical models and empirical analysis32. This extremely rational approach of course is not shared by everyone: others researchers believe that aesthetics can be evaluated without any measurable characteristics, and is ultimately founded in the human experience. So people can find an artwork or a visualization aesthetically pleasing without knowing “why”, in a non-discursive way33.

New Media for Scientific Data Visualization

While each and every project has its own equilibrium of art, design and science, nowadays it can nevertheless be clearly distinguished if they are addressed toward aesthetic value or scientific clarity, creating a sort of â&#x20AC;&#x153;buffer zoneâ&#x20AC;? that sits right in-between them and is currently extremely small populated, if not at all. Design could move in this zone, somehow bringing those two directions together and blurring the line to create a kind of visualizations that fuses them, maintaining readability of the data and also bringing in aesthetic value. This could improve the experience both of experts, creating better tools and practices to handle and work with data, while also creating new artworks to popularize science among the public, that readability and instructional value.

Art, Science and Desig

The role of Art

About the role of art and artist, there is a very interesting case study from Daniel F. Keefe, David B. Karelitz, Eileen L. Vote, and David H. Laidlaw at the Brown University Visualization Research Lab, where they investigated both the potential role of artists in designing new visualization techniques and in the impact that virtual and augmented reality technologies can have on the future of visualization. They got together programming, art and design students, and a team of scientists to explain the various tasks and topics they had to cover. They started from simple 2D fluid flow visualization, to scale up with increasingly difficult task as 3D and visualization in a Virtual Reality environment. It quickly became clear that artists can be very useful in the initial design and conceptualization stages, as they often provided new insights to the team. Working together with scientists they achieved a very effective feedback loop with them, also influencing the way data was collected to better suit the artistâ&#x20AC;&#x2122;s proposals. Cooperating with designers and programmers was also very proficient for they artists, since they often lacked the technical knowledge to convey their concepts. A useful tool that brought all the participants together was the critique and discussion moments, both on a visual and on a scientific standpoint. Proceeding in the work, some difficulties emerged when approaching VR: the main obstacle with it is the lack of prototyping tools that allow nonexperts to work with it. But since VR is such a unique media, it is almost impossible to evaluate and critique a design proposal without trying it in the appropriate environment, and nowadays this requires days of specialized programming work, thus breaking the possibility of quickly testing, alter or discard concepts, a methodology widely used and of proven effectiveness in the nowadays design practice. To work around this difficulty, the team realized a quick augmented reality sketching system for the artists, that involved

New Media for Scientific Data Visualization

a fictional paintbrush that traces “virtual 3D strokes”, that effectively sped up the work and allowed for a better evaluation and refinement process. Working with VR and data has proven to be a delicate work of balance, because while it allows to represent much more information in an amazing immersive environment, it must be clear from the beginning how a design idea will work with the actual data to display: it can happen to prototype a concept based on a personal perception of the data to find out that it seems effective, to alter discover that that concept does not go well with the actual data once it’s implemented, and considering how VR is difficult to implement right now it may turn out to be a considerable waste of time. In conclusion, this experiment shows a few important points on the relationship of art, science and design: artists proved that their creativity can be very useful in the conceptual and initial designing process to create new techniques, better if in a multidisciplinary team. When it come to virtual reality and similar novel and specific medias it also must be noted that the developing of rapid prototyping environments should be a priority in the next future, to allow for the experimentation of novel visualization techniques34.

Virtual Reality bloodflow visualization, created with the visualization prototyping system developed at Brown University’s Visualization Research Lab

34: Keefe D, Karelitz D, Vote E, et al. “Artistic Collaboration in Designing VR Visualizations”. IEEE Computer Graphics and Applications 2005

Art, Science and Design

CASE STUDIES

case studies selection

Please note that some tags will be counted more than once for tha last two cases, which embed more small projects.

There is currently a notable number of databased projects that crosses ways with art, design and science, using both more “traditional” forms of expression (graphics, video, sculpture…) and new and “non-conventional” medias (sound, interaction, installations). There is a selection of works in different field, including some “institutional” projects created directly by research center and agencies, and some works that does not directly involve data, but acts as means of science popularization, with a very good response from audience. Each project has tags to distinguish its field and features: 1 - R. Ikeda, Supersimmetry - Art, Installation, Particle physics, Music, Projections 2 - Miebacht, Weather Scores - Weather data, Sculpture, Music, Art 3 - A. Parker, Worlds - Video, Planetary Science, Design 4 - eCloud - Weather data, Installation, Art 5 - Living Light - Weather data, Installation, Design 6 - L. Pagliei, Les Invisibles - Music, Particle physics, Art 7 - J. Park, LHC CERN Data Audio Generator - Music, Particle physics, Application, Design, Science 8 - J. W. Tulp, GoldiLocks - Planetary Science, Web, Design 9 - In a nutshell - Web, Video, Animation, Design 10 - NASA Scientific Visualization Studio - Weather data, Planetary science, Science, Map 11 - NASA Hyperwall Visualizations - Planetary science, Science, Weather Data

New Media for Scientific Data Visualization

Tags recurrency

Science Weather Data Planetary Science Animation Design Art Music Map Installation Particle Physics Video Projections Sculpture Application

Case studies mapping

Design

09 08 05 04

01 06

Art

07 03 10 11

Science

Case Studies

Ryoji ikeda Supersymmetry Audio visual installation, 40 projectors, 40 computers w/ monitor, speakers, 2014

35: http://www.ryojiikeda.com/biography/ Ryoji Ikeda during a performance

Ryoji Ikeda is a Japanese sound and visual artist, whose musical interest is mainly in the most “raw” states of sounds, like fundamental tones and various kinds of noise, that he often uses to play with the limits of human earing, with extremely low or high-pitched sounds. His visual aesthetics is based on similar principles, often visualizing the sound he produces with massive, abstract and geometric black and white projection. He often used data, mathematics and physical concepts as “sources” for the composition of his works, and is considered a pioneer of minimal electronic music35. His project “Supersymmetry” was developed during his residence at CERN in Geneva in 2014 and 2015, where he had the chance to cooperate with the researchers working on the supersymmetry theory: at the time they were running experiments at the LHC particle accelerator to find some yet-undiscovered fundamental particles, like the Higgs Boson, that would prove the reliance of the Standard Model. Even if this work does not include data directly from the LHC experiments, it is trying to describe in an artistic and “experiential” way what these physics theories are and imply, so it is more of a conceptual project than an actual data visualization system. The installation itself is constituted of two sections, named “experiment” and “experience”: the first one revolves around observation of natural phenomena, which is exemplified with a number of small

New Media for Scientific Data Visualization

Supersymmetry (experience), whole view

Supersymmetry (experiment), particular

36: http://special.ycam.jp/supersymmetry/en/ work/index.html

Supersymmetry (experience), one of the monitors

pellets of different materials, moving on three lightened surfaces that are constantly changing their inclination. The pallets move following the angle of the surfaces, colliding with each other. A laser scanning system detects the position of each particle and sends it to the second section, â&#x20AC;&#x153;experienceâ&#x20AC;?, where sound and images are produced according to movements and collisions detected between the pellets. This creates a kind of detachments in the userâ&#x20AC;&#x2122;s perception: while they try to follow and understand at least one of the events, hundreds of other events are happening each second around them, in a way simulating the enormous amount of data and complexity that particle physics researchers have to face with each experiment36.

Supersymmetry (experience)

Supersymmetry (experiment)

Nathalie miebach Weather Scores Scuplture, music

37: http://nathaliemiebach.com/statement.html Nathalie Miebach

Nathalie’s work focuses on the intersection of art and science, taking data from astronomy, ecology and meteorology and transforming it into woven sculptures: in this way she uses the natural “grid” that basket weaving creates as a coordinate system to displace three-dimensional data. In Nathalie’s opinion, her work pushes the boundaries of traditional ways to interpret and visualize data, playing with the expectations of audience about what is considered to be “science” and “art”37. Additionally, since 2009 she collaborated with musicians and composers, to transform a series of sculptures she based on weather data into musical scores for them to play: this both gives a diverse emotional level to her primarily visual work, and might also reveal patterns in the data that may otherwise remain unnoticed. Her work can be considered quite rare in data art, mixing together media that are not really much used in the field, thus creating a unique way to see and experience the datasets she uses. Her work also maintains a very “analogic” feeling, relying on an art that requires a lot of manual work and skill as basket weaving and preferring traditional scores and human musicians for her music, rather than relying on electronics music or computer-guided composition as most sound/data artists uses to.

New Media for Scientific Data Visualization

The “Weather scores” exposition

Musical score for “Duet of Blizzards”

Case Studies

“External Weather, Internal Storms” 2009

New Media for Scientific Data Visualization

“O Fortuna Sandy Spins,” 2013

Case Studies

Alex H. Parker Worlds Animation, Music

38: http://www.alexharrisonparker.com/

39: http://www.alexharrisonparker.com/ datavisualization/

Dr. Alex H. Parker

Alex Harrison Parker is a planetary astronomer at Southwest Research Institute, currently collaborating with NASA’s New Horizons mission38. He is active also as a science popularizer, creating illustrations, video and musical pieces based both on scientific data and on topics from his research. For example, “Worlds: The Kepler Planet Candidates” is an animation video he created were he visualizes all the Kepler planet candidates around a single star, each one of them with their real diameter, orbit, and speed. It was awarded with the CinèGlobe “Data Visualization Award” in 201339. The value of this work resides in the very straightforward and clear way it presents such a rich topic: it can be considered a very good work of design rather than art, but nevertheless very interesting, giving in little more than 3 minutes a good perspective on the incredible amount and variety of exoplanets that the Kepler space telescope is discovering. Other projects from him include “Painted Stone”, a visualization of 100,000 asteroids and orbits, observed by the Sloan Digital Sky Survey, with the color representing their material composition; “Kepler 11: A Six-Planet Sonata”, a sonification of the transits of the six planets in front of their star in the Kepler 11 planetary system, that demonstrate through sound the orbital resonances that system; “Beyond Neptune”, an animation about trans-Neptunian objects, visualizing when they were discovered, their size and orbit.

New Media for Scientific Data Visualization

Worlds

mid: Beyond Neptune bottom: Worlds

Painted Stone

Aaron Koblin, Nik Hafermaas, Dan Goods eCLOUD Dynamic sculpture, 2010

40: http://www.ecloudproject.com/tech.html

41: http://www.ecloudproject.com

eCLOUD is a dynamic sculpture made from suspended polycarbonate tiles that can change from transparent to opaque, to simulate the volume and behavior of an idealized cloud. It was created in collaboration with NASAâ&#x20AC;&#x2122;s Jet Propulsion Laboratory and the Art Center Collage of Design. The system takes real-time weather data from NOAA and simulates how a cloud would behave in the conditions of the picked location40. The project includes a dynamic display system that shows a preview of the simulated cloud behavior, the current location, and details about the weather data received. eCLOUD was created for and is installed at the San JosĂŠ International Airport as a centerpiece artwork, as part of the Art & Technology program41. This is a valuable example of the possible intersections of data, art and new media: it uses an innovative technology to display real-time data through a very effective visual metaphor, that people passing by can easily understand, together with a notable and positive aesthetic value.

New Media for Scientific Data Visualization

eCloud installed in the San Jose Airport

mid: eCloud installed in the San Jose Airport bottom: demonstration of the liquid crystal tiles technology

eCloud dynamic display

Case Studies

Soo-in Yang David Benjamin Living Light Public art installation, 2009

42: http://www.interactivearchitecture.org/ living-light-2.html

Living Light is a functional art installation located in Seoul, created to inform the citizens about the air quality in the city. The canopy has a city map etched on its glass surface, and is divided in 27 different blocks representing the different areas of the city where the air monitoring stations are situated. When an improvement in the air quality of a district is detected, the corresponding block of the system lights up; otherwise the system is automatically updated every 15 minutes, lighting up all the blocks in order, from the one with the best air quality to the worst one. The structure also responds to text messages, blinking and then sending the data about the specific location that the user requests. â&#x20AC;&#x153;Living Lightâ&#x20AC;? is fully integrated with the already existing air-quality monitoring system active in Seoul, receiving data from its sensors network42. This is a notable example of public art: its beautiful design became an attraction for the city itself, and it is also very useful for the population, informing and raising awareness about the condition of air and pollution in the city.

New Media for Scientific Data Visualization

Living Light installed at Peace Park, Seoul

Lorenzo Pagliei Les Invisibles Sound Installation, 2001 / 2015 43: http://www.lorenzopagliei.com/site/bio.html 44: Personal interview with Lorenzo Pagliei, February 2016

Lorenzo Pagliei

Lorenzo Pagliei is an Italian composer, electronic musician, pianist and orchestral conductor, currently living and working in Paris at IRCAM as composer and researcher. His work focuses on sounds not realizable with acoustical instruments and innovative ways to interact and control sound43. In 2001 he took part in a collaboration effort made from the Centre des Arts of Paris, aiming at making scientists and artists work together on a piece of art involving concepts from physics and nanotechnology: Lorenzo worked with the Centre national de la recherche scientifique – CNRS, that was currently working on nuclear magnetic resonance. It’s a new technique to investigate the properties of molecules and nanoparticles, that works by analyzing the particle’s oscillations when hit by an electromagnetic pulse. This kind of data is almost directly translatable into sound, since particles and molecules threated this way behave like regular objects that get hit, vibrating and thus generating sound. Analyzing this oscillating response from molecules allow to deduce its composition and various other properties. Each particle generates a different sound or rhythm, that were used as the composing material for a 1-hour music piece created for the exposition. “Les Invisibles” was performed again in October 2015, in the Crypt of the Ville d’Orsay, this time as a realtime performance controlled live by the composer44.

New Media for Scientific Data Visualization

Sonogram of a section of â&#x20AC;&#x153;Les Invisibilesâ&#x20AC;? mid: Setup for the live performance botton right: Developing of the sound synthesis system bottom left: Visualizations of the results of some experiments

JeongHo Park LHC CERN Data Audio Generator Data analysis Software

45: http://jeonghopark.de/z

46: https://github.com/jeonghopark/lhcCernMac

Jeongho Park is a developer and audiovisual artist that created various data-oriented and interactive projects, applications and installations45. This project is a software application that generates a tridimensional visual models and a sonification from event data of the LHC particles accelerator at CERN in Geneve. Some events are bundled with the application, and new ones can be download from CERN website, since the application is natively compatible with their databases. Each line of the visualization is the path of a particle, and when an event is â&#x20AC;&#x153;playedâ&#x20AC;? also a sound from each one of them is generated, with a different frequency according to its mass and properties. Tools like this prove the didactic and research value that an aesthetic approach can have, giving a useful and explorable 3D view of an event, while the sound gives information in an intuitive way, making easy to spot the general layout of an event, or quickly pinpointing abnormal or strange particles, that will stand out for their different tone.

New Media for Scientific Data Visualization

Screenshot from the application

Interface of the application

Case Studies

Jan Willem Tulp Goldilocks Data Visualization, website, 2015

47: http://tulpinteractive.com/goldilocks/

The â&#x20AC;&#x153;Goldilocksâ&#x20AC;? name comes from the homonym effect, that states that some phenomena must fall within certain margins rather than reaching extremes: this principle is applied to astrobiology to determine when a planet could theoretically support life, since it depends on a series of parameters that must fall within certain boundaries47. This interactive visualization displays all the known exoplanets (1942 confirmed, at October 9th, 2015) and their host stars. Size and orbital parameters of the planets are based on the observed data, as well as star size, color and position. Alongside the main visualization, displaying all planets around their stars in their respective positions, there are five other visualizations, more dataoriented, focused on: distance of each planet from their star, underlying the ones in their systemâ&#x20AC;&#x2122;s habitable zone; Earth Similarity Index; atmosphere and composition compared to Earth; temperature and mass; temperature of their stars and amount of energy they receive from them.

New Media for Scientific Data Visualization

Various visualizations from http://goldilocks.info/

kurzgesagt In a Nutshell YouTube channel

48: http://kurzgesagt.org/profile/

49: https://www.youtube.com/user/Kurzgesagt

“In a nutshell” is a YouTube channel started in 2013 by a Munich design studio, working with animation and information design48. Their videos deal with various topics, like physics, sociology, computer science, biology, astronomy and politics, and always explain a particular subject related to some contemporary matter, to mention the most recent topics: the war on drugs, what are red dwarf stars, what is a quantum computer, how Facebook steals video views, and the refugee crisis. It’s always some kind of demanding argument, that may somehow “scare away” people looking for entertainment: but the channel has nevertheless almost two million subscribers and is getting close to a hundred million visualizations, thanks to the their engaging style of drawing and animations, and for the easy but informative way they deal with these topics49. Very little of their content is related with data itself, even if they use data visualization from time to time, but the value of this case is in the way they are doing science popularization and information, in a way demonstrating that it is possible to talk to regular people about “difficult”, niche and specialized arguments with the right design language.

New Media for Scientific Data Visualization

Various still frames from â&#x20AC;&#x153;In a nutshellâ&#x20AC;? videos

Case Studies

Nasa Scientific Visualization Studio

NASA Scientific Visualization Studio works to popularize various contemporary science topics covered at NASA, like Earthâ&#x20AC;&#x2122;s climate or planetary observations. They also cover various techniques, from simple picture elaborations or combinations to animations and various data visualizations.

50: https://svs.gsfc.nasa.gov/cgi-bin/details. cgi?aid=12094

Projects between the most popular include: Human Fingerprint on Global Air Quality: thanks to to high-resolution global satellite maps, NASA tracked air pollution trends in the last decade around the world. They compiled a map summarizing that data, and a video to explaining how and why air quality is changing aroud the globe. This project demostrates how valuable emission control laws are being in the United States and in Europe, observing how general air quality improved in these regions in the last years, while emerging and less regulated countries like China or India are seeing a decrease in air quality. Air pollution is linked to other factors too: for example since the beginning of the civil war Syria is decreasing in population due to emigration and most economical activities are shutting down, thus, ironically, leading to a better air quality. Data is collected with the Ozone Monitoring System aboard the NASA Aura satellite50.

New Media for Scientific Data Visualization

“Human Fingerprint on Global Air Quality” map

Case Studies

Frames of “Five-Year Global Temperature Anomalies from 1880 to 2015” 51: https://svs.gsfc.nasa.gov/cgi-bin/details. cgi?aid=4419

Five-Year Global Temperature Anomalies from 1880 to 2015: This project from NASA Scientific Visualization Studio shows the temperature anomalies on Earth’s surface from 1880 to 2015: this means that the colors does not represent the actual temperature, but the difference from the normal temperature. Data is taken from the GISTEMP (GISS Surface Temperature Analysis, NASA Goddard Institute for Space Studies) analysis system, combining data from NOAA GHCN v3 (meteorological stations), ERSST v4 (ocean areas), and SCAR (Antarctic stations)51. Dial-a-Moon: This project is mainly a picture visualizer, that takes a date and time as input and outputs an image of the Moon in that moment, plus some textual informations about phase, apparent diameter, distance and precise positioning. The very high resolution and detail of the pictures comes from the fact that they are taken from the Lunar Reconnaissance Orbiter, a NASA probe orbiting the Moon, that pairs its LROC camera to a laser altimeter, LOLA, allowing for a precise mapping of our satellite’s surface. It is also possible to download a high resolution map for each picture, with the name of each location written on it. Together with the visualization are presented some video explaining related topics, for example how the Moon rotates around Earth and what its libration is52.

New Media for Scientific Data Visualization

Space Weather: This animation shows the hypothetical voyage of a photon of light from the center of the Sun to the Earth’s atmosphere. The video displays also various phenomena that are encountered in such a trip: it shows nuclear fusion and ignition processes inside the Sun’s core; loops, flares, prominences and coronal material ejections on the surface; and then follow the solar particles until they reach Earth’s magnetic field, causing aurora53.

Still frames from the “Space Weather” video

52: https://svs.gsfc.nasa.gov/cgi-bin/details. cgi?aid=4404

53: https://svs.gsfc.nasa.gov/Gallery/ SpaceWeather.html

Case Studies

Nasa Hyperwall Visualizations

Dionysius Craterâ&#x20AC;&#x2122;s rim

The HyperWall is a visualization system with 8x16 = 128 LCD modular screens, backed up by 128 computational nodes and connected to the Pleiades supercomputer (a 11312 multi-CPU nodes calculator). NASA internally develops various kind of media and simulation for this system, leveraging both on the huge screen size available and the massive computing power behind it. The visualizations include: Dionysius Crater: Itâ&#x20AC;&#x2122;s a series of pictures taken from the Lunar Reconnaissance Orbiter of a crater in the Sea of Tranquility, interesting for non-uniform and striated materials of which the crater seems to be composed.

NASA’s Hyperwall

Pluto’s surface composition: A series of picture of the nano planet Pluto, taken with various instruments aboard the New Horizons probe. Together with regular pictures, other instruments toke images in other wavelengths to investigate Pluto’s surface composition. Also, High-resolution b/w images and low-resolution color images where combined to obtain some of the most iconic pictures released.

Observations on Pluto’s surface

Case Studies

GEOS-5 Nature Run Collection: This series of massive visualizations displays data from various simulations about weather and changes in Earthâ&#x20AC;&#x2122;s atmosphere, run with the NASA Goddard Earth Observing System Model (GEOS-5). Topics covered include the evolution of surface temperatures, clouds formation and movements, presence of water vapor, dispersion of aerosol from dust, biomass burning, fossil fuel emission and volcanoes, and winds. The full simulation took 2 years to complete, running on 3750 processors on the Discover supercomputer at the NASA Center for Climate Simulation, producing 400 terabytes of data.

On the other page, top to bottom: Earthâ&#x20AC;&#x2122;s surface temperature (colors) and outgoing radiation (white); surface (white) and upper level (colors) winds; Dust (red), sea salt (blue), organic/black carbon (green), and sulfates (white). On this page, top: total precipitable water (white) and rainfall (colors, red is highest); middle: aerosol and human-initiated burning simulation.

Case Studies

top: sea-surface temperatures visualization mid left: sea-surface temperature anomalies visualization mid-right: accumulated thermal stree visualization

NOAA Coral Reef Watch 2015: A series ofanimated Data Visualizations showing the real time conditions of reef environments, helping to identify where the corals are losing their symbiotic algae, so losing their color and bleaching: a condition that heavily endangers the survival of entire colonies. Data is taken from a variety of satellites, including the polar orbiters Suomi-NPP/VIIRS and MetOp-B/ AVHRR, and the geostationary satellites MSG-3, MTSAT-2, GOES-East, and GOES-West. Data itself consist mainly of temperatures, from which various other indexes are calculated, according to average or instantaneous values.

New Media for Scientific Data Visualization

Solar Wind Strips the Martian Atmosphere: While today Mars is covered by a dry desert and has a very thin atmosphere, scientists agree it used be quite different, with a thick atmosphere and probably bodies of liquid water like seas, lakes and rivers, similarly to modern Earth. The change happened because the atmosphere dispersed into space, and the most probable cause is solar wind: in this video the simulations made to verify this theory are confronted with the MAVEN probeâ&#x20AC;&#x2122;s observations of ions leaving Marsâ&#x20AC;&#x2122; atmosphere to disperse in space.

top: Solar wind hitting Mars mid left: Mars atmosphere stripped away mid-right: Ions speeds observed by MAVEN

Case Studies

a design-centered evolution

Design as the unifying force

Analyzing projects from both scientific visualization and data-driven art, looking at their history and at some case studies, it emerges that, while these disciplines underwent a great evolution and has reached very remarkable levels of advancement, there is still room for notable improvement, especially if we consider the role that design methodology could have in the future of these disciplines. Nowadays, in fact, SciViz and data art projects are running in two different and almost parallel directions, with almost none contact points; but they could both benefit a lot from some kind of interaction between them. Design could be the force driving them to cooperate, being almost “equidistant” from both of them, but still close enough to be able to interact with both. Recognizing the potential of design for these fields and involving more designers into the developing processes of scientific visualizations and data art would bring a more reliable design methodology, that would improve, for scientific visualization, important aesthetic qualities like the overall experience and the design of interfaces, easing the work for researchers and allowing the creation of better popularizing and generally public-distributed material; at the same time data-oriented art projects could rely on a more solid work methodology and on a good technologic know-how. There are various examples of designers and researchers endorsing the potential of art: to mention one, in the report made for a project experimenting visualization techniques for Virtual Reality, Daniel Keefe and his colleagues stated concluded that “artists [have] key roles in working closely with scientists to design novel visual techniques for exploring data and testing hypotheses”, and strongly suggest a interdisciplinary approach to visualization, “Teaching art to computer scientists, computer science to artists”.

New Media for Scientific Data Visualization

With the technological evolution of the last decades and the popularization of developing and easy-to-use creative tools, itâ&#x20AC;&#x2122;s always more common to see projects that involve interaction, immersive video projections, sound and similar state-of-the-art features. In this way the general audience is becoming more and more acquainted to this forms of media, and always more difficult to impress. This implies that any forward-looking project must take into serious consideration the usage of the most interesting and appealing of these technologies: it must not be modernism just for the sake of it of course, but it must be taken into account during the design process that it may be more useful and far more compelling for the audience to rzepresent a certain phenomenon, for example, with sound, or giving the possibility to explore a particular dataset through virtual or augmented reality. Itâ&#x20AC;&#x2122;s obvious that a weak design idea wonâ&#x20AC;&#x2122;t magically acquire value thanks to the use of any technology, but the right combination of a well-thought concept and the appropriate technical mean will much more easily appeal and attract the public, and the more time they will spending interacting or observing the project the more it will possible for the to become interested in the underlying information and

Usage of New Media

Example of projection mapping, a new media that can be seen more and more in public events

A Design-centered evolution

Interactive virtual reality simulation of a brain tumor removal surgical operation.

54: Teodor P Grantcharov, “Is Virtual Reality Simulation an Effective Training Method in Surgery?” - http://www.medscape.com/ viewarticle/575183

55: https://virtualrealityreporter.com/virtualreality-surgical-medical-application

56: Robert M. Candey, Anton M. Schertenleib , Wanda L. Diaz Merced, “Xsonify sonification tool for space physics”

57: Wanda Diaz Merced, “Sonification for the exploration of 1D Astrophysics data” - South African Astronomical Observatory Colloquium, 17 April 2014

topic. It must also be considered the value of new media for researchers and professionals, for example: in the medical field the use of virtual reality has already proven its outstanding value both for surgical training and as assistance during surgeries54, and some theorize that technologies such as Microsoft Handpose, an extremely precise hand-tracking system, will be paired with already-existing robotic precision surgical arm to use during operations, allowing much more accurate actions and preventing human mistakes55. The use of sound is also proving its usefulness in the field of astrophysics research: NASA is using data sonification through a software called “Xsonify” to translate observations made in various fields into sound. For example, they translated x-ray observations, radio signals from Jupiter and micrometeorites impacts on the Voyager II probe56. The project was born to help visually impaired researchers, but it has been found to be useful also for normally seeing individuals: for example, for x-ray observations data was usually visualized, but that method proved to be scarcely effective due to the high level of noise and the difficulty to visually identify events and patterns. Moving to a sound-based system proved to be a better solution to pinpoint different kinds of events, that presents themselves as different timbres, or patterns, audible as rhythms57.

New Media for Scientific Data Visualization

Considering projects directed at the general public also the choice of the topics to cover can be relevant: while most of the population has only a general and superficial science knowledge, often as old as their high school years, most of them have heard from media of various relevant events in contemporary science. For example, they might know about Higgs Boson observation at CERN, the New Horizon probe reaching Pluto, the Zika virus, the discovering of gravitational waves, or got interested in fields touched by movies like Interstellar: this can be made use of to engage people in a project and going deeper on the same topic. It may also be interesting because most of contemporary science topics are full of open and unanswered questions, and being presented with an indevelopment topic can be far more engaging, pushing the audience to wonder and make hypothesis, rather than dealing with a fully known and already disclosed argument. These kind of discoveries are also changing our point of view on reality itself and may make way to life-changing technologies or scenarios: this also is a possible engagement factor, and at the same time a knowledge to spread in order to limit misinformation and anti-scientific propaganda.

Coverage of contemporary and popular science topics

Still frame from a video from â&#x20AC;&#x153;Kurzgesagt â&#x20AC;&#x201C; In a Nutshellâ&#x20AC;? explaining quantum computing

A Design-centered evolution

Usage of datasets and real-time data sources

Progress in data-collection techniques in the last years has been incredibly fast and prolific, creating the necessity of always more refined and powerful data analysis and visualization algorithms. So, doing research and experimenting new techniques with big datasets and real-time data sources, even if only for aesthetic purposes, can lead to the creation of novel systems that may be useful also in the scientific research field. Also using real-time data sources can increase the user’s engagement in case of popularization-aimed or exposition projects, creating always-changing scenarios that can also bring useful information to the public, like in the “Living light” project by David Benjamin and Soo-in Yang, “eCloud” by Aaron Koblin et al. and other public data display projects, that serves both as beautiful “public art” or “urban furniture” and as providers of useful and meaningful informations.

NASA Deep Space Network Now visualizes in real-time all communications with far away spacecrafts and probes

New Media for Scientific Data Visualization

This is of course the most important requirement for a project aimed at pure research and investigation purposes, which would be useless if the data is not somehow clearly readable and understandable. Data can be accessed even if an indirect way: an easy expedient that a lot of works use is to present a general overview of the whole involved data in the main screen, while allowing the user to select one or more data points to access the details about it. This point may be important also for projects directed to the general public, allowing for a more precise understanding of the presented phenomena and maybe also giving the piece an instructional value. If the data is difficult to understand, maybe because of its unit of measure, or for the very wide order of magnitude, a number of devices can be used to overcome the problem, for example translating the raw numbers with some metaphor related to some more understandable measures, or related to everyday objects. Instead of using magnitude for the brightness of a star the value can be related to the brightness of a far-away light bulb, or power consumption/absorption can be related to the amount used by home appliances.

Keeping the data understandable

A Design-centered evolution

Application of design reasoning and verifiable methodology

3D visualization design for the bat flight problem inspired by the Miró painting The Gold of the Azure, one of the results of Keefe’s project

A fundamental step in any data visualization project is to map data to visual attributes and structures. This procedure, to be effective, requires a high degree of design reasoning, especially if it involves the creation of new techniques: in this case it is required to invent an original visual metaphor that fits well with the abstract structure of the data. Considering the three space dimensions, time, all the possible transformations in positioning, size, color and so on, there are almost infinite possible combinations of data elements with visual parameters and attributes, so data-to-form decisions need to be explicit. Even if design reasoning is always present, it may be unconscious, and if this happens the designer must pay attention to reconstruct and document the path that brought him/her to those particular decisions, and try to objectively evaluate or even test them. Even if there is no “right” or “wrong” in this field, there are various ways to support a design choice: from established empirical results, like the Gestalt laws, to various studies on human perception, or by feedback from other designers or the users themselves. Building up this kind of objective documentation on visual mapping decisions, including the use of new technologies, will increase the amount of good examples and best practices that future projects will be able to build upon, in a very open-source manner. While this aspect may be already of use for some or a minor addition to some other’s designer workflow, it may a different matter for the art universe: since it’s a personal form of expression from the artist to the world, it’s not obvious that the author has the intention to explain the thought process behind his/her artwork, or it can even be interested in giving the idea that the inner processes behind an artwork are just purely irrational, or dreamt up, to leave the work open for different possible interpretations. However, if a piece is dedicated to a purpose as science popularization or raising awareness about some

New Media for Scientific Data Visualization

phenomenon like global warming, it should be considered in a slightly different way. Artists working in this particular sub-field should be aware that the value of their work can be also in the research and in the thinking they made and in the rationale in their decisions; and that their work has the possibility to ease scientific research if they invent a valuable data rendition technique. There are some cases of good documentation of the design processes in the last years: for example, Daniel F. Keefe et al. explains in a IEEE Computer Graphics and Applications article the process and the results of a project that involved both professors and students to develop new visualization techniques in a virtual reality environment. This kind of documentation can be extremely valuable: VR as a working technology is still in its first years, so there is no consistent literature or examples about possible uses in design and visualization; in fact, also Keefe and his colleagues highlighted the complete lack of design tools to rapidly sketch and prototype58. Also Byron and Wattenberg explained in detail in a paper the design rationale behind their Streamgraph, an new kind of stacked graph, from the algorithmic level to every visual choice they made59, creating a collection of generalizable knowledge that can benefit other designers and developers who want a similar approach. Together with the current approach of interaction design, were “aesthetics to be an integral part of functionality, with pleasure a criterion for design equal to efficiency or usability”60, these and other examples show and explain the potential of integrating design thinking in the research process, how aesthetics is valuable to any visualization, and how it can be somehow estimated and evaluated.

Byron’s Streamgraph example

58: Keefe D, Karelitz D, Vote E, et al. Artistic Collaboration in Designing VR Visualizations. IEEE Computer Graphics and Applications 2005; 25(2): 18–23. 59: Byron L and Wattenberg M. Stacked Graphs - Geometry & Aesthetics. IEEE Transactions on Visualization and Computer Graphics 2008; 14(6): 1245–1252.

60: Gaver B, Dunne T, and Pacenti E. Design: Cultural Probes. interactions 1999;

A design-centered evolution

Project: Sounds from the sun

Sound and psychoacoustics

100

Vision is widely considered to be the most important and defining sense of human beings: it is indeed true that the ability to see colors, for example, gave a great evolutionary advantage to our ancestors; and it is also one of the rarest and most powerful features between mammals, reserved almost only to humans and some great apes. At the same time, the hearing system of mankind is considered to be less evolved and refined than the one of other species of mammals: most of them have a wider range of audible frequencies, are more sensible and can better localize the source of sounds around them. The hearing system of humans was, and still is, nevertheless fundamental for the survival of our specie, being the most effective way to detect and localize possible dangers: a sudden noise startles us, and triggers in the brain and in the body a chain reaction that gets us ready either for fighting or for fleeing. The effectiveness of this system is due to a series of different reasons: -Hearing is much quicker than seeing. Vision is a slow chemical process: molecules in the retina change their characteristics when struck by light, this change is transformed into electrical signals that the brain has to deeply elaborate to transform them in the images we see. This process is considered to take between 1/30 and 1/20 of a second, resolving in the general knowledge that we can see no more that 20/30 different pictures per second. We can express this as a value with a unit of measure: 30 Hz. Meanwhile hearing is based on a quick mechanical process, with the hearing bones vibrating and exciting a nerve connected to the brain. In result this process is much quicker, allowing mankind to process sound with a resolution 1.000 times greater, around a maximum value of 20.000 Hz. In other words, the shorter event that the eyes can perceive has to last at least 50 milliseconds, while the shortest hearable event can

New Media for Scientific Data Visualization

last around 0.05 milliseconds. - Vision is focused in front of us, while we can hear all around us. Humans, mammals and other animals can only see in front of them, and checking all around us require to move our whole body, and this process of course takes some time. We also have to focus on what we’re seeing to recognize if something may be of our interest. On the other hand, hearing is always working in every direction around us, and it usually takes way less than a second to identify the location, possible source and our possible interest of a sound. - We can close our eyes, but never our ears. We can effectively “shut down” our seeing in any moment by just closing our eyes, becoming visually isolated from the surroundings, but there is no effective way to shut down our hearing. Even while sleeping, it is easier to be awakened by a noise than by a change of light. This is linked to other interesting features of our hearing system: for example, it is very easy to become acquainted to, ignore, and do not hear anymore the majority of background noises. A constant sound is taken for “granted” from our hearing system, and so ignored to focus on more interesting sounds, while we instantly notice any new sound that appears around us, or even any relevant change in the sound scape surrounding us.

Project: Sounds from the Sun

101

applications to data

102

The peculiarities explained, together with others of minor relevance, can have some interesting consequences. For example, the extreme precision of the hearing system enables us to easily detect both established patterns and irregularities in them: for example, we instinctively detect and recognize temporal patterns as rhythms, often following them with some kind of body movement. In the same manner we can notice if something breaks out of an established pattern, breaking a rhythm. The mechanisms applies in the frequency domain: we instantly distinguish an harmonic sound (where all sounds have a frequency multiple of the same number) from any kind of non-organized sound, that we call noise. If any kind of “harmonic context” is established, like an orchestra playing a classical piece, any kind of sound incoherent with that background pops up, for example a badly tuned instrument. This kind of ability has, in fact, already proven its value for data investigation: NASA is using it with its “Xsonify” software, to translate observations made in various fields into sound. For example, they translated x-ray observations, radio signals from Jupiter and micrometeorites impacts on the Voyager II probe. The project was born to help visually impaired researchers, but it has been found to be useful also for normally seeing individuals: for example, for xray observations data was usually visualized, but that method proved to be scarcely effective due to the high level of noise and the difficulty to visually identify events and patterns. In this specific case, the ability of our hearing system to ignore background noise made easier for researcher to isolate interesting events, while temporal patterns in the data that corresponds to repeating or cycling phenomena are easier to find, also quickly spotting irregularities that are much more difficult to identify with traditional visualization.

New Media for Scientific Data Visualization

The Sun is Earth’s source of energy, in the form of light and heat, but its activity does not limit to that: it also emits every other form of electromagnetic radiation, from radio to gamma waves, together with a massive stream of plasma and charged particles, like protons, electrons and alpha particles. Its magnetic field extends well beyond the eight planets of the Solar System: this is constantly changing due to Sun’s rotation and activity, and has a notable influence on each planet. The variations in these phenomena leads to peak values that can influence our life on Earth: the most harmless and spectacular being aurora borealis and australis, caused by the interaction of high amounts of charged particles from the Sun with Earth’s magnetic field. The most powerful emissions, classified as Solar flares, Coronal mass ejections and Geomagnetic storms, in order of magnitude, can have far less desirable effects on our technological lives: they can disturb and create malfunctions in the communication and geolocation satellites in geostationary orbit; on Earth they can disturb and interrupt wireless networks, like cellphone network, influence the behavior of compasses and other location and orienting devices, and even induce huge electrical currents in the Earth’s crust. To mention the most catastrophic event linked to solar activity, a geomagnetic storm occurred on March 13 in 1989 caused first the lost of control of various satellites in polar orbit around Earth, interrupted the communication with GOES weather satellites, caused anomalous operations to NASA’s TDRS-1 communication satellite, and after only 90 seconds broke the majority of power breakers in Quebec power grid, causing a massive 9-hours blackout on a large area of Canada61.

the Solar Wind and its consequences Above: a mid-level solar flare emitted on Jan. 12 2015, pictured by NASA’s Solar Dynamics Observatory

61: Morin, Michel; Sirois, Gilles; Derome, Bernard (13 March 1989). “Le Québec dans le noir” (in French). Radio-Canada: http://archives. radio-canada.ca/environnement/catastrophes_ naturelles/clips/14150/

Project: Sounds from the Sun

103

Solar Wind Observation

62: Morin, Michel; Sirois, Gilles; Derome, Bernard (13 March 1989). “Le Québec dans le noir” (in French). Radio-Canada: http://archives. radio-canada.ca/environnement/catastrophes_ naturelles/clips/14150/

Seeing the consequences that Sun can have on Earth, the close observation of its activity has grown in interest in the last decades: since 1995 NASA and ESA launched 4 different satellite probes to monitor solar wind: Solar and Heliospheric Observatory (SOHO), Global Geospace Science (GGS) Wind, Advanced Composition Explorer (ACE) and Deep Space Climate Observatory (DSCOVR). Observatories in outer space are much more effective than Earthbased ones: our planet’s surface is almost completely shielded from solar wind by its magnetic field, thus making impossible any kind of direct observation, and only some indirect evaluations can be made for example observing some atmospheric phenomena. All of these probes are orbiting between Earth and Sun at a distance around 1’500’000 kilometers from our planet: their signals take around 5 seconds to reach us, while solar wind has a largely lowest speed and takes around one hour to cover the same distance62. This gives us at least a minimum time to get ready for exceptional solar activities, to minimize the consequences on our technological infrastructures, since as of today we have no model or way to forecast solar activity: in fact, this is another challenging

ACE satellite during preparation and testing phases

104

New Media for Scientific Data Visualization

objective for astrophysicists, that hope, with the data collected by these probes, to design a model of solar dynamics good enough to forecast solar wind and its peak activity. The forecast of dangerous events in solar wind is currently based on the ACE Real Time Solar Wind program, which issues warnings and alerts for anyone with systems sensitive to high levels of magnetic activity. Four of the instruments on board of ACE produce a low-resolution continuous stream of data that can be received from different stations on Earth, the stations forwards the data to the Space Weather Prediction Center in Colorado, which delivers it to other centers around the world and is also published on the World Wide Web. This whole process takes 5 to 10 minutes, from the measurements took by ACE instruments to the to the publication on the SWPC website, available on http://www.swpc.noaa.gov/ products/ace-real-time-solar-wind.

The solar arrays on NOAAâ&#x20AC;&#x2122;s Deep Space Climate Observatory spacecraft, or DSCOVR, are unfurled in the Building 1 high bay at the Astrotech payload processing facility in Titusville, Florida, near Kennedy Space Center.

Project: Sounds from the Sun

105

Sound rendition of Solar wind data

Considering on one side the particularities of our hearing system, the differences with visual perception and, on the other, the continuous and realtime nature of the data taken into account, it can be interesting to consider a different approach to the rendition of solar wind data, that leverages on different mechanisms of perception rather the purely visual one; and since this kind of data needs to be constantly monitored, but at the same time is almost constant or with small variations of little interest, it can pair well with the innate abilities of our hearing system to ignore constant background sounds, while promptly reacting to changes in it, without requiring the specific attention and focus of the listeners, that can, in this way, prioritize other more demanding tasks. So, in my project I took into account the different phenomena and their parameters monitored by the ACE Real Time Solar Wind program: energy of charged particles stream, components of the solar magnetic field and density, speed, temperature and density of the plasma stream. Each phenomenon is translated with a specific sonification system that reflects its characteristics: The particle stream is rendered by a discrete stream of microsounds, where the energy is mapped to the frequency of the sounds;

Data about Magnetic Field & Solar Wind Electron Proton Alpha Monitor from the SWEPAM instrument, as presented on http://www.swpc. noaa.gov/products/ace-real-time-solar-wind

106

New Media for Scientific Data Visualization

the plasma flux is rendered by a noisy, continuous sound, where the temperature is mapped to the timbre of the sound, the density to width of the noise and the speed to the central frequency; the magnetic field is rendered by an harmonic sound similar to a chord, where the total magnitude is the fundamental pitch and each component of the magnetic field corresponds to the volume and beating cycle of each harmonic. These choices are not casual, and are made for different reasons: first of all, metaphorical, their perceptions mimic the nature of the phenomenon they replicate, and the sound parameters are mapped taking into consideration which one of them may better represent their physical correspondence. Another aspect taken into consideration is if the sounds can be identified even if played together: in this case the sharp attacks of the particles is always clearly distinguishable from the other continuous sounds; while these other two are distinguishable from two different aspects; they take different registers of the audible spectrum, and also the harmonic nature of the magnetic field rendition stands out from the noisy sound generated by the solar wind plasma.

Project: Sounds from the Sun

107

Techical aspects

The data is published in form of plain text on the http://services.swpc.noaa.gov/text/ web directory, in three different text files according to the instrument that generates the data, and is updated every minute for solar wind plasma and magnetic field, and every five for the electron and protons flux. Data is retrieved from those files within thirty seconds from a Processing script, that translates the plain text in scientific notation into the regular positional notation, labels and then sends them via OSC to a Max/ MSP patch. This passage is required because Max/ MSP cannot directly use values in scientific notation written as strings in plain text, while Processing can translate them in a very efficient way. In Max the values are routed to the proper sound generator, that adapts the data to the appropriate ranges of values, and then transforms them into parameters for the sound synthesis engines.

Source data in raw format, as presented on the SWPC web directory

108

New Media for Scientific Data Visualization

Data on SWPC website

Wait 30 seconds

Download on local computer

Software Flow Chart

Conversion to positional notation Processing

Labeling

OSC send

Values to sound parameters mapping

Max/MSP

Sound synthesis

Spatialization

Output

Project: Sounds from the Sun

109

Testing Process

110

The described system has been tested for some days, both from myself to refine the initial calibration, and from ten non-trained users with only a general knowledge of the involved phenomena, to verify the capability of the system to effectively translate into distinguishable and perceivably different sounds the original data. The system that proved most quickly its effectiveness was the “particle synthesis” applied to the proton and electron stream: the only adjustment required was to double all to frequencies generated, to create a clearer difference between the low-frequency sounds associated to the low-energy particles and the highest pitched sounds related to the high-energy particles. For most of the time the stream is constituted mainly of some thousands of low-energy particles per second, with some dozens of high-energy particles. Occasionally this balance changes, and the solar wind composition shifts, with the most energetic particles increasing their density by ten or more times. After applying the frequency range adjustment, I requested to the users to say whenever they could notice this shift while I was checking the real-time data: the change was correctly noticed in almost every occasion, both if the subject was paying attention and if they were involved in casual chatting. The subtractive synthesis engine working on the solar wind plasma data required some more testing and refining: the speed and density parameters, mapped respectively to the center frequency of the filter and to its bandwidth proved to be quite effective and describing, with the changes being almost immediately noticed by the test subjects, even the less relevant ones. The last parameter, the temperature of the plasma mapped to the timbre of the noise, proved to be more difficult to render effectively, mostly because of the wide range of possible values, from 1’000° K to 1’000’000° K. Small changes, around 50% or less of the previous value, can be difficult to detect without

New Media for Scientific Data Visualization

some focus; anyway this parameter has often very noticeable and sudden changes, with the value doubling or increasing even more, often in less than a minute: these shifts became clearly noticeable with a careful calibrating of the related parameters, allowing to pin down most of the relevant changes in the phenomenon. The “harmonic synthesizer” connected to the magnetic field parameters proved to be the most difficult to map, and various different approaches were tried before getting to a satisfying one. In the definitive system the total magnitude of the magnetic field is considered to be the most relevant, and thus is connected to the fundamental frequency of the sound generated, which is the most clearly distinguishable parameter. The three components, x, y and z, of the magnetic field are mapped to the amplitude of harmonic sounds multiple of the fundamental: in this way the overall character of the sound changes with the “balance” of the magnetic field, exposing the dominant component. In this way sudden changes both in the magnitude and in the rotation of the magnetic field became clearly audible.

Sonograms of the three sound synthesis systems, top to bottom: particles, solar wind plasma, and magnetic field

Project: Sounds from the Sun

111

References Conclusions Thanks Colophon

Bibliography RONALD BAECKER, Sorting Out Sorting: A Case Study of Software Visualization for Teaching Computer Science, “Software Visualization: Programming as a Multimedia Experience”, MIT Press,1998,pp. 369 - 381 LEE BYRON, MARTIN WATTENBERG, Stacked Graphs – Geometry & Aesthetics, “IEEE Transactions on Visualization and Computer Graphics” ,Vol 14 Issue 6,2008,pp. 1245 - 1252 ROBERT M. CANDEY, ANTON M. SCHERTENLEIB, WANDA L. DIAZ MERCED, Xsonify Sonification Tool For Space Physics, 12th International Conference on Auditory Display, London, UK, June 20-23, 2006 CHARLOTTE HESS, ELINOR OSTROM,IDEAS, ARTIFACTS, AND FACILITIES: Information As A Common-Pool Resource, “Law And Contemporary Problems”, vol 66, Winter/Spring 2003, pp. 111 - 145 DANIEL F. KEEFE, DAVID B. KARELITZ, EILEEN L. VOTE, DAVID H. LAIDLAW, Artistic Collaboration in Designing VR Visualizations, “IEEE Computer Graphics and Applications”, March/April 2005, pp. 18 - 23 DANIEL F. KEEFE, DANIEL ACEVEDO, JADRIAN MILES, FRITZ DRURY, SHARON M. SWARTZ, DAVID H. LAIDLAW, Scientific Sketching for Collaborative VR Visualization Design, “IEEE Transactions On Visualization And Computer Graphics”, Vol 16 n. 4, July / August 2008, pp. 1 - 13 ANDREA LAU, ANDREW VANDE MOERE, Towards a Model of Information Aesthetics in Information Visualization, Information Visualization IV, 2007 J. A. LEWIS, E. E. ZAJAC, A Two-Gyro, Gravity-Gradient Satellite Attitude Control System, “The Bell System Technical Journal”, November 1964, pp. 2705 - 2765 LIPYEOW LIM, HAIXUN WANG, MIN WANG, Semantic Queries by Example, 2003 DAVID CHEK LING NGO, LIAN SENG TEO, JOHN G. BYRNE, Modelling interface aesthetics, “Information Sciences”, 152, 2003, pp. 25 - 46 BRUCE H. MCCORMICJ, THOMAS A. DEFANTI, MAXINE D. BROWN, Visualization in Scientific Computing, “Computer Graphics”, ACM SIGGRAPH, July 1987

114

ROBERT K. MERTON, The Normative Structure of Science, “The Sociology of Science: Theoretical and Empirical Investigations”, University of Chicago Press, 1973 A. MICHAEL NOLL, Computer-Generated Three-dimensional Movies, “Computers and Automation”, Bell Telephone Labs., November 1965, pp. 20 - 23 A. MICHAEL NOLL, Computers and the Visual Arts, “Design and Planning 2: Computers in Design and Communication”, Hastings House, 1967, pp. 65 - 79 THERESA-MARIE RHYNE, Information and Scientific Visualization: Separate but Equal or Happy Together at Last, “IEEE Visualization”, 2003 WARREN SACK, Aesthetics of Information Visualization, “Context Providers: Conditions of Meaning in Media Arts”, Intellect Ltd, 2011, pp. 123 - 144 ANDREW VANDE MOERE, HELEN PURCHASE, On the role of design in information visualization , “Information Visualization”, 2011, pp. 356 - 371 FERNANDA B. VIÉGAS AND MARTIN WATTENBERG, Artistic Data Visualization: Beyond Visual Analytics, “Online Communities and Social Computing”, 2007, pp 182-191 E. E. ZAJAC, Computer-Made perspective Movies as a Scientific and Communication Tool, “Communications of the ACM”, Vol 7, Num 3, 1964, pp. 169 / 170

115

Sitography AT&T Archives, “Simulation of a Two-Gyro Gravity-Gradient Attitude Control System” http://webcache.googleusercontent.com/search?q=cache:wlCuGlowlpgJ:techchannel.att. com/play-video.cfm/2012/7/18/AT%26T-Archives-First-Computer-Generated-GraphicsFilm+&cd=1&hl=en&ct=clnk&gl=it Ronald M. Baecker, “Sorting Out Sorting” Video - https://www.youtube.com/ watch?v=SJwEwA5gOkM BBC Data Art, “A Quick Illustrated History of Visualisation” - http://data-art.net/resources/ history_of_vis.php Tim Berners-Lee, “The next Web of open, linked data” - http://www.dailymotion.com/ video/x8omty_tim-berners-lee-the-next-web-of-ope_tech#.UVWqxhnGrJw Tim Berners-Lee, “Linked Data” - https://www.w3.org/DesignIssues/LinkedData.html Jorge Camoes, “Infographics vs. Data Visualization” - http://www.excelcharts.com/blog/ infographics-data-visualization/ Wayne Carlson, “A Critical History of Computer Graphics and Animation, Section 18: Scientific Visualization” - https://design.osu.edu/carlson/history/lesson18.html Simon Chignard, “A brief history of Open Data” - http://www.paristechreview. com/2013/03/29/brief-history-open-data/ Centre des Arts, “Invisible et Insaisissable” - http://www.cda95.fr/content/invisible-etinsaisissable Michael Friendly, “Milestones in the History of Scientific Visualization” - http://www. datavis.ca/papers/vita/Friendly08aaas.html MARIEKE GUY, “OPEN ACCESS TO RESEARCH DATA – TIMELINE” - http://access.okfn. org/2015/04/30/open-access-to-research-data-timeline/ Jeremy Norman, “Edward Zajac Produces the First Computer-Animated Film (1963)” http://www.historyofinformation.com/expanded.php?id=1002 Barack Obama, “Transparency and Open Government” Memorandum - https://www. whitehouse.gov/the_press_office/TransparencyandOpenGovernment

116

Alex Harrison Parkers’s website - http://www.alexharrisonparker.com Lisa Charlotte Rost, “The Line between Data Vis and Data Art” - http://lisacharlotterost. github.io/2015/12/14/The-Line-between-Data-Vis-And-Data-Art/ Moritz Stefaner, “Little Boxes” - http://well-formed-data.net/archives/1210/little-boxes Edward Tufte, “Scientific visualization: evidence and/or fireworks” - http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0002T8 Edward Zajac, “Simulation of a two-giro gravity attitude control system” video - https:// www.youtube.com/watch?v=m8Rbl7JG4Ng

117

Conclusions

The objective of this thesis is to present some possibilities for the improvement of a field, scientific visualization, that has seen a steady increase in its importance since its beginning. First of all, it has been necessary to make clear, both for me and for the possible readers, how scientific visualization can be defined, explaining why and how it is different from the most generic information visualization projects that we are more and more used to see almost every day, ranging from user-generated statistics to popular visualizations spreading through Internet about any possible field of knowledge or interest. The fact that it requires a deep understanding of the underlying scientific phenomena pushed it in a niche, effectively separating it from the more generic fields of visual arts and design that nowadays are pushing experimentation in information design, especially thanks to the technological innovations of the last decades. These innovations, for example virtual reality, nowadays are almost confined in the field of entertainment, while they have a great underlying potential for visualization and design for a great number of different topics. In fact, the interest for contemporary science topics is well alive, and the projects created by artists and designers in the last years, thanks also to the Open Data revolution, demonstrate this, together with the some of the possibilities that new forms of media could have to investigate even the most complex concepts related to science, medicine, physics, and so on. There are nowadays some very prolific examples of cooperation between scientists, artists, programmers and designers, that created some new and novel ways to approach visualization, and more generally rendition, of various data and phenomena, but they are still considered “extraordinary” and “experimental” examples, while my research suggests that that kind of approach should become more and more

common in the next years, involving artists and designers not only in the finishing phases of visualization projects but also in the concept and development parts: this could bring great positive effects both for researchers, that could have better instruments, involving new technologies and approaches to investigate their fields; and also for the general public, that could have access, for example, to pieces of public art with an informative value, supported by strong scientific principles. The project I developed demonstrates how a new and different approach, that revolves around the conscious use of sound, can be an effective and useful way to to represent real-time data, even in a mission critical environment. To conclude, it has been a difficult but satisfying work, that required a broad research and the conceptual effort to put together many different fields, that I hope will at least contribute to open the discussion on how to reconsider the respective roles of science, art and design, how they could interact for us and how they could positively effect our lives in the future.

119

120

First of all I have to thank my family, that made this 2 years experience in Venice even possible, supporting me in all the ways possible for them. A huge thank you goes to the colleagues I met here in Venice, the ones that became some of my best friends and really my other family, making this amazing city not only the place where I studied, but also my home and the place I loved the most to live in. Your caring support made this experience a wonderful one, that made me grow way more that you all can imagine. A special thank goes to my relator, Marco Ferrari, that really guided me in the right way to face this thesis project, starting from my initial vague and unclear concepts about the themes I wanted to investigate. I realize that my previous formation and my way of working may not entirely fit in this field of design, but I guarantee that I absorbed the most that I could from each project I realized and each course I took here, and for that my thank goes to Marco and all the other professors.

Thanks

121

Colophon New Media for Scientific Data Visualization IUAV University of Venice Design and Planning in Complex Environments Department Master Degree in Product and Visual Communication Design AA 2014 / 2015 Supervisor: Marco Ferrari

Typefaces: Aleo, Alessio Lasio Bebas Neue, Ryoichi Tsunekawa Software: Adobe InDesign CC Adobe Illustrator CC Adobe Photoshop CC Processing 3.0 Nodebox 3.0 Cycling â&#x20AC;&#x2DC;74 Max 7

122