by David HandDark Data

RECENTLY PUBLISHED

DARK DATA Why What You Don’t Know Matters DAVID J. HAND

David Hand shines a bright light onto the dark corners of statistics. This is a learned book but a witty, readable and important one. I learned a lot and so will you – Tim Harford, author of FIFTY INVENTIONS THAT SHAPED THE MODERN ECONOMY and presenter of the BBC radio series ‘More or Less’

It is hard to think of anyone having anything at all to do with data-driven decisions who couldn’t benefit from reading this book. David Hand effortlessly guides the reader through the many pitfalls of dark data – Arno Siebes, Universiteit Utrecht

This unique and much-needed book provides an accessible guide to dark data at a time when general awareness of the phenomenon is declining – Geert Molenberghs, Universiteit Hasselt and KU Leuven

A much-need counterpoint to the big-data hype of recent years – and a clarion call for us all to be constantly on the alert to unknown unknowns as well as the known unknowns.

A good cartoon captures the important features of a face or behaviour, but there is no guarantee of this. It can easily miss much that matters. Indeed, it can easily miss the most important things.

Big data is like a cartoon simplification. Although it’s meant to represent and describe the world, its abundance can mislead people into thinking they know everything. In DARK DATA, the eminent statistician David Hand explores the implications of what we might be missing. He shows, through many real examples, just how serious things can get – how missing data can lead to death and disaster, failed economies and societies, and ruined lives.

Hand lays bare the ubiquity of dark data, what causes it and where it is likely to manifest itself. It can arise for many reasons, which themselves may not be obvious – asymmetric information in wars, time delays in financial trading, dropouts in clinical trials and deliberate selection to enhance apparent performance in hospitals, policing and schools. What is clear is that measuring and collecting more and more data are not guaranteed to lead to more relevant information or to better understanding.

But there’s also a more positive side to dark data. When approached from the right angle, it can lead to insights that cannot be obtained any other way. Counterintuitive though it might seem, deliberately obscuring some of the data can lead to improved predictions and better understanding – providing, of course, the right data are obscured in the right way.

The modern world of big data holds huge potential for improving the human condition as well as for misleading us. DARK DATA shows how to achieve the first while avoiding the second.

DAVID J. HAND is a senior research investigator and emeritus professor of mathematics at Imperial College, London. He is also chief scientific advisor to Winton Capital Management. He is a fellow of the British Academy and an honorary fellow of the Institute of Actuaries, and has served (twice) as president of the Royal Statistical Society. He is a non-executive director of the UK Statistics Authority, and is chair of the board of the UK Administrative Data Research Network. He has published 300 scientific papers and 28 books, including THE IMPROBABILITY PRINCIPLE (Scientific American/FSG, 2014). In 2013 he was made OBE for services to research and innovation.

Agent: Peter Tallack

Publisher: Princeton University Press Publication: 18 February 2020 Length: 344 pages

All rights available excluding World English Language (Princeton University Press), China (CITIC), Italy (Rizzoli), Japan (Kawadeshobo-Shinsha), Korea (Gilbut), Russia (Alpina), Taiwan (Locus)