EXTREME MEASURES SCIENTISTS HAVE CREATED AN
EXCITING AND EXTREME NEW LEVEL OF VIDEO GAMING
CONTENTS EDITOR Welcome to CONNECT, the science and technology magazine for the inquisitive mind. Every month CONNECT covers all that's new and exciting, packed with gripping features and beautiful illustrations. Our main feature always covers an interesting, sometimes ground breaking issue. This month CONNECT discusses Bio-feedback (which is a scientific way of monitoring human behaviour.) Recently, an extreme new level of video gaming has been created whereby a chip is placed into the human body, monitoring their physiological funtions and thereby computing useful data from it! - What would you do for the ultimate gaming experience?
02 COVER FEATURE
EXTREME MEASURES By using Bio-Feedback, a method used to monitor human behaviour, an extreme new level of video gaming has been created.
3
08 FEATURE ARTICLE
MAD SCIENCE In Tim Burton’s charming and chilling stop-motion movie Frankenweenie, a young boy called Victor brings his beloved dog back from the dead. It is, But in a series of amazing experiments, the doctors at the Safar Center for Resuscitation Research at the University of Pittsburgh have created their own real-life Frankenweenies.
COVER FEATURE
LOADING
EXTREME MEASURES
BY USING BIO-FEEDBACK; A SCIENTIFIC METHOD USED TO RESEARCH HUMAN BEHAVIOUR - AN EXTREME NEW LEVEL OF VIDEO GAMING HAS BEEN CREATED.
2 | CONNECT
Last year, officials from the Food and Drug Administration (FDA) and the Defense Advanced Research Projects Agency (DARPA) met in Arlington, Va. to discuss the future of biological feedback. The sort of machines they were talking about—machines that could quantify physiological functions (blood pressure, skin temperature, muscle tension) and compute useful data from it—were first introduced in the late 1960s by the mathematician Norbert Wiener at MIT. Soon the machines were in universities’ labs and hospitals. By 1980, the Biofeedback Certification Institute of America oversaw the best practices in the burgeoning field; the book Biofeedback: Clinical Applications in Behavioral Medicine came out the same year. By the 1990s, nearly every hospital in America had means for electronically measuring a patient’s response to external stimuli, and so did many doctors’ offices.
Biofeedback has been around for nearly 50 years, but it wasn’t until the last decade, in particular the past few years, that the machines used to measure physiology became mass market. In 2006 Apple and Nike teamed up to release the Nike+ platform, a combination shoe-embedded sensor and iPod adapter that tracks your run and delivers the report to Nike’s website or your smartphone. When DARPA’s scientists and engineers met with the FDA, they were thinking beyond shoe sensors and apps. The meeting last year was about putting biosensors inside people. The scientists and engineers at DARPA are asked to think about the future, imagine what might be possible, and work to create it. Around the same time Wiener coined the term biofeedback, DARPA helped create ARPANET, a computer network that grew to become the global internet. In the last year it helped invent a very small robot that looks and flies just like a hummingbird. At the meeting with the FDA last year, Daniel J. Wattendorf, a DARPA program manager, said that his goal was to measure biomarkers “on- person” in real time. A continuous monitoring device of the sort Wattendorf went on to describe—able to last a lifetime with little to no effect on its wearer—is not so far out. In fact, speeding up both the development and approval of such a device is the FDA’s role, and why the administration was there. When such a device is developed, the FDA will have to gauge the efficacy of implanting sensors with no immediate curative properties in people. To work well, the sensors have to function for a very long time, which raises some interesting questions: What if someone wants to opt
"Designing our emotional reactions IS no longer an art,but a quantifiable science."
out soon after a biofeedback device is implanted or ingested or tattooed? Can someone decide exactly what they want monitored, a la carte? Who will host that data? And even more basic, what if a person’s body doesn’t react well to having a chip implanted in it? Despite these hurdles, there is currently a very large portion of the population willingly participating in an ongoing experiment that isn’t all that far removed from the kind of thing DARPA is working toward. They do not have chips implanted in them, but they do have a computer or smartphone—some even have wristbands and shoes that talk to one another. All are gaming. Outside Seattle, across Lake Washington, an experimental psychologist named Mike Ambinder is measuring the heart rates, skin conductance, facial ticks, and eye movements of videogame players to deepen his understanding of what happens when we play—using biofeedback to inform game design. “Measuring sentiment,” he calls it. Ambinder works for Valve Software, the company that created the zombie shooter Left 4 Dead. Using heart rate and sweat monitors, he traces a Left 4 Dead player’s stress levels. Left 4 Dead is dark and moody; and the thing that stands out about the game is how unsettling it is, how it’s designed to keep players off-balance. The zombie horde comes in waves, and these waves are diabolically unpredictable, even after playing the same level over and over. The unevenness makes the calm in-between moments just as stressful as the times dozens of undead are ripping at your face. Ambinder is interested in measuring how we feel—both during the face-ripping moments and those in between. The ultimate goal is to allow the zombie horde to respond to our stress and, at the right moment, up it. In-game response to player stress is pretty basic, says Ambinder. “It’s actually fairly trivial to create an accurate device that measures SCL, or skin conductance level—a correlate of physiological arousal,” he wrote to me. “In theory, a mass-market controller incorporating this technology could be done cheaply and with minimal disruption to existing form-factors ... measuring facial
CONNECT | 3
expressions with webcams [is another] example.” In other words, what Ambinder is measuring in his lab could be repeated in a living room. Tools for biofeedback are improving so quickly, Ambinder explained, that in a decade or so we’ll have come from knowing the difference between binary emotions (happy and sad) to real nuance—the difference between boredom, frustration, and bliss. He described how “particular emotions can become part of the gameplay—imagine a lie-detection game where a player needs to maintain a calm level of arousal and neutral facial expression to advance, or a competitive multiplayer game where your score depends in part (or in whole) on the level of arousal you’re able to incite in your opponents.” In one of his tests, Ambinder found that the simple SCL biofeedback systems he had rigged were especially popular in multiplayer competition. This isn’t surprising. The fun in sports often comes from knowing you are frustrating your opponent, and what better way to know that someone is truly frustrated than to see data tracking their physiological reactions? We already infer this—we read people’s expressions on the court or the field—but soon, we’ll be able to quantify their frustration. We’ll be able to see our opponents, wherever they may be, literally sweating it out. For game designers, the opportunity to measure a player’s pulse or smile presents something of a Pandora’s Box. Real-time biofeedback is something game designers have never had. Once the game is out in the world, it’s being tested and tweaked, adapting to the bodies of its players in real time. Designing our emotional reactions to a game is no longer simply an art, but a quantifiable science.
About 15 years ago, Rosalind Picard began to explore what she calls affective computing, which, she says, is a way to “teach computers to recognize emotion.” Picard is director of both the Affective Computing Research Group and Autism & Communication Technology Initiative at the MIT Media Lab. A professor of Media Arts and Sciences at MIT, she’s also the cofounder and chief scientist at a company called Affectiva. One sensor Affectiva makes picks up electrodermal activity—small changes in conductance sensed from the surface of your skin that can tell a machine that you are about to sweat before you do. Affectiva also sells facial recognition software developed by Picard that identifies 25 points on a user’s face and tracks those movements, identifying the likelihood that when two corners of a user’s mouth rise up and his eyes crease, he is smiling and probably happy. The Affectiva software works with a standard webcam. One of Affectiva’s clients is using its facial recognition system to track the emotions of online shoppers as they move through the decision process. Another—a university—is using the software as a noninvasive method of testing how autistic children respond to external stimuli. Affectiva’s literature describes how the company’s software might be used in videogames: to “measure gamers’ actual emotional response to your game over the web” and “gain a scene-byscene understanding of areas that are most engaging or not.” There’s one gaming company that’s taken a particular interest in Affectiva’s research: Zynga. The pairing makes sense. Gamers, particularly in a Zynga game like FarmVille, are consumers, and Affectiva’s feedback systems have proven successful for companies like Boston Market and a market research group called Shopper
"A CHIP IS PLACED INTO THE HUMAN BODY, MONITORING BEHAVIOUR THROUGH THEIR PHYSIOLOGICAL FUNCTIONS AND THEREBY COMPUTING EXTREMELY USEFUL DATA FROM IT."
4 | CONNECT
CO
Science. Zynga, which claims that some 60 million people play its games every day, supposedly employs more economists and analysts than game designers. Theirs is a calculated art, and their profit structure more closely aligns with a casino than a game company. Most of Zynga’s earnings come from a small number of players willing to spend a lot of money (in one instance $75,000 in a year on a single game, Bloomberg Businessweek reported) on its virtual items and special features. After OMGPOP’s game Draw Something became so popular Zynga bought the studio (for $180 million), one of OMGPOP’s engineers, Shay Pierce, quit. Pierce went on to announce why he was quitting on the games industry- focused site Gamasutra: “An evil game company isn’t really interested in making games, it’s too busy playing a game—a game with the stock market, usually. It views players as weak-minded cash cows and it views its developers as expendable, replaceable tools to create the machines that milk those cows.” Even so, by sheer profit Zynga seems to be winning.
“There’s no reason to think that entertainment consumers would be any different than any other consumer,” says Dmitri Williams, an associate professor at the University of Southern California’s Annenberg School for Communication. His research focuses on online games and their social and economic impact. “There are
T ON INUE?
"Soon, we’ll be able to quantify our opponents' frustration. We’ll be able to see them literally sweating it out."
"what better way to know that someone is truly frustrated than to see data tracking their physiological reactions?"
"There is also the ultimate question, the double-edged sword of it all: the data itself. Who owns it? What might it be used for?"
6 | CONNECT
two goals,” he says of his work. “One is to understand what’s going on inside of these worlds. The second is to figure out if we can use these spaces as petri dishes for social sciences.” One of Williams’s former students, Dr. Rabindra Ratan, is using biofeedback to learn about our emotional response to our avatars in games. The studies can be as basic as watching for an uptick in skin conductivity while one’s avatar is under attack. More sweat means a deeper connection. After Ratan’s participants filled out a questionnaire about their experience, he had them watch a video of their character being beaten up. They still had a heart monitor on. Time after time, and even though they weren’t playing, their pace quickened. This empathy toward our avatars, even after the fact, has enormous implications for how games are built. But it also points to how we might use games for things we are only beginning to imagine. Williams told me about how videogames and virtual worlds are now being used as a form of therapy. He gives, as an example, groups of veterans using virtual- reality goggles and Second Life to
help treat post-traumatic stress disorder (though this is coupled with traditional, realworld therapy sessions). Online, a therapist can slowly dial up stimuli, which helps vets re-sensitize in a safer environment. Once biofeedback systems exist on controllers and screens, once everyone else outside of labs is allowed to tap this flood of extremely personal data, “You’d begin to see it used in all sorts of creative ways,” Williams says. Not long ago I walked across Los Angeles, from LAX to downtown to Los Feliz, then back west on Sunset until I again reached theacific. When Rockstar released Team Bondi’s game L.A. Noire—with its sprawling and cinematic vision of late-1940s Los Angeles— and before I worked the Black Dahlia case and drove and drove and drove through the city, I walked. It’s not the best way through the game, but moving across the landscape any faster misses the immense amount of work that went into rendering the city. The game’s central action—finding clues at crime scenes and interrogating everyone in your path—is far less gripping. Yet it was impossible not to consider the possibilities a biofeedback
"Once biofeedback systems exist on controllers and screens, this flood of extremely personal data, WILL BE used in all sorts of creative ways."
system running through the game might allow. The facial capture Team Bondi used on actors to build lifelike models in the game could be reversed, though crudely, using Picard’s facial recognition software and a Kinect. I imagined—during moments that were meant to be dramatic, that were supposed to command my close attention on the couch, sitting up at attention, my brow slightly furrowed in a look of concern, or at least interest. What if SLC sensors on the controller read that my palms remained dry, and gave me a slight edge for remaining calm during questioning or a car chase? What if, as I started sweating more, steering became more difficult, or the questions I could ask in the interview less cool-headed? What if, when I stood up during an interrogation, my character did too? And what if that made my threats all the more threatening? Imagine that there was no controller, no chip under my skin, yet the game still knew my position on the couch, my mood, how much my hands were sweating, and my heart rate—all things that are, or soon will be, entirely withinthe realm of possibility. I still might prefer walking through the landscape of postwar Los Angeles to the investigations and interrogations. But I might not. One possible approach to seeing a heart beat could be through blood, which is remarkably good at absorbing light, even through skin. Our skin’s reflectivity changes slightly, concurrent with our heartbeat, so it’s possible to register a pulse not with sound or feeling but by sight.
But Ming-Zher Poh, a graduate student at MIT, invented a mirror that can. A fine-tuned instrument such as this camera, capable of measuring slight variations in light reflectivity, is surprisingly cheap. Poh used a simple, off-the-shelf webcam and altered the software so that it filters all ambient light and measures only the ebbs and flows from the face of the person staring into it. “Mirror mirror,” the user might say, and the camera behind the reflective glass captures the light bouncing off her face and turns it into ones and zeros. An algorithm Poh created translates the ones and zeros into a number of heartbeats per minute. A camera-mirror like Poh’s is a marvelous tool, but for game designers it could possibly destroy what they (and we) love about games. Williams put it most succinctly: “The great irony of feedback is that if you rely on it too much you get in the way of the art.” There is also the ultimate question, the double-edged sword of it all: the data itself. Who owns it? What might it be used for? Immediately following her TED talk this year, Regina Dugan, DARPA’s outgoing director, had a brief Q&A with TED curator Chris Anderson. Near the end of her talk Dugan had a DARPA engineer fly the remotecontrolled hummingbird and the crowd gasped and applauded. The excitement and positivite feedback from people made sure that this project had to carry on to the end. A few questions are still being raised about the “scramjet,” a Mach 20 glider that was illustrative of the sort of audacious dreaming and repeated failure necessary for the breakthroughs that make DARPA famous.
CONNECT | 7
FEATURE ARTICLE
MA D
S CIENC E DIRECTOR TIM BURTON USES STOP-MOTION TO BRING A DOG BACK FROM THE DEAD. IN A US LAB IT’S BEEN DONE FOR REAL...
In Tim Burton’s charming and chilling stop-motion movie Frankenweenie, a young boy called Victor brings his beloved dog back from the dead. It is, Burton says, his most personal movie to date – Frankenweenie is based on his own pet that died when he was a creative and lonely young boy, but it is also universal. Weeping over a pet or person who has been dear to us, we’ve all wished we could reverse the cruel hand of the Grim Reaper. But in a series of amazing experiments, the doctors at the Safar Center for Resuscitation Research at the University of Pittsburgh have created their own Frankenweenies.
8 | CONNECT
They resurrected dogs that had been dead for hours by zapping them with electricity, just like in the classic Frankenstein movies. And it’s not stopping with prized pets; they’re going for humans next. Insisting the work is practical, not ghoulish, the centre’s associate director and head of resurrection experiments, Dr Samuel Tisherman, is aiming to use this research to save humans, by buying time for surgeons to treat trauma victims who would otherwise (irreversibly) die. “Trauma patients that have suffered cardiac arrest almost never survive,” says Tisherman. “So we’re trying to improve upon that. One of the biggest problems as
a trauma surgeon is not having enough time to take control of the bleeding and the injuries before the patient has spent too long a period with no bloodflow to the heart and brain. So what we’ve been working on is a way to basically buy time for the surgeons to deal with the injuries and to control the bleeding, then to fully resuscitate the person in a delayed fashion.” The experimental procedures should begin on humans by the end of this year, but the results in dogs have already been astounding. In 2005, the Safar Center team plunged several dogs into total clinical death – no heartbeat, no brain activity – before bringing them back ito life.
CONNECT | 9
"WE ARE LEARNING to understand the line which seperates life and death.." 10 | CONNECT
it' s
a
The apparently miraculous result was achieved by flushing the dogs’ bodies of blood and replacing it with cold saline solution laced with oxygen and glucose. After the dogs had been dead for three hours – or as the scientists would prefer we saw it, in a state of “suspended animation” – they reversed their steps, removing the saline solution, replacing the blood and giving the heart a little electric jolt to restart it. The procedure will be essentially the same for humans. If a victim of a stabbing or a gunshot wound is treated while in cardiac arrest and the usual resuscitation techniques are not working, the team will then switch to this groundbreaking method, which has been dubbed Emergency Preservation and Resuscitation (EPR). Tisherman says the resulting window of time, when the brain is inactive but protected by an artificially induced state of hypothermia, this could give surgeons enough time to fix the patient. “In general, if you become hypothermic, it’s because you go outside on a cold winter day and don’t come back in,” says Tisherman. “Trauma patients tend to cool down because they’ve been in shock or that sort of thing. In all of those situations hypothermia may be bad. But here, in what we called controlled or therapeutic hypothermia, it can be more beneficial. It can fulfil the requirements of the cells and tissues until we’re able to restore enough blood to the body." “It started with the observation that there are people that drown in cold water and somehow survive even if they’re under water for an hour; the reason being that when that happens the brain cools fast enough that by the time the heart stops the brain is protected.”
After surgeons repair the patient’s injuries, they will then be resurrected using the full heart and lung machine. “The goal is not just to have the patient live but to have them live and go back to having a normal kind of life,” says Tisherman. The Safar Center team’s research is the most dramatic of a series of recent experiments that chip away at the boundary between life and death. In 2010, the pioneering US geneticist Craig Venter created the world’s first synthetic life form, paving the way for ‘designer organisms’ that are built rather than evolving. The first aims are to create bacteria that churn out biofuels, soak up carbon dioxide from the atmosphere and even manufacture vaccines. In 2005, working in the Centre for Stem Cell Biology at Newcastle University, Professor Miodrag Stojkovic turned dead cells into living tissue. The cells were taken from an arrested human embryo and were used to make stem cells. Now running an infertility clinic in Serbia, Stojkovic hopes the advance will remove some of the ethical objections to stem-cell treatment, since experiments would no longer have to use living embryos. Embryonic stem cells are prized by scientists because they are capable of turning into any tissue type in the body and could provide the key to treating diseases such as, diabetes, Alzheimer’s, Parkinson’s and heart disease. This month, at Kyoto University, stem cells made from skin became ‘grandparents’ after scientists created generations of life in experiments. A Japanese team created
! e v I I i al CONNECT | 11
"our job is to help SERIOUSLY ill people. ESSENTIALLY WE ARE FIGHTING DEATH."
12 | CONNECT
after scientists created generations of life in experiments. A Japanese team created eggs from the stem cells, which were then fertilised to make baby mice. These mice then went on to have their own children, whose ‘grandmother’ was a cell in a laboratory dish. Dr Katsuhiko Hayashi said the ultimate aim was to help infertile couples to have children. As outlandish as their work may sound, few researchers are happy when it is compared to horror movies. Packed with mad scientists and their ill-advised creations, the classic Universal monster movies of the 1930s, ’40s and ’50s kick-started the horror genre and were precision engineered to fill our heads with fear of science. There was the distinct impression, not unrelated to the spectre of chlorine gas at Ypres and the thousands vaporised in Hiroshima and Nagasaki, that man should not tinker with nature and that Bad Things would come of it. Frankenweenie is bursting with references to these enduring chillers – there are nods to Frankenstein and Bride of
Frankenstein, The Mummy, The Wolf Man, Dracula and more – but represents a more positive modern take on science. Here, science is only dangerous when the power is in the wrong hands. When the young hero Victor brings his dog Sparky back from the dead, it’s more Lassie Come Home than Pet cematary. “For me, art and science are similar,” explains Tim Burton. “Science can be used for good or evil. The same with art, to a lesser degree. For me, science was a symbol of thinking about things creatively and passionately and outside the norm.” If this was how all movies saw science, the researchers might be more laidback about the references, but for most the fear of being seen as a mad scientist is still strong. When the Safar Center announced their results in dogs, the internet responded with a predictable howl of warning about the ‘zombie dogs’, along with a screed of references to Frankenstein and Pet Sematary. “We’re trying our best to get away from any of those kinds of connotations,” says Tisherman with a sigh.
“What we’re doing is scientific. It’s real and hopefully we will be able to fix people.” Stojkovic agrees: “This is not a scary movie, or Frankenstein or The Island. The fact is that there are too many diseases where we have no answer, but now there is something in our hands; not to cure all diseases and disabilities, but some, for sure.” Of course, we should remember that humanity has always chipped away at the kingdom of death, and each new breakthrough has engendered fear before it proved its worth. Not long ago, transplants were the stuff of science fiction, now they save lives every day. The same with bionic arms or test-tube babies - just imagine what the future holds. “People are trying to understand the line which separates life and death and, for sure, we are learning,” says Stojkovic. “I believe that our job is to help seriously ill people to improve their quality of life and to fight debilitating diseases. If we will be able to achieve this by serious research, essentially we are fighting death.”
"science is only dangerous when the power is in the wrong hands."