Blind Driver Challenge

Amazing race: blind, but now able to drive

By Danielle Lucey

Mark Riccobono becomes the first driver to loop the track at Daytona International Speedway. Photo courtesy NFB.

F

or the first time in history, blind driver is not an insult.”

Fresh from the driver’s seat, face still flush, Mark Riccobono’s first words to the press and attendees from the National Federation of the Blind brought laughs and a room full of applause.

Riccobono had just turned what some consider a joke into reality, becoming the first blind driver to ever go around the racetrack at Daytona International Speedway.

A precursor to the Rolex 24 — a 24-hour Grand-Am feat of driver endurance — the

Blind Driver Challenge was the brainchild of the NFB, whose members have spent more than a decade waiting for their goal to be achieved.

You don’t need to see to appreciate racing at Daytona. Huge wafts of gasoline and burning rubber hang in the air each time a car blasts by grandstands, which hold the nearly 400 members of the NFB who turned out from all over the country for the event. And then there’s the noise: When the speed of the cars out for the Rolex 24 race hit that wall in the crowd’s ears, Doppler shifting from an impending roar to a passing whiz, the speed and pure mus-

cle of what’s in front of them is evident.

For the NFB, the January event was about proving what they’ve always known: You don’t need to see to appreciate driving.

What members did need was a little help and ingenuity from some friends in the robotics community to make that possible. The NFB issued a call to action to all American universities, and experts Virginia Tech and TORC Technologies answered. Using technology from the DARPA Urban Challenge, TORC Technologies took its ByWire XGV roboticized Ford Escape and coupled it with Virginia Mission Critical

•

Spring 2011

11

Blind Driver Challenge — continued from page 11

We thought about our independence and what we could do to make the world better, and we built the idea that we would bring together inspiration and technology.

Riccobono lost almost all of his sight by age 13. Photo courtesy NFB.

Tech’s driver feedback inventions to turn robotic autonomy into personal autonomy, for the first time enabling blind people to drive.

Mark Like most blind people, Riccobono never imagined that he would be able to drive.

Riccobono, 34, has been legally blind, meaning he only had 10 percent of normal vision, since he was 5 years old due to a congenital eye condition. He continued to lose his vision as he got older and was completely blind in his left eye by age 13. Though he is still partially sighted in his right eye, Riccobono says it’s only colors and shapes.

“There was no prospect of being a driver; there was no thought of being a driver. I put the idea out of my head,” Riccobono says of the year he turned 16, usually the time when kids reach for the keys. He relies on carpools to get to work and uses public transportation to get his four-yearold son Austin to daycare every morning.

The NFB got on Riccobono’s radar when he attended the University of Wisconsin Madison in the mid-1990s.

“I didn’t know any real tips or tricks or

alternative techniques that blind people use,” says Riccobono. Unable to read Braille, Riccobono says he was “really struggling” when he came across NFB’s Madison affiliate. Interest piqued, he attended the national convention that year in Anaheim, Calif.

“I found a whole network of people that were willing to challenge me to go beyond where I thought I could,” he says. Riccobono rose through the ranks, becoming the Wisconsin location’s president from 1998 to 2003 before accepting the position of executive director of research and training programs at NFB’s national headquarters in Baltimore, Md. NFB’s headquarters is fairly large, about 100 employees. Riccobono spends his days interacting with members and coworkers, and through the use of innovative technologies, his typical day is pretty standard.

He uses a standard PC with a screen-access software, which reads what’s on his desktop out loud to him, that lets him interact with email, the Internet and Microsoft Word. He navigates a page using keyboard commands instead of a mouse. He keeps print documents on his desk, some in Braille and some that his assistant reads to him. Through a software that NFB helped

develop through a spinoff company, called K-NFB Reader Mobile, Riccobono’s cell phone is outfitted with a camera that can read what it sees to him.

“Wherever I am, I can read using this technology,” he says. He can write hardcopy notes in Braille through a slate and stylus he carries with him.

For the last decade there have been rumblings inside the NFB of increasing the autonomy of blind people in outside-ofthe-box ways. Putting a blind driver behind the wheel of a car was first pitched in 2001 at an NFB national conference by the organization’s president Marc Maurer.

“Eleven years ago, we started talking about the blind driver,” said Maurer at the Daytona event to a crowd of NFB members, worked up to fever pitch by a man they clearly revere. “Building an automobile the blind could use, and they said we couldn’t do it. … When we started thinking about what blind people can’t do, how many of us have said the first thing is that we can’t drive? And we thought about our independence and what we could do to make the world better, and we built the idea that we would bring together inspiration and technology.

“And today, to all of those to said it was just a stunt, and to all of those that said we couldn’t, and to all of those who said that it would never happen, we say you just watch us move.”

It is under Maurer’s reign that the national Mission Critical

•

Spring 2011

13

Blind Driver Challenge — continued from page 13 headquarters broke ground on its new Jernigan Institute in 2004, a facility focused on advancing technology to increase the independence of the blind.

“It was just part of the visionary set of things that was being talked about,” says Riccobono. The foundation set its timeline for driving, among its other goals, in the 25-year time range, he says. But the opening of the institute in 2004 marked a switch from talk to action.

“That was the first time it was apparent that it was a little more serious than just a bold vision,” he says. In 2004, the NFB pursued universities to work on the challenge.

Getting set Michael Fleming has been working in the robotics industry since college. He got his

start as a student at Virginia Tech, where he got both his bachelor’s and master’s in mechanical engineering. He went on to work as a research associate for the college. Fleming was inspired to start his own company after realizing that much of the university’s technology wasn’t being used in real-world applications.

“I came to conclusion with some of my colleagues that we had all this great technology, but unfortunately the technology wasn’t going anywhere,” said Fleming in an interview with Virginia Tech. “When the graduate students would leave, they would leave a thesis, and that technology was not being ported to address critical needs that we see within society.”

In 2006, with the help of Blacksburg, Va.’s VT KnowledgeWorks — an entrepreneurial incubation program for the Virginia

Tech community — Fleming secured a spot for TORC in the Virginia Tech Corporate Research Center (CRC). The company is still located there five years later, having grown from four employees to nearly 30.

TORC focuses on empowering other engineers by offering a suite of modular and easy-to-use products that can be used to more rapidly customize, integrate, and deploy safe and reliable robotic systems. Its products are now used by leading academic, commercial and government organizations to shorten the development process, lower costs and mitigate development risks.

In the mid-2000s DARPA, the mad scientists of the Department of Defense’s array of agencies, began a series of high-profile unmanned ground vehicle competitions designed to build the bridge between re-

Virginia Tech students test out the roboticized Ford Escape before the Blind Driver Challenge.

AUVSI photo.

Mission Critical

•

Spring 2011

15

Blind Driver Challenge — continued from page 15 Wire modules, which allow for computer control and monitoring of the vehicle’s systems, the SafeStop wireless emergency stop system and PowerHub power management and distribution modules necessary to power the additional components, sensors, computers and technologies adding to the system.

Light detection and ranging (lidar) scanners on the vehicle detect objects by sending out thousands of light pulses. The return light establishes distance, and those thousands of light points determine the profile of an object or negative space.

The camera looks at those pixels from the lidar and populates them with color that helps classify the environment. The software then takes that image and tries to predict what those objects are doing in real time — it’s essentially the same way eyes and the brain work together.

As for the additional hardware, a combination of scanning laser rangefinders and machine vision cameras were added to provide real-time sensory input.

The SpeedStrip interface sent vibrations up and down Riccobono’s back, letting him know how much to accelerate or decelerate. AUVSI photo.

search and real-life, working autonomous systems.

In its last feat, the 2007 Urban Challenge, DARPA tasked innovators to come up with a way that a completely autonomous car could navigate a 60-mile urban area course. Stop lights, traffic, obstacles and all, the participants had a six-hour window to complete the course.

Eighty-nine teams from around the globe competed to win the glory and prize money of the Urban Challenge. Undergrad and graduate students from Virginia Tech joined up with TORC to form Team VictorTango for the competition. Averaging just about one mile per hour slower than the top team, VictorTango placed third.

16

Mission Critical

•

Spring 2011

The competition vehicle, a modified 2005 Ford Escape Hybrid, was the same platform used for the Blind Driver Challenge.

TORC helped create all the software that allowed the Escape to autonomously navigate.

“The [Blind Driver Challenge] vehicle has its roots in the DARPA Urban Challenge technology. The new challenge was adding a blind driver in the loop,” says Andrew Culhane, business development manager at TORC. “We were able to insert several TORC products on day one to get us 80 percent there — the remaining 20 percent was the tricky part, communicating with Virginia Tech’s nonvisual interfaces.”

The ByWire XGV is a packaged vehicle conversion that includes TORC’s By-

All of this information gets fused together with data from a GPS unit, which works with TORC’s autonomous navigation software and finally passes the information back to the driver through the Virginia Tech Robotics and Mechanisms Laboratory’s nonvisual interfaces.

But it was not the XGV alone that powered Riccobono’s car. He had to actually maneuver the vehicle himself.

Students at Virginia Tech masterminded the key pieces missing for this odd retrofit — turning an autonomous vehicle back into a manned car, but a manned car that still uses all the same software as a robotic vehicle to help make decisions.

The students took to task, working under Virginia Tech professor Dennis Hong.

They came up with two innovations, technologies that would give Riccobono feedback as he was driving the car. The technologies didn’t force him to perform a task, but told him what to do.

Through a senior design project, the students developed haptic sensors — devices that provide touch sensation — that would steer Riccobono in the right direction.

Four of the eight seniors sent to the Daytona event worked on SpeedStrip, a pad the driver sits on that gives him information on his acceleration. The interface uses vibrations up and down the driver’s legs and back to let him feel how fast to drive.

When that speed is neared, the vibration slows to a stop. The students started work on the device in 2006.

The second interface that involved the other four seniors was DriveGrip, a pair of fingerless gloves that send vibrations down the driver’s fingers, which was designed in the last year. A soft right turn would send a vibration to Riccobono’s right pointer finger, and a harder right turn would go across more of his right-hand fingers.

Like SpeedStrip, once the car is correctly oriented, the vibration stops. The gloves initially experienced wire fatigue when the students tested them, so they replaced them with wire cables, like the ones used on a computer.

…some actually do very, very well on their first try. It shows that Virginia Tech’s technology is intuitive… valuable to get a sense of the interfaces.

And what testing on the simulator lacked in real-world feel, it made up for in safety, he continues.

Multiple people tested on the simulators so the NFB could downselect to one driver for the Daytona race day.

“We’ve had varying success,” says Riccobono. “Some people have very low percentage [accuracy] on their first run, being in the lane and on target, but some actually do very, very well on their first try.” It shows that Virginia Tech’s technology is intuitive, says Riccobono, but that training on any new device is also necessary.

Locking up the deal to drive the real system at Daytona before the Rolex 24 Grand-Am

race was surprisingly easy, he says.

“The whole team, Grand-Am, they run the race, but the Daytona international Speedway team [is] also a tremendous team of folks,” he says. “Not one person we met said, ‘Well, what do you mean you’re going to bring a blind person out here?’

They bought into the vision. And it just shows that they have imagination, because we certainly didn’t expect it to be as friendly, open, warm, when we said, ‘Hey, we want a blind person to drive on your track.’”

Riccobono tested at Daytona only during the week of the race.

“Daytona had a parking lot that they allowed us to use, and we did some testing

Go Virginia Tech and TORC tested the vehicle for two days in December at the Virginia International Raceway. The NFB tested on its own in Baltimore using simulators that the students sent headquarters.

The simulators were about getting used to the nonvisual interfaces and how they communicate information to a blind driver, says Riccobono. The simulators were equipped with video game pedals, DriveGrip and SpeedStrip.

“The limitation of the simulator is of course you do get valuable information from getting in the driver’s seat,” says Riccobono. “The noise of the engine, there are all these environmental cues you pick up on.”

Though a simulator can’t give that kind if information, Riccobono says it was

Virginia Tech engineered DriveGrip to let the driver know which direction and how hard to turn. They’re looking to continue improve on interfaces in the coming year. AUVSI photo.

Mission Critical

•

Spring 2011

17

Blind Driver Challenge — continued from page 17

…for the first time I could strap my family in and take them somewhere on my own direction. in a parking lot, mostly it was just to make sure, to keep the driver in shape and to test the system, test some of the things we were doing in the demo,” says Riccobono.

The day before the Blind Driver Challenge, a number of NFB members showed up to the test parking lot, and Riccobono drove them around in what he calls the first blind driver taxi service.

Most meaningful to Riccobono was being able to take his wife and two children on their very first family drive.

“The first family road trip,” says Ricconbono, “and it was wonderful — a really moving experience personally to make that realization that for the first time I could strap my family in and take them somewhere on my own direction.”

The day of the race was “a little nerve wracking,” he said. A few hours before the challenge, Riccobono joined the 400 members of the NFB and spoke with some of the group, which put him at ease, he says.

“I was really at peace at that point. It was actually much more nerve-racking waiting to get out to the car.”

Once he was on the track, “it was about the moment,” he says. “It was about showing really the achievement of the vision and the work that we had put in.”

Driving at speeds up to 30 miles per hour, Riccobono drove around the track at Daytona, dodging cones and barrels and avoiding boxes being thrown out of a van in front of his car.

“The truth is I was so focused it just seemed, well at the beginning it seemed very slow, much slower than I expected.

Maybe I was driving a little slower because

TORC Technology’s ByWire XGV system turns any manned vehicle into a robotic platform. AUVSI photo.

I wanted to be right on, but once I hit that International Horse Shoe [the infield track section at Daytona], that was really the key moment for me.”

Riccobono aced the course.

Though the Blind Driver Challenge proved that robotic technology could enable drivers that need assistance, Riccobono says the NFB’s relationship with TORC is likely not over.

“We think the guys at TORC are great, and they’ve done great work with us,” he says.

And Virginia Tech isn’t done innovating new ways to make blind drivers a reality.

The students are now working on AirPix, a third interface that outputs tactile information through small holes, much like how air blows through a table hockey board. That device may cut down on the numbing effect a driver would experience after a long time of using an interface

like DriveGrip — much like how motorcyclists’ hands grow numb after hours of vibration.

The next step for the NFB is to build out the challenge, says Riccobono, getting more universities engaged and finding funding to continue academic research.

“The ideal would be to have a series of challenge events where universities could enter their interfaces,” he says. Then they could have a series of blind drivers test the technology at once. “At some point, we’d love to have a race.” n Danielle Lucey is managing editor of Mission Critical.

To watch a video of the Blind Driver Challenge, scan this barcode with your smartphone.

Mission Critical

•

Spring 2011

19