Helping Hawking
September 12, 2012

Intel’s Effort to Help the World’s Most Famous Physicist Communicate

By Justin Rattner, VP and CTO, Intel Corporation

As Chief Technology Officer at Intel, it’s my job to invent new technologies and to judge their market potential. We have been working for some time at Intel on something called facial feature recognition or FFR.  Imagine how much time you could save if instead of having to enter a password, your computer could look at your face and verify your identity. We’ve also been looking at using FFR to enable something we call avatar chat. Instead of sending a video stream of you sitting in front of a camera, we first send the other party your avatar, a 3D representation of you. It can be anatomically correct or it can take certain liberties with your image making you look younger or older or having more hair.

It was in October of last year when I realized there was another application for FFR that nothing to do with identity or chat. I was visiting with the eminent astrophysicist and popular author, Dr. Stephen Hawking, at his office in Cambridge, England. Hawking had invited me in the hope of finding some new technology that would allow him to improve his ability to communicate. Hawking suffers from what the Brits call motor neurone disease or, as it is known in the U.S., Lou Gehrig’s disease.

This degenerative disease causes the patient to lose muscle function over time. Most people with MND don’t live past their 40s, but thanks to the extraordinary care he receives, remarkably Stephen just celebrated his 70th birthday. Over time he’s tried a number of different communication schemes, but none has worked as well as the simple system he began using more than a decade ago. It consists of a light sensor attached to his eyeglasses that can detect when he squeezes his right cheek muscles. The light changes on the sensor and is detected by a bit of electronics which in turn tells his PC to stop moving its cursor.

Stopping the cursor over a particular character on the screen of the PC selects it and appends it to the end of the text string he’s typing. The cursor then resumes its scan of the characters on the screen. As Professor Hawking has aged, the system has become more error prone and his word output has declined. His hope was that I might have an insight on how best to improve on the simple light sensor.

While visiting Hawking and his staff It didn’t take me too long to realize that FFR might be just such an improvement. I asked the staff if Stephen had more than one facial expression. They hesitantly acknowledged that he did indeed. Well, then, I said, “if you can tell them apart, we can teach a computer to do the same.” They seem intrigued, but concerned that like many earlier efforts he might find such a system too invasive or too cumbersome. While searching the lunch buffet for something sweet, Professor Hawking’s administrative assistant came up to me and quietly said, “Stephen has five or six expressions that I can recognize.” With that I knew we were on to something. I explained the idea to Stephen who seemed genuinely excited. Returning home, I quickly engaged the Human Interface Technology team at Intel Labs in an effort to test my belief that FFR could be the next step in helping Stephen communicate with greater speed and accuracy.

To make sure we didn’t unduly burden Professor Hawking, the team scheduled a visit to see him in Cambridge.  They spent the better part of a day capturing his various expressions on high definition video. Stephen was the perfect subject and never tired of the exercise despite many hours of recording.

Once they’d captured the video they returned to the U.S. and began the analysis of Professor Hawking’s facial expressions. To everyone’s delight, the facial feature recognition technique worked quite well. They also went to work on improving the software that Stephen had used for many years. They built an improved version that Stephen would feel quite at home with as well as a new version that was designed to increase his writing speed. They were quite excited about testing it on him. Despite his condition Professor Hawking comes to the U.S. every spring to visit colleagues at Caltech and around the country. This year’s visit created an opportunity to test the FFR technology at Intel rather than at Cambridge. The tests were very successful, and Stephen encouraged the team to move ahead with plans to bring the system to Cambridge once it was fully implemented and debugged.

We are planning to install the system using the personal computer in Stephen’s wheel chair. An unobtrusive video camera will be mounted on the bracket that holds the computer screen. The light sensor on his glasses will be removed, and the system totally controlled by his facial expressions just as I first imagined it would work less than ten months ago. The whole team at Intel Labs is excited to see how well it all works in the field, and we expect we’ll all find out in the blink of an eye.  I’ll be back with more when I have the field trial data to report.

Justin Rattner is Intel’s Chief Technology Officer, Vice President and an Intel Senior Fellow.   He is responsible for leading Intel’s microprocessor, communications and systems technology labs and Intel Research at universities and research labs around the world. .

Email this to someoneShare on TumblrTweet about this on TwitterShare on FacebookShare on Google+

Read more articles in