Student software projects aim to help teen communicate

To signal “yes,” Grace Simon raises her right fist. For “no,” she shakes her head. And that, for the most part, is how the 13-year-old communicates with the world. Diagnosed with cerebral palsy at six months old, Grace doesn’t have the muscle control to speak, sign or walk. But her capable mind understands spoken language. (She’s actually kind of nosy, says her mother.) She reads chapter books. Her favorite subject is math.

“She’s really bright on the inside, but she just can’t get it out,” said Dr. David Chesney, a University of Michigan lecturer in computer science and engineering who has built a syllabus around the sixth-grader.

This semester, the 73 undergrads in Chesney’s EECS 481 senior-level software engineering course are devising systems that could make it easier for Grace to communicate, play, or act more independently at school or home. The course requires no textbook. Teams of undergrads are working on 15–20 Grace-centric projects that the students design themselves. They might enable her to play a board game with friends, to make her own art, or to share her thoughts in writing or perhaps even (electronically) spoken words.

Grace and her parents, Jennifer and Eric Simon, of Westphalia, Mich., are working closely with Dr. Chesney. They’ve come to class several times already. During their first visit, the sprightly teen dressed in a hot pink sweater and silver sandals steered her powered wheelchair across the room to demonstrate the motor skills she’s refined over the past decade to get herself around. To steer, Grace maneuvers a sort of joystick on the chair arm with her fingers and the heel of her right hand.

She also took questions from the class. Can she use a calculator? No. Does she get tired from moving? Yes. Does she like puzzles? No. Would she rather be able to type out words or speak through a text-to-speech program? With help from her mom, Grace answered yes to “speak.”

It’s an ambitious goal, but perhaps the greatest hope for the class.

“It’s been hard for us to get something specifically for her that could give her verbal communication. We have never come up with anything so far,” said Jennifer Simon, Grace’s mother.

Today Grace doesn’t have a good way to enter information into a computer – whether that information is individual letters, whole words or selections from a list of options. And enabling her to do that is a crucial first step.

Grace has limited control of her hands and arms, so the students will design special interfaces that rely on tools other than a mouse and keyboard. They’ll explore a joystick similar to the one on her chair. They’ll look into an operating system called ASK Interfaces that previous students in Dr. Chesney’s class developed a few years ago. Designed to help people with other forms of cerebral palsy send email, ASK turns a whole tablet computer screen into one big button and tabs through options.

She could also potentially control a computer with her eyes. Grace recently tested that approach with Dr. Jacqueline Kaufman, the clinical assistant professor of physical medicine and rehabilitation at the U-M Medical School who connected the Simons with Dr. Chesney.

“I was impressed with how well she did,” Jennifer Simon said.

Communication systems such as the Eyegaze Edge track pupils with a camera and allow users to select letters, words, blocks of text or other items with their eyes. The system costs more than $13,000. Dr. Chesney challenged his students to improvise with some of the cheaper technologies companies have donated to the class. Intel gave six Creative Interactive Gesture Cameras, which can recognize motion in three dimensions, as opposed to the typical two. Microsoft gave a dozen Kinect motion sensors that can track facial movement.

As for Dr. Chesney’s expectations, it’s hard to say. The Simons know there are no guarantees. “My threshold for success is widely varying,” Dr. Chesney said. “On the low end, my students would come out of the class with a better understanding that there are people out there with special needs, and that they carry that with them once they become software engineers. On the high end we’d have 15 systems that we give to Grace and say, “Here you go!”

And on the highest end, Grace would get a voice, or something like one.

“It’s exciting to think that’s a possibility,” Jennifer Simon said. She looked at her daughter and smiled, then teased: “But she might never shut up!

“I’d want to know what she’s thinking,” she continued. “What are you thinking? What do you know? What have you been dying to tell us and you haven’t been able to?”

More info:

See a photo collection by Joseph Xu, Michigan Engineering photographer.