Technology to communicate with someone with expressive aphasia?

Question

A family member had a stroke a few years back and now has expressive aphasia. She understands what is said to her, is cognitive of what is going on, but cannot express herself. She is able to respond to yes/no questions (do you want to go shopping? are you looking for your earrings?) She is not, however, able to read (English is not her native language and she hasn't read Hindi for decades.)

I am the technologist in the family, and I intend to come up with something to help us communicate. The idea is to have some sort of picture book where she can point to what she wants.

My first question: does some sort of assistive technology for people with expressive aphasia already exist? These can be hardware or software devices?

If not, then such a software doesn't seem difficult to write. My initial thought is to have an interface with pictures - maybe separated by category (food, shopping) - where she can point to an individual picture to indicate what she needs. We could easily add more items with such a software, and we could have an interface where she (or we) could "flip pages".

Which suggests that the best solution would use a touch screen rather than a mouse. It would be really difficult to train her to aim a mouse or find keys on a keyboard.

We're thinking of maybe getting a tablet and writing some basic software. But tablets computers are expensive and fragile - I'm not sure if it would be able to stand spills or being knocked about in a nursing home.

So my next question: what kind of tablet-like devices are out there which I can program on? I don't know anything about hardware, but if there is something then we could special-order it. What would be safe and durable for such a project? We could do something on an iPod or cell phone, but I feel like that interface would be too small.

Finally, does anyone here have experience with this kind of assistive technology? Things I might not anticipate when designing such a system?

edit

I've added a (pretty hefty!) bounty. I'd kinda like to open this question up to any suggestions, comments, and experiences that people might have. This is a pretty real and important project, so while we will (are working on) a solution, any insights would be particularly helpful.

Right now the plan is to mount a screen in her room. We'll either teach her to use a trackball or use a touch-screen panel, after seeing what she is able to use with a simple prototype. Then software akin to an old "hypercard" deck:

----------------------------------------------------------------
|   --------------          --------------                     |
|   |  Clothes   |          |    Food    |      ...            |       
|   --------------          --------------                     |
|                                                              |
|    Pic of item 1     Pic of item 2     Pic of item  3        |
|                                                              |
|                                                              |
|                                                              |
|                                                              |
|    Pic of item 4     Pic of item 5     Pic of item  6        |
|                                                              |
|                                                              |
|                                                              |
|                                                              |
|  <-Back                                       Next->         |
----------------------------------------------------------------

commentcommentcomment!

Answer 1

You might consider EEG Headsets as a platform of exploration. Much has been done in recent years in so much that dry EEG Headsets can now be programmed to relate particular brain waves with messages. You have to go through a calibration, since no two people have similar brain waves (much like finger prints), but after that you should be able to think whatever it is you wish to do. Additionally, these won't present language barriers since her thought would be language-agnostic to the device itself.

I've even seen these controlling wheelchairs, and video games.

Emotive has an SDK for their $299 Headset.

Answer 2

A local person has a sort of electronic picture book for this purpose. I've never examined it in detail. If you seek out the local experts in dealing with intellectual disability I'm sure you can find your way to the same sort of device.

Answer 3

another possibility is something like this device:

http://www.aphasia.com/patients/product.aspx

the user creates sentences/phrases from icons that represent concepts or phrases. the phrases are played back in a synthesized voice.

there are some tablets designed for rugged applications and medical uses. One list here:

http://www.tabletpc2.com/CompareRugged.htm

And some tablets like Dell's Latitude XT2 XFR are also ruggedized.

iphone or android could be used as a touchscreen platform, but there's not a lot of screen real estate.

Answer 4

Interestingly enough, I've seen exactly this sort of system in use recently but perhaps not where you'd expect it.

Woolworths (in Australia) recently introduced self-serve checkouts where you can scan you own stuff through the bar-code reader and basically take the checkout person out of the loop.

Scales at each station, and a single person monitoring 8 checkout stations, are used to ensure you don't rip them off, and you can now fit those eight checkouts in a space that used to have two. But this is really an aside.

The meat of this answer is the things that don't have bar-codes (deli meats, fruit and vegetable mostly). For these, you place them on the scales and then use exactly the system you describe: selecting a broad category and eventually drilling down to a specific item (fruit/apples/pink-ladies, vegetables/green/peas/minted-peas or deli/meat/sliced/ham/champagne-ham for example).

I'm willing to bet Woolies did a bucket-load of research before foisting this on the general public so it's probably one of the better solutions.

The trick is to figure out the most efficient way of getting from 3 or 4 image selections to a coherent sentence. One example springing immediately to mind is the difference between "I want a both" and "Oh, my God, you really need a bath". Using the same images but just having an arrow that points to either your family member or the party they're talking to could be one solution.

For that reason (and many others), you should probably make the imagechain-to-sentence part of your software as configurable as possible. You will have many false starts so making this so will ease your workload considerably.

As to development, I'd do that on a regular machine (with mouse and keyboard) but make everything in the software reliant on the mouse clicks. A finger touch sensitive screen will be indistinguishable from a mouse-driven one provided you make the images large enough and leave enough spacing between them. People writing programs for the iPhone don't actually write them on the iPhone - there's no reason why you should have to develop on a tablet.

You will need one for testing eventually but that doesn't have to happen straight away. You can even test with your family member without a touch screen. Just sit with them and you click the mouse in the appropriate place when they touch the screen. That way you can delay the costs until the project is well along (and you're more certain of success).

And you shouldn't need to worry too much about reliability. You can get very robust portable PCs. I did some work for a Telco some ten years ago which involved getting jobs out to the field workforce (the bods that fix your handsets or local loop) and they were the roughest group of users I've ever seen.

We used to get machines coming in labeled "communications problem" that were covered in mud because 'Joe' had thrown in through the open van window into a cow paddock because it couldn't contact the base station :-)

Answer 5

As you said, the interface doesn't seem difficult in theory, and it might already exist something... I'm leaving that part to others.

I'd like to give you suggestions from the HW point of view, though.

1. touchscreen monitors are already available, and they are even cheap. E.g. HP.
2. full-fledged touch-based tablet computers exist, but they tend to be expensive. E.g. Dell
3. touchscreen book readers can be a much cheaper alternative to 2. (and much portable than 1.). But you might need to hack a lot, since they are usually TiVoizated. Here is a list of such devices
4. there are lot of rumors about Apple releasing a tablet in 2010 - so your best bet would be to wait, before investing lot of money in the hw

Answer 6

Please don't be shocked, but these things are all pretty expensive.

Hardware
So called talkers exists. Devices where you press a button and a corresponding word/sentence is spoken. With these devices you can also call a person who is in the next room. Look for alpha and delta talkers or have a look here.

Software
For PC software solutions you can look at MindExpress and The Grid.

Answer 7

I can't help much with software, but as a hardware platform you might consider a Nokia N800 "Internet tablet". I've had one for a couple of years, and it's basically a Linux platform with gtk. More details about software at maemo.org. The main thing about the Nokia is that the display and touch screen are outstanding. (The radios are also outstanding but that's not so relevant to you.)

If you just want to get started with some photos, the Quiver Image Viewer is a native Maemo application and does very well at managing large numbers of photos. It would be relatively easy for your relative to point to photos of what she wants.

A brand new N800 costs $179 as of December 2009. They are selling on eBay for circa $100. Quiver is free.

Answer 8

What about a good old fashioned paper book? (This is not meant to be snarky) A touchscreen interface like you are talking about would be very cool, but in the meantime you could print out several pages with the most common images and laminate them. They would have instant boot time, be robust in the face of spills/drops, and cheap enough you could have one in each room plus one in the car.

I'm not suggesting this is the ultimate solution but as intermediate step it gets you a lot of functionality for not much effort, and may even serve as a prototype that gives you insight into how your family member will interact with your device. It's also something a non programmer can be tasked with while you work on the fancier solution.

Answer 9

Emotive was mentioned earlier.

Here is a little more information from their site:

Expressiv™ Suite

The Expressiv suite uses the signals measured by the neuroheadset to interpret player facial expressions in real-time. It provides a natural enhancement to game interaction by allowing game characters to come to life. When a player smiles, their avatar can mimic the expression even before they are aware of their own feelings. Artificial intelligence can now respond to players naturally, in ways only humans have been able to until now.

Affectiv™ Suite

The Affectiv suite monitors player emotional states in real-time. It provides an extra dimension in game interaction by allowing the game to respond to a player's emotions. Characters can transform in response to the player's feeling. Music, scene lighting and effects can be tailored to heighten the experience for the player in real-time. The Affectiv suite can be used to monitor player state of mind and allow developers to adjust difficulty to suit each situation.

Cognitiv™ Suite

The Cognitiv suite reads and interprets a player's conscious thoughts and intent. Gamers can manipulate virtual objects using only the power of their thought! For the first time, the fantasy of magic and supernatural power can be experienced.

---

As I understand, each of these "Suites" is available via a different level of their SDK. You don't have to know the hardware, you just have to know their API.

If you go with a headset like this, you'll need a pretty robust platform to run it (based on the developer headset requirements):

Requirements Windows XP or Vista 2.2 GHz Dual Core CPU or faster 
1.5GB RAM 150MB Hard drive Graphic cards with 256MB memory

As far as hardware to run this, you'll have to use Microsoft (windows embedded?) to leverage their SDK. You might consider a ruggedized notebook from dell. The screen would be easier for all to see the output.

This is a very noble project, and I wish you the best of luck!

Answer 10

Some suggestions late in the night:

a) I am very sure that Apple will release a 10" tablet device with multitouch OS similar to the iPhone in the near future. Might be an option, and could be covered into a soft case that protects it from falling down and damage.

b) The iPod Touch is pretty small. But it has some advantages: b.1) Small! So she can take it with her, anywhere. b.2) Well-documented SDK to code against. Not too hard, in my opinion. b.3) Excellent touch screen (far better than anything else I know about) But it has a big drawback: You may have to "jailbreak" it to get rid of the blocking start-screen after turning it on. She will always have to unlock the screen and find the app icon in the doc, every time she wants to communicate. Probably a jailbreaked device can get around this burden.

c) A digital photo frame with forward/backward buttons. Just compose images like cards for different topics. A picture with clothes, a picture with shoes, a picture with activities...she could zap through these to find the right one. Also, this hardware is not too complex, so it may be possible to buy a device and engineer a better fitting case for her, with much bigger buttons and more safe from falling down / damage.

d) Tobii, eye tracking device. but quiet expensive yet. Maybe this allows her to communicate even much faster. Alternative: There are glasses on the market that can project an image in front of the left or right eye, while they remain transparent. I slightly remember there was an augmented reality device like that with built-in eye tracking. Something like this, coupled with transmitting the image she sees to a portable device could lead to very fast communication. But again, very expensive and hard to develop.

Maybe I would start out with an iPod touch which simply slides images. The one she wants to communicate to could switch on the device for her. Then you could use the built in photos app which features an square-thumbnail overview of photos, and add composed pictures to it. This is very simple to do, and the device is quiet affordable compared to other solutions. She can quickly flip through images and then point on what she means. But it would also not be hard to make an better suiting application which features different category selection thumbnails. I think categorization is really important here. The iPod touch also has another benefit: If you develop something for this, many others could easily take advantage of it by just downloading it from the app store.

Theres another advantage with the iPod touch: If Apple releases a tablet, and I am 99,999% sure they will, you will be able to port your software to that device with ease. And maybe you can take advantage of touch gestures as well, to very quickly step into a category of picture cards. Even movement-gestures are possible, thanks to accelerometers. I think the screen of an iPod touch may hold not more than 6 items at a time. Depends on how small a tab target is allowed to be.

BTW: I'm not a doctor, and I don't know a lot about medicine. But I've heard great stories about a technique called "Bio Feedback". Maybe you could also think about developing an Bio Feedback device/application which allows her to re-train her skills. Also, music therapy is still in early research stage but has shown great success already. Maybe it's also a good idea to think about incorporating sound/music.

Edit: I see many people have provided good ideas here. I wish you the best success with your project, and hope you will share your solution! Maybe you can find related people who are willing to co-invest into your efforts, maybe in the form of an beneficial organization or foundation, to develop something really amazing.

Answer 11

This is a pretty interesting question, one of my family members also suffered from several strokes but he can still speak fairly well, so actually I mostly responded due to your hardware/programming questions because I got a friend who is also disabled and cannot use normal off-the-shelf hardware because he would otherwise inevitably break it rather quickly (Tourette's syndrome). Regardless of that, I found several things on line specifically related to aphasia, so I added those links here, I do however not have any personal experience with any of the mentioned companies or products.

My first question: does some sort of assistive technology for people with expressive aphasia already exist? These can be hardware or software devices?

My findings indicate that "Hardware" solutions for people with difficulties to articulate themselves include:

• http://www.pictoworld.com/
• http://www.cacfrenchay.nhs.uk/comm%5Fover.htm (printed, cardboard, laminated as previously mentioned)
• http://www.cacfrenchay.nhs.uk/comm%5Flowaac.htm
• http://www.cacfrenchay.nhs.uk/comm%5Fhighaac.htm
• http://www.gusinc.com/2010/Home.html (full fledged tablet computers with speech synthesizer)
• http://www.gusinc.com/2010/Communication%5FPackages.html

Solutions specifically for aphasia patients include:

• http://www.aphasia.com/patients/index.aspx
• http://www.aphasia.com/flash/video%5Fdemo%5Fsmalltalk.aspx
• http://www.aphasia.com/flash/video%5Fdemo.aspx
• http://www.epill.com/aphasia.html
• http://www.textspeak.com/aactalkingdevice.htm
• http://www.bestsyndication.com/Articles/2006/Nicole-WILSON/Health/02/022006-aphasia%5Fstroke%5Fspeech%5Fgenerating%5Fdevice.htm

You may find this review interesting, which reads:

There is a new mobile accessory called SmallTalk for those suffering from Aphasia [...] this device is very portable, light and pocket-friendly and beneficial for communication and practice on the go. This device enables users to carry along with them a collection of phrases, icons, and videos anywhere anytime. The Lingraphica speech-generating device, coupled with the SmallTalk provides a complete all-round system for aphasia communication

And interestingly also:

The Lingraphica device can be reimbursed through Medicare. It arrives with the SmallTalk mobile accessory for a no-obligation trial

The review contains demo videos, but there's also a youtube channel showing the whole thing in action: http://www.youtube.com/user/Lingraphica

In addition, you may want to check out the following blog posting which lists other manufacturers of such devices (the blog also contains some reviews): http://aphasiainfo.blogspot.com/2009/03/augmentativealternative-communication.html

• Dynavox
• Mercury
• Say-It!Sam
• Lingraphica

Therapeutic software packages specifically for people with aphasia are reviewed here: http://www.aphasianow.org/Aphasia%5FRecovery/Aphasia%5Ftherapy%5Fsoftware,%5Faids~211/

Software-wise I could find something called "SentenceShaper":

SentenceShaper® is an innovative, patented software program that helps people with aphasia to communicate in their own voices.

We're thinking of maybe getting a tablet and writing some basic software. But tablets computers are expensive and fragile - I'm not sure if it would be able to stand spills or being knocked about in a nursing home.

If I were to prototype/tailor such a custom system, I would personally look into industrial computers (specifically tablet/panel PCs), i.e. those that are "rugged" or meant to be used for outdoor or even military purposes.

Those are usually very durable and designed to be used under heavy conditions. In other words, they are meant to be used under these circumstances, and as such you even have a warranty for this type of use.

So my next question: what kind of tablet-like devices are out there which I can program on?

As long as you aim for something that is PC/WinTel based, i.e. which runs any "current" 32 bit version of windows (i.e. >= Win98), you should be able to write software using standard tools, like .NET, Java, Python etc.

Personally, I would however refrain from choosing any exotic platforms, especially during the prototyping phase. That would make it unnecessarily expensive and complicated.

I don't know anything about hardware, but if there is something then we could special-order it. What would be safe and durable for such a project? We could do something on an iPod or cell phone, but I feel like that interface would be too small.

And IPhone or cell phone is usually not meant to be used under such "heavy" conditions, therefore I would refrain from prototyping/developing anything on these platforms, but also due to the likely UI shortcomings that you will eventually encounter.

While you might be able to develop something useful, you will surely encounter platform limitations, no matter if it's lack of hardware support, screen estate or whatever.

Similarly, developing for any sort of mobile phone will require the use of phone-specific SDKs/APIs and/or other vendor specific dependencies, i.e. your solution will not be particularly portable, it will be highly platform specific and restricted to the features provided by that very platform.

Rugged tablet PCs:

• http://www.winmate.com.tw/Rugged/RuggedType.asp?Type=B010407
• http://www.scanplanet.com/tabletpc.asp
• http://www.ruggedtabletpc.com/

So, while industrial/rugged tablet PCs (with touchscreens) are indeed fairly expensive computers (usually in the range of $2000-$5000 US), older ones (i.e. refurbished, with less up to date specs) can be usually obtained on ebay -or other second hand hardware retails- in the range of $200-$500 US or even less).

If size, weight, mobility, battery power etc should eventually turn out to become crucial factors, a rugged mini notebook (sub notebook) might be a better option - yet, developing for/on an industrial tablet PC ensures that your work is portable and can be easily migrated to another target.

Rugged tablet PCs are reviewed here.

Also, for your further research, according to website of the The National Aphasis Association, these devices are called "Augmentative Communication Devices":

Augmentative Communication Device is a phrase coined to describe various aids to communication. It can refer to a low tech device (such as an alphabet board) or a very high tech device (such as a computer driven by the eye blinks of its user). The critical thing is to match the communication needs and abilities of the user to the right device.

Maybe some of this is useful, all the best!

Answer 12

http://www.pecsusa.com/

This is a commonly used item in the therapy world.

Answer 13

I am very sorry to hear about your family member. My mother has expressive aphasia and lacks the ability to find the words she needs to express herself.

I've seen a notebook of pictures and I'm trying to create one for Mom. Thought I'd share my thoughts and see if you are looking to do something similar.

I'd like to include pictures of family members with their names. Although Mom knows the people, she can no longer find the way to say their names.

I'll also be starting with common words/phrases she needs to use, most of which can be illustrated with a simple picture: - I want to use the bathroom. - I am hungry and would like to eat - etc.

Answer 14

Check out taptotalk.com