Assistive technologiesfor persons with mental challenges

Sustainable Development of Communication Technologies
for Children with Mental Challenges

The central role of communications

"When I was four, there was an elevator operator in the school. At that age, I was terribly tense. To relieve the tension, I stretched out my right hand and watched my fingers. This procedure, however weird it may appear to onlookers, soothed my tension, but to the elevator operator, I looked insane.

He ridiculed me with nasty jokes. My total lack of communication worsened the situation – I could nor even utter words the! Every day, he made fun of me. Ridicule hurts. I was young and innocent. The ridicule hurt all the more because I could neither stop his behavior nor offer a retort."

Krishna Narayanan describes the humiliation he faced as a child on the first page of the first chapter of his book "Wasted Talent" , which was the first he wrote, when he learnt how, almost 20 years later. Arguably the most serious effect of mental challenges is in the inhibition of communication.

Understanding communication blockages

Suggestions for how communication with persons who have autism may be understood and structured, are provided by the excellent book 'Brain Behavior Connections in Autism,[1] by Nancy Minshew and Diane Williams. They explain that, the acquisition of information in autism, that is the capacity to attend to, perceive and remember information is not impaired. Formal language abilities like spelling, reading words and non-words, speaking words are sometimes even superior to those of the age-and IQ-matched controls. The capacity for learning the attributes or characteristics of objects and rules are also intact.

Relative impairments are present in higher cortical sensory perception (what is the object in your hand; are you being touched at two analogous spots on opposites sides of your body; what is the number being drawn on your fingertip); skilled motor movements, memory for complex material, higher-order language, flexibility of shifting strategies when one does not work in the circumstances and concept formation.

Using fMRI , it was shown that the brain is "organized the same but did not achieve the same degree of connectivity", thus making it harder for persons with autism to combine information from different parts of the brain. Thus, tasks involving several parts of the brain at once, or the coordination of a large number of muscles, as in writing or speaking, were far harder for persons with autism.

We can look upon this technologically, as a kind of plumbing problem, only one involving information flow from source to destination. The inability to communicate complicates all the other problems persons with autism encounter, in dealing with society, as it does the solutions society applies to them. The plumbing of information may be as mundane as that of water, but it is arguably no less critical. Our effort is to explore ways for the mentally challenged person to communicate with, and via, a computer.

Why computers?

"The Geek Syndrome" December 2001, Wired, talked about how autism was far more common in Silicon Valley than in the rest of the US, and that computers came naturally to persons with autism: "Replacing the hubbub of the traditional office with a screen and an email address inserts a controllable interface between a programmer and the chaos of everyday life... computers are an ideal interest for a person with Asperger's syndrome ... they are logical, consistent, and not prone to moods."

If computers were attractive to many with mental challenges, might they serve as a communication prosthetic -- an alternate means of communication? If we could communicate better with children with mental challenges, perhaps we could teach them enough for them to be able to live and work productively, instead of compounding their disability with illiteracy.

Information Flow Diagrams

The degree and nature of the information blockage depend of course on the disability and its severity. Often, however, there are combinations of disability. To help organize these, we use an information flow model, which is divided into two parts, the input, in which data flows from the individual to the computer, and output, in which data flows in the other direction, from machine to the user. While the focus is on communication through the Internet, the same treatment could be applied to the telephone or other communications medium.

Image 1 : Simple information flow chart for computer input

The above block diagram shows how people normally produce content for communication for the Internet. First, you must know language, then be able to use your hands to operate the keyboard, to produce text for email, let us say, or use the mouse to click on an on-screen keyboard or a browser link. For audio communication, you need to be able to produce content in the language, and via your mouth, you talk into the microphone of a computer which converts the audio into digital signals that software such as Skype then transmits.

This diagram is also helpful in identifying where the blockages might arise. To start with, you may not know the language or lack the vocabulary, a blockage that could be addressed by education. Then, you may not possess the strength and dexterity in your hands to operate a keyboard or mouse, or you may not be able to speak. Here, special input devices can be used. Similarly, we can look at problems on the output side, as below:

Image 2 : How output of information is managed using assistive technology

This diagram shows how users typically get information from a computer. Mail, for instance, is displayed on the monitor, which the eyes of the user perceive, and assuming you are able to decode the language, the information you receive becomes knowledge. Alternately, as the lower part of the flow chart shows, software such as Skype produces sound via a sound card and speakers, which the ears of the user convert into signals for the brain. Once again language decoding is involved in coming to know what the person at the other end said. This facility can be used also by persons with visual challenges.

What might prevent you from accessing mail delivered to you via the Internet, assuming you have a connected computer in front of you, could be some problem with your optical system, a "hardware" problem, if you will. Or, you may be unable to understand the language in which the information is composed. This one might consider a "software" problem, that can be addressed by education, while the "hardware" problems typically require technology.

Placing these diagrams one below the other, keeping in mind that one path is blocked for the blind while the other is not, suggests an obvious diversion, which is what the screen reader achieves.

Free and open source software for children with mental challenges:
the Skid Initiative

This is an area that has not been well served by the open source community. For developing countries, the few products available from the developed world are expensive, and culturally often inappropriate. To address the varied abilities and needs of children with mental challenges, a software platform was needed that could easily be adapted and built upon. The skid initiative began in response to needs of Arpit, a brilliant child with cerebral palsy, with very little locomotor control. The challenge was (1) to help him communicate, not only receive information but also produce and transmit it (2) to find an input method for him that could take maximum benefit of his limited physical abilities.

Aript displayed his fondness for computers and showed that he could reliably provide input to the computer, through a racing wheel, the kind found in video game parlours.[2] We then developed simple software that allowed him to type with it. It took him mere minutes to understand the software and master the complex motions it needed, which he demonstrated by triumphantly typing, "I am Arpit." This was the beginning of SKID initiative, which was launched at and is a free download.

developing skid for children with mental challenges

The next step was to identify how might a person with mental challenges communicate? A clue for this was provided by Krishna Narayanan in "Wasted Talent", when he talked about the "pathetic" efforts of his mother to educate him. It started with his mother trying to get him to speak. It took him a long time to say "mommy", when she held up her picture, because he had to train his muscles to act in unison to produce that complex sound -- not a small achievement. Next, his mother held up a picture of his dad, however, the only word Krishna knew how to say was "mommy". When he laboriously trained his muscles to say "daddy", he no longer could say "mommy." This drove his mother crazy for a few months, wondering if he was retarded, before she came up with the better idea of holding up both pictures and letting him answer by pointing to the correct one.

This provided useful insight that was helpful in the design of the Skid software. Its picbrowser module allowed a child to use pictures as a means of communicating. The child could browse through pictures, and show people the correct one in answer to a question. While this did constitute an important step in being able to communicate, there were other uses in the teaching process to which this simple module could be put.

The approach

Simple, Consistent Interface

Persons with mental challenges typically lack flexibility and therefore crave consistency. Every module in skid has the same look and feel. Irrespective whether the user is adding photo effects to a picture or playing a simple game against the computer, clear choices are laid out for the user, which are relevant to the specific micro-task at hand. Extraneous information that might confuse or distract is kept out.
Module based

In her autobiography, 'Thinking in Pictures', Temple Grandin compares her mind to a VCR. When she hears the word dog, she mentally replays what she calls "videotapes" of various dogs that she's seen, to arrive at something close to the average person's abstract notion of the category that includes all dogs. Pictures grouped in some logical fashion in folders can easily be played back like Grandin's VCR, perhaps allowing the person with mental challenges to grasp and remember the corresponding categories as folders. In picbrowser, pressing a key changes the folder from which images are shown. There are buttons for browsing within a category, but there is also a button for changing the category of picture you are browsing through.

Within each module, a variety of input methods are available. For those with poor motor control, the software permits control via just a single button. With a second button allowing for switching between modules, full control of all parts of Skid is possible for persons for whom a keyboard is too complex. Input can also be provided using a mouse, joystick, game controller, racing wheel or, arguably the most intuitive for a small child, a touch screen.
Image 3: Recognize module

For a visually oriented user, the next step we recommend is the functional literacy module, named "recognize." As the Image shows, this module shows the user one large image, and several small ones. One of the small ones is always the same as the large image, and the task of the user is to spot that one. If the user guesses wrong, she is politely told so, and allowed to keep trying till she gets it right. When she produces the correct answer, she is congratulated, and presented with a fresh problem using pictures from the same folder. In this manner we address Minshew and Williams' caution, that we must seek feedback through action that demonstrates learning, instead of merely relying on the student being able to reproduce the correct answer.

They also point out that "learning and memory could be improved by presenting material to be learned or remembered in small chunks." In Skid, we attempt to simplify a complex task such as writing, in a series of steps. In the first step, the user selects the modules she will need to carry out the task. For writing, a readymade combination is available for selection, which includes the modules abcd, words, nextword, sentences, easypic, backspace, scroll, and scrolldown, which represent micro-tasks that you sequence in the process of writing, and are explained below.

The next steps help the user construct a sentence, through the carrying out of suitable micro-tasks. One way to do that, is to type some words found in the sentence, and to use the sentences module to locate a "canned" sentence using the typed words. To type words, the easypic module offers an easy method of locating a picture of the object whose name is to be typed. The user only needs to type the first few letters of its name. Another way to type words is to use the abcd module to type in a few characters in the word, and then shift to the words module where an attempt is made to match the characters typed with words in the database. A 3 minute video demonstration[3] explains this better.

Image 4 : How writing is simplified into smaller tasks and modules

The image above depicts some of the modules involved in this process in a flow chart, but there are many more that seek to reduce the effort involved in this. The nextword module, for instance, takes a guess at the next word the user may wish to type, based on sentences in its database. The scroll and scrolldown modules allow the user to change the position of insertion or deletion of text.
Image 5 : The backspace module, showing information essential for purposes of deletion

In each module, only a limited amount of information is presented, along with limited choices, as shown in a screenshot of the backspace module above. The original text is show on the right, and options for deletion are the choices: does the user wish to delete a character, a word, or the entire sentence?
Functional Literacy

Working with the Four Steps special school in New Delhi, we have been able to improve on the capabilities of skid. The quiz module allows the student to take multiple choice tests, while the oddmanout module helps the child test her understanding of categories by presenting her with three pictures from one folder, say flowers, and one from another, say fruits, and awarding her a point for selecting the one that does not belong.

Image 6 : The oddmanout module helps the child understand categories

This could be used to train her in distinguishing family members from other persons she regularly encounters. For this, it would only be necessary to create folders such as Family, Neighbors, Teachers, etc. and populate them with appropriate pictures.
Innovations in Input

Looking at the unique abilities of the children, ski has been made such that it can work with a host of input devices like keyboard, mouse, gaming devices like joystick, racing wheel. It also can be easily adapted to work with specially made switches. For example for one of the child with very little locomotor control a special homemade switch can work efficiently.[4]

Further challenges in writing skid

Child is growing

To address growing communication needs and the abilities of diverse children, the software needed to be flexible and capable of easily expanding on an ongoing basis. As the child is growing the demand on skid is also growing. The modular approach of skid helps address this problem. New modules can be added anytime and added to the existing skid platform.


As mentioned in the paper the solution needs to be inexpensive. In order to meet this skid is written on an open source language and is free of cost. Low cost input devices can easily work with skid. A platform that caters to such 'agile' programming, the Ruby on Rails platform was found very suitable, since it encouraged the highly modular approach that is the basis of Skid. Another advantage of Ruby on Rails is that the resulting software is Web-based. Not only is this useful to let people try out the software, but in a school computer lab, such software would also be useful , because you would not need to install it on every computer.

sustainability : student's training

In order to keep the costs reduced and to make developing of skid a sustainable activity we teach students how to write software and in turn students complete one module of skid. From the point of view of students seeking practical training in agile web programming, the advantage Ruby on Rails offers, is that besides html, it only requires you to know Ruby, which is very easy to learn. Conventionally, web programming involved the use of many languages at once, such as php, perl and sql. Ruby on Rails allows you to do the tasks for which three languages were designed, in one.

The Bidirectional Access Promotion Society ( has taken on the task of carrying this activity forward. We welcome the participation of students seeking to learn programming, and are always on the look out for children with mental challenges who might benefit from our efforts.

All software developed on the platform is free of cost and distributed under the GNU Public License.

Testing Skid

As skid is growing, it is being tested in two schools in Delhi. In addition, parents in diverse countries have begun to use skid with their children.

There can be little doubt that children with mental challenges like computers at least as much as other kids. Systematic efforts involving care givers, educators, occupational therapists and other experts in the area need to work together with programmers (or learn programming themselves) to explore how computers may help special children communicate, learn, and play. With computers becoming ever cheaper, the availability of such free software should make it possible to bring this technology within reach of the millions who might benefit from such an approach. Not just persons with mental challenges, but also the environment in which they live would be much more harmonious, if better communication could thus be established.

[1] Nancy J. Minshew Diane L. Williams. Brain Behavior Connections in Autism,

[2] Video "A child with cerebral palsy communicates via a wheel"-


[4] Video on low cost input device for severely disabled to access computer,