I said I would produce a description of my final year project. Sorry its a little longer than I hoped! Simon Coles [log in to unmask] --------------------------------------------------------------------- CANDi ===== A communications aid for the neurologically disabled Copyright Simon Coles 1994 Abstract: ========= This document describes CANDi (Communications Aid for the Neurologically Disabled), a software package under development for Apple's Newton range of Personal Digital Assistants. CANDi hopes to improve on existing communications aids for people with severe disabilities reducing the key presses required to form a message. Introduction: ============= This is the first time CANDi has been described to a wide audience. I welcome (need!) frank and constructive criticism. All correspondence should be directed to "[log in to unmask]". The document is split into several sections: - Background. - A brief introduction to Newton Technology. - The aims of CANDi. - Main functions of CANDi. - Functional enhancements to be added at a later date. - A discussion of the feasibility of CANDi's aims. - Why Apple's Newton, an "off the shelf" hardware platform, has been chosen, and the advantages this brings. - A discussion on the possible future directions that CANDi could progress over the longer term. - An estimate of progress so far, and possible dates when the software will be available. - Some definitions. Please feel free to pass this document onto whoever you feel might be interested. Background: =========== Target Audience: ---------------- CANDi, as the full name suggests, is aimed at people with severe neurological disabilities & associated conditions. These people are generally not well served by existing communications aids. The initial release is aimed at the most severe Parkinsonians, as I have personal experience in caring for these people, but there is no reason why it cannot be useful for other conditions. Problems: --------- Severe Parkinson's Disease (PD) sufferers often have reduced motor control, and badly impaired speech. Whilst conventional communications aids are helpful, they have the following problems: - use of a keyboard is difficult, slow, and prone to errors (hitting the wrong key, or hitting it more than once). - co-ordinating their movements is often difficult, making "grid scanning" systems impractical. Some Parkinsonians will have these problems only during a bad "off" period; very severe cases may have them continuously. PD is a very individual condition; each person has specific symptoms and needs. This makes the task of "occasional" carers difficult - you cannot go on a course and be taught how to care effectively for all Parkinsonians. Communication with the sufferer is essential for effective care, and often this communication with the sufferer itself requires extensive background knowledge of the individual. The occasional carer's lack of detailed background information tends to cause problems with any kind of respite care: - they lack the detailed experience and techniques needed to communicate with the sufferer. - they lack an understanding of the sufferer's specific requirements, which has a detrimental effect on the sufferer (sufferers often return worse from respite care). - principal carers feel guilty for putting the sufferer in for respite care because they know they are condemning their charge to a period of well-meaning incompetence. It is difficult to over-emphasise the dependence the sufferer has on the principal and secondary carers, and the additional stress this causes on both sides. CANDi hopes to reduce this dependence by allowing sufferers to communicate effectively even with inexperienced carers. In addition, it is hoped it can provide the means for them to communicate easily with other people and so enhance the quality of their life. History: -------- My Mother suffered from a condition known as Multiple System Atrophy, which presents itself as a very severe case of Parkinson's. CANDi originated from my observations of how I and other carers communicated with her. Some people were obviously better "listeners" than others; we began to realise this was due to: - techniques to make the best use of the sufferer's limited communications abilities. - background knowledge to reduce the amount the sufferer had to say. Given a suitable (portable) computer, the process of communication, even with inexperienced people, could be made much easier. Just before I started my final year project, Apple released the Newton; a suitable hardware platform. During my background research, it became apparent there were some theoretical reasons for believing communication could be made easier. Later, more functions and ideas came along - there are a lot of things you can do when you put a large amount of computing power in someone's hand. The CANDi concept sort of migrated from a simple communications device into an expert system/carer's aid. Aims: ===== CANDi has two principal functions: - to help the sufferer to communicate effectively, even when they have reduced motor function. - to provide occasional carers with some of the background information they require to care effectively for the individual. It is hoped this will therefore improve the quality of life for all concerned. Newton Technology: ================== "Newton" is a line of Personal Digital Assistants produced by Apple Computer. They are small, portable computers which recognise your handwriting when you write on their touch sensitive display - they have no keyboard. Eventually there will be many different devices implementing this technology, from many manufacturers. The software should work unchanged on all models, so users may pick the device most suited to their needs. The first Newton is the MessagePad, a small device the size of a VHS video cassette. The touch screen is 3" wide and 4" high. I am using the MessagePad to develop CANDi, and to show the software's potential. Initial indications are that CANDi will be better suited to larger devices. To give you an idea of price, apparently these are MacWarehouse's latest prices for the MessagePad - treat these only as a ballpark figure! Newton MessagePad $799 2 meg storage card $279 Apple fax modem $139 ext. battery pack $ 25 Which gives you a CANDi capable system for $1,242 (painful!). This will hopefully drop to more reasonable levels soon. UK users will probably have to pay around 813 Pounds Sterling at today's prices. Main Functions: =============== CANDi has two main communications modes: - "Typing" (the user types and the system fills in the word) - "Phrase" (the system tries to predict the phrase) The distinction between the two is somewhat arbitrary, and they will probably be merged in the future; at the moment they are separate because there isn't enough room on the MessagePad screen. There is an additional "Question and Answer" mode which helps inexperienced carers diagnose problems. All modes are based on the techniques carers use to communicate with sufferers. The prediction mechanism, used in all modes, constantly attempts to tune itself to the individual user's habits, presenting only the most likely choices. Typing Mode: ------------ This is the simplest mode, and is a replacement for the conventional typewriter based communication aid. The intention is to support as full a range of communication as possible, without the user having to do any unnecessary key presses. A keyboard is displayed on screen, together with a list of words. As the user taps on the keyboard, the machine searches an 8,000 word dictionary and presents the user with a list of words which they might be typing. For example, if the user typed "a", 16 of the most likely words beginning with "a" would be presented. The user may make a selection from this list, or continue typing. If the next letter was "t", then the most likely words beginning with "at" would be shown, and so on. With this mode, the user will probably be able to type any message with around 2 or 3 key presses per word, as opposed to an average of 5 with a conventional typewriter. The biggest problem with this on the MessagePad is screen area - I can't fit a usable keyboard on the screen, let alone a decent sized word list. Therefore phrase mode is provided for those situations where the sufferer has difficulty with the small keyboard. With a bigger screen, the two modes can be merged, which will make it more effective. The keyboard can be one of many layouts, e.g. QWERTY, Alphabetic, etc. - the user can choose the most familiar. Phrase Mode: ------------ Most messages are fairly standard, for example when the sufferer needs to communicate a simple request to the carer: "I want corn flakes for breakfast" "Please turn the TV to Channel 4" etc. Phrase Mode allows the sufferer to communicate these simple everyday requests, even when they are having an "off" period (and hence reduced typing ability), by reducing the number of presses required. The user is presented with a list of 16 words in large buttons on the screen. They chose one, and then the list is refreshed with new words which follow. In this way, short phrases can be built up with just a few presses. Initially, the machine will come with a few phrases built in, enough to get you started. Over time, you can teach the machine your own. The phrases are in fact stored in a tree structure inside the machine. A diagram might help: |--- Tea | |-- Drink --|--- Coffee | | |--- Food ---| |--- Water | | | | |--- Biscuit | | | | |-- Food ---|--- Chocolate | | | |--- (..... etc. ....) | | Personal -|--- Toilet (..... etc. ....) | |--- Move (..... etc. ....) | | | |--- Hand |--- Pain --| |--- Arm The user's first choice would be "Personal", followed by "Food" (from a list of "Food", "Toilet", "Move", "Pain"), followed by "Drink", then "Water". This allows simple, common phrases to be built up with only one press per word. The system can be personalised, i.e. if the sufferer only drinks Coke, and never water. This mode has several problems, which is why it is only used when the user can't operate the typing mode. - the phrase must be known by the machine (new ones can be added with the keyboard & word predictor from the typing mode). - the tree must be structured so the most frequent words appear in the top levels. - all unnecessary words should be omitted from the phrase. e.g. "now want coffee", rather than "Please can I have some coffee now". Grammatical niceties are out. Question and Answer Mode: ------------------------- There are occasions where the sufferer is obviously distressed, although the cause is not known. Because they are distressed, they find it difficult to communicate their need to the carer. Carers will know how frustrating this situation is - it is obvious something needs to be done, although it is difficult to know what. In these situations, experienced carers go through a process of elimination, getting the sufferer to signal "yes" or "no" to a series of questions: "Do you have cramp?" "Do you need to go to the toilet?" Not solving the sufferer's problem correctly in a short time can cause additional stress. It is primarily because of these situations that principal & secondary carers feel uneasy about leaving their sufferer in the care of occasional carers. Q & A mode leads the carer through a series of questions in order to ascertain the problem & solution in the shortest possible time. These questions will have to be programmed by the principal carer before hand. Basically all the problem diagnosis skills of the carer are placed within CANDi - an expert system, configured to the sufferer's individual problems & needs. General: -------- The three modes are designed for different levels of disability; this reflects the fluctuations in symptoms that Parkinsonians can experience. The important thing is to gain maximum benefit from whatever motor control the sufferer has. The size of the buttons will be very important; Parkinsonian's physical co-ordination can be poor, so buttons should be as large as possible. If the buttons are too large, there won't be many of them on screen. Hence a rather difficult trade off. Eventually it is hoped the size of the buttons can vary dynamically - the system can monitor how co-ordinated the person is, and adjust the button size accordingly. Other Functions: ================ The "Main Functions" section described the areas I am committed to completing for my Final Year Project; they form the basis for the device. This section explores the other ideas I have - long term dreams rather than plans. A lot of these functions rely on the "central machine", so perhaps it would be best to describe this first. A user will have a lot of information in their CANDi; perhaps many years worth of configuration. It is sensible to provide some way for them to back this up. It is unlikely they will have access to a PC or Mac to do it themselves. Fortunately, there is a fairly cheap Newton modem available (and people can share one); users could phone up some central machine, and send all their data down the phone line. If all users have modems and are contacting the central machine on a regular basis (every month? every week?), other things become possible. a) To get the CANDi software, users go down to their local electronics shop and buy a Newton and Modem. They then phone up and download the software and initial configuration. This is slightly expensive in phone bills, but cheaper than organising a distribution network. b) To back their CANDi up, they just phone the central machine. c) As the central machine has all the data from all the machines, it could merge them into one big pool. This can be pushed back to individual machines - so they all gain from each other's experience, learning much faster than they would individually. This is what I call "Federated Learning". d) I can release software on a "little & often" basis, without bothering the users. This should allow the development of CANDi to proceed faster than it would otherwise. e) I can gather metrics on how successful the prediction mechanisms are, and adjust them to maximise performance. f) Questionnaires prepared by other researchers can be downloaded and answered by people at their convenience. This will be cheaper and more effective than any other method. g) Users can decide what reference material they want to download. h) Information can be gathered on how co-ordinated the user is, day-to- day, and hour-to-hour. This could allow medical personnel to measure the effectiveness of drug treatment. There is no possibility of confidentiality being compromised by sending data to the central machine; it will be in such a form as to make reconstruction of original messages impossible. The system could store people's basic preferences - what foods they don't like, how they take their coffee etc. It could also document their drug regime and record medical/care history (as entered by the carer). There is no reason why CANDi cannot be used either as a keyboard for a conventional computer (i.e. IBM PC compatible), or as a serial terminal onto another system. In addition, the typing mode keyboard can be used instead of the normal Newton keyboards & handwriting recognition. So all the normal functions of the Newton are still available to you. A reference library could be included on the device (as a collection of Newton-format books), to show occasional carers general information about the disease. Unlike paper, this can be dynamic, so an animation could show how to move a sufferer, for example. One of the more crazy things is "Automatic Politeness". Often sufferers want to say "thank you" when they have been helped, or to put "please" in front of requests. Often its too much hassle - I can remember spending 10 minutes getting very frustrated with Mum, and all she wanted to say was "thank you". So automatic politeness would put "please" and "thank you" in the sentence where appropriate (it sounds a little automatic, but I think no one would notice after a while). How & Why: ========== When I first told them I wanted to predict speech, most people laughed. This is a very difficult problem! Yet the task becomes easier when you use the symptoms of the disease to help. Parkinsonians don't want to have wide ranging philosophical debates (they probably do, but I can't cater for that yet). What they need to do is communicate needs to their carer. This means the communication is over a very restricted domain. The grammar can also be simplified - the carer will share a lot of context with the sufferer, so a lot of the information which is carried in full grammar can be guessed (i.e. tenses and modality). There is no need to produce perfect English prose. So: - The vocabulary can be restricted - The grammar can be simplified & abused. This makes prediction practical. As the technology improves, a wider range of conversation can be supported. Technology: =========== The use of a consumer hardware platform like the Newton brings several advantages over custom solutions: - Development and support should be easy enough that I can to it in my spare time. - The hardware is going to evolve at a much faster rate, and produce more varied products than could ever be achieved with a low volume custom device. CANDi should be usable on all Newtons, so pick the one that suits you best. - The hardware is more powerful than anything I could produce for the same price. - Newton is available from your local shops. - Spares & Servicing can be done by your local dealer. Which means I can distribute CANDi far and wide without having to worry about a support system. - Other Newton software products will be useful: Mail systems, fax, books, games etc. - Its fun (for me to write, and for you to use). - It doesn't look like a "disabled device", more like a yuppie toy. There is no embarrassment carrying this in a social function. CANDi at the moment runs on a Newton MessagePad with a 2Mb RAM card. The Future: =========== The biggest problem I have at the moment is screen size - especially for the typing mode. There isn't really enough space to fit everything on in a useful size. For the moment I will do the best I can with a screen 3" by 4". But CANDi will only really work on a larger screen - wait for the "SlatePad" out in the latter half of the year. This should have a much larger, backlit screen which will help a lot. In the mean time, on a MessagePad, I feel the following features will be usable: - Prediction Mode. - Question and Answer mode. - Typing mode will not be at its best. As Apple and other manufacturers produce further Newton devices, users can pick the one most suited to their needs. It may be that a small, hand held device would be appropriate initially, with an upgrade to a larger machine later. CANDi is at the moment being privately financed. With the resources I have it should be possible to place pilot devices with a small selection of people, provided their Newton is financed from some other source. However, more resources make progress easier, and there is no doubt that running this project on a shoestring is an impediment. So offers of help, or suggestions of who to approach would be very welcome! In addition, if anyone has leaflets or books which they are willing to have distributed electronically with CANDi, please let me know. I'll probably start thinking of including a reference library in about 6 months. CANDi will be a very effective way of getting large amounts of information to people cheaply, and I would hope a good selection would be forthcoming (as most societies give their information away free, or at cost). Progress to Date: ================= This section describes where I am at this moment in time, and where I hope to get to soon. - The phrase mode works, although you can't add any new phrases yet. - The typing mode works, although the keyboard is too small. - The Q & A mode is still very much an idea at the moment, although I have a good idea how it will work. It will probably get written as "relaxation" during my exams. Before I release beta versions to people who already have Newtons (I don't want the responsibility of people spending their hard earned cash on this yet) I need to: - build a list of words. - build a list of phrases for the prediction mode. - solve the problem of how to efficiently get 1.5Mb of initial data into each person's machine. - write a user manual. I would hope to get this done so people can start playing with this in Q2/3 1994. I don't know if Q & A mode will get done by then. In March, I disappear for a while, for my final exams. After the exams (which end in May), I work full time on the project. This will probably involve all the boring stuff I have to do to pass my degree - documentation etc. I would hope to get a more polished version done though, and incorporate any comments from the first version. Once I get out of University, I would hope to get a Version 1.0 release out in Q4 of 1994, although I may well wait for Apple to release the SlatePad. So the upshot is: - beta versions, which will show potential but not a lot of utility, will be out by Q2/3 1994. - Version 1.0 will be out in Q4 1994. After the first release, I will launch a programme of incremental improvements, with a version coming out probably every month. This is where most of the "Other functions" described above will happen. Definitions: ============ Carer: A person engaged in assisting the Sufferer. The distinction between the three types of carer is due to the different amounts of familiarity they have with the suffererUs condition, and hence how much background knowledge they have during communication. Occasional Carer: A person who might care for the sufferer infrequently. For example, nurses on respite care wards, members of the general public, relatives who do not live with the sufferer. These people typically do not have the opportunity to learn the suffererUs condition in detail. In order to care effectively for the sufferer, occasional carers generally need to communicate with them. Principal Carer: The suffererUs most frequent carer, generally their husband, wife or other close relative. Whilst these people will generally have little or no medical background, they will have the most experience of the sufferer's condition. Secondary Carer: People who care for the sufferer on a regular basis, generally members of the immediate family. Whilst they may not have all the knowledge of the Principal Carer, they will have a detailed knowledge of the sufferer. Catastrophic Disability : "Disability" has become a very imprecise word, and can be taken to mean almost any impairment of function. Therefore I have used the term "Catastrophic Disability" to emphasise the multiple problems typical of very severe Parkinsonians (and other similar conditions). With such conditions there are a great many problems - sufferers have great difficulty moving, speaking etc. Thankfully such conditions are very rare indeed, and few Parkinsonians ever reach this stage. But I needed to emphasise CANDi is not needed by, or intended for, the typical sufferer. Button: Area of the screen which has some text in it and does something when pressed. Respite Care: When the sufferer goes into short term residential care (e.g. into hospital) to give their principal carer a rest. Beta Version: A release of a software package which has not yet been finished or fully tested. Use at your own risk. This document was written by Simon Coles 30/1/94. Last updated 10/2/94. --