The Killer Robot Interface

The Robbie CX30 industrial robot was supposed to set a new standard for industrial robot intelligence. Unfortunately, one of the first Robbie CX30 robots killed an assembly-line worker, leading to the indictment of one of the robot's software developers, Randy Samuels. This paper propounds the theory that it is the operator-robot interface designer who should be on trial in this case. The Robbie CX30 robot violates nearly every rule of interface design. This paper focuses on how the Robbie CX30 interface violated every one of Shneiderman's eight golden rules.

1. Introduction

On May 17, 1992 a Silicon Techtronics Robbie CX30 industrial robot killed its operator, Bart Matthews, at Cybernetics, Inc., in Silicon Heights, a suburb of Silicon Valley. An investigation into the cause of the accident led authorities to the conclusion that a software module, written and developed by Randy Samuels, a Silicon Techtronics programmer, was responsible for the erratic and violent robot behavior which in turn lead to the death by decapitation of Bart Matthews.1

As an expert in the area of user interfaces,2, 3, 4 I was asked to help police reconstruct the accident. In order to accomplish this, Silicon Techtronics was asked to provide me with a Robbie CX30 simulator which included the complete robot operator console. This allowed me to investigate the robot's behavior without actually risking serious harm. Due to my extensive understanding of user interfaces and human factors I was able to reconstruct the accident with uncanny accuracy. On the basis of this reconstruction, I came to the conclusion that it was the interface design and not the admittedly flawed software which should be viewed as the culprit in this case.

Despite my finding, Prosecuting Attorney Jane McMurdock insisted on pursuing the case against Randy Samuels. I believe that any competent computer scientist, given an opportunity to interact with the Robbie CX30 simulator, would also conclude that the interface designer and not the programmer should be charged with negligence, if not manslaughter.

2. Shneiderman's eight golden rules

My evaluation of the Robbie CX30 user interface is based upon Shneiderman's 'eight golden rules' 5. I also used other techniques to evaluate the interface, but those will be published in separate papers. In this section, I offer a brief review of Shneiderman's eight golden rules, a subject which would be more familiar to computer interface experts such as myself as opposed to the robot hackers who read this obscure journal.

The eight golden rules are:

Strive for consistency.
Enable frequent users to use shortcuts.
Offer informative feedback.
Design dialogues to yield closure.
Offer simple error handling.
Permit easy reversal of actions.
Support internal locus of control.
Reduce short-term memory load.

Strive for consistency.

As we shall see below, it is important for a user interface to be consistent on many levels. For example, screen layouts should be consistent from one screen to another. In an environment using a graphical user interface (GUI), this also implies consistency from one application to another.

Enable frequent users to use shortcuts.

Frequent users (or power users) may be turned off by overly tedious procedures. The interface should allow those users a less tedious procedure for accomplishing a given task.

Offer informative feedback.

Users need to see the consequences of their actions. It can be confusing and disorienting for the user if the computer does not show either that it is processing or has processed a command that the user has entered.

Design dialogues to yield closure

Interacting with a computer is somewhat like a dialogue or conversation. The dialogue for every task should have a beginning, a middle, and an end, and it is important for the user to know when a task is at its end. The user needs to have the feeling that a task has reached closure.

Offer simple error handling

User errors should be designed into the system. To state this another way: no user action should be considered an error beyond the ability of the system to manage. If the user makes a mistake, he or she should receive useful, concise, and clear information about the nature of the mistake, and it should be easy for the user to undo the mistake.

Permit easy reversal of actions

More generally, users must be permitted to undo what they have done, whether it is an error or not.

Support internal locus of control

User satisfaction is high when the user feels that he or she is in control, and user satisfaction is low when the user feels that the computer is in control. Interfaces should be designed to reinforce the feeling that the user is the focus of control in the human-computer interaction.

Reduce short-term memory load

Human short-term memory is remarkably limited. Psychologists often quote Miller's law to the effect that short-term memory is limited to seven discrete pieces of information. The interface should do everything possible to lessen the user's memory burden. For example, instead of being asked to type in the name of a file to be retrieved, the user might be presented with a list of files currently available.

3. Robot console overview

The Robbie CX30 operator interface violated each and every one of Shneiderman's rules. Several of these violations were directly responsible for the accident that ended in the death of the robot operator.

The robot console was an IBM PS/2 model 55SX with a 80386 processor and an EGA color monitor with 640x480 resolution. The console had a keyboard, but no mouse. The console was embedded in a workstation which included shelves for manuals and an area for taking notes and for reading manuals. However, the reading/writing area was quite a distance from the computer screen, so it was quite awkward and tiresome for the operator to manage any task that required looking something up in the manual and then acting quickly at the console keyboard. The operator's chair was poorly designed and much too high relative to the console and the reading/writing area. This placed much strain on the operator's back and also caused excessive eye strain.

I cannot understand why a sophisticated system such as this would not include a better device for input. One can only conclude that Silicon Techtronics did not have much experience with user interface technology. The requirements document6 specified a menu-driven system, which was a reasonable choice. However, in an application where speed was of the essence, especially when operator safety was at issue, the use of a keyboard for all menu selection tasks was an extremely poor choice, requiring many keystrokes to achieve the same effect which could be achieved almost instantaneously with a mouse. (See the paper by Foley et. al.)7. (Actually, I had most of these ideas before Foley published them, but he beat me to the punch.)

The robot operator interacted with the robot and made an impact on its behavior by making choices in a menu system. The main menu consisted of 20 items -- too many, in my opinion -- and each main menu item had a pull-down submenu associated with it. Some of the submenus contained as many as 20 items -- again, too many. Furthermore, there seemed to be little rhyme or reason as to why the menu items were listed in the order in which they appeared. A functional or alphabetical organization would have been better.

In the pull-down submenus, some items had up to four pop-up menus associated with them. These would appear in sequence as submenu choices were made. Occasionally, a submenu choice would cause a dialogue box to appear at the screen. A dialogue box requires some kind of interaction between the operator and the system to resolve some issue, such as the diameter of the widgets being lowered into the acid bath.

A menu system presents a strict hierarchy of menu choices. On this system, the operator could backtrack up the hierarchy by pressing the escape key. The escape key could also terminate any dialogue. In addition, the use of color in the interface was very unprofessional: There were too many colors in too small a space. The contrasts were glaring and the result, for this reviewer, was severe eye strain in just fifteen minutes. There was also excessive use of screen flashing and silly musical effects when erroneous choices or erroneous inputs were made.

One has to wonder why Silicon Techtronics did not attempt a more sophisticated approach to the interface design. After a careful study of the Robbie CX30 applications domain, I have come to the conclusion that a direct manipulation interface, which literally displayed the robot at the operator console, would have been ideal. The very visual domain that the robot operated within would lend itself naturally to the design of appropriate screen metaphors for that environment, metaphors the operator could easily understand. This would allow the operator to manipulate the robot by manipulating the graphical representation of the robot at the computer console. I have asked one of my doctoral students, Susan Farnsworth, to give up her personal life for the better part of a decade in order to investigate this possibility a bit further.

4. How the Robbie CX30 interface violated the eight golden rules

The Robbie CX30 user interface violated each and every golden rule in multitudinous ways. I shall only discuss a few instances of rule violation in this paper, leaving a more detailed discussion of these violations for future articles and my forthcoming book8. I will emphasize those violations which were relevant to this particular accident.

4.1 Strive for consistency

There were many violations of consistency in the Robbie CX30 user interface. Error messages could appear in almost any color and could be accompanied by almost any kind of musical effect. Error messages could appear almost anywhere at the screen.

When Bart Matthews saw the error message for the exceptional condition that occurred, an exceptional condition that required operator intervention, it was probably the first time he had seen that particular message. In addition, the error message appeared in a green box, without any audio effects. This is the only error message in the entire system which appears in green and without some kind of orchestral accompaniment.

4.2 Enable frequent users to use shortcuts

This principle did not appear in any way in the entire interface design. For example, it would have been a good idea to allow frequent users to enter the first letter of a submenu or menu choice to effect a choice, in lieu of requiring the use of the cursor keys and the enter key. The menu selection mechanism in this system must have been quite a mental strain on the operator.

Furthermore, a form of type-ahead should have been supported, which would have allowed a frequent user to enter a sequence of menu choices without having to wait for the actual menus to appear.

4.3 Offer informative feedback

There are many cases in which a given sequence of keystrokes represents one holistic idea, one complete task, but the operator is left without the kind of feedback which would confirm that the task has been completed. For example, there was a fairly complicated dialogue necessary to remove a widget from the acid bath. However, upon completion of this dialogue, the robot operator was led into a new, unrelated dialogue, without being informed that the widget removal dialogue had been completed.

4.4 Design dialogues to yield closure

There are many cases in which a given sequence of keystrokes represents one holistic idea, one complete task, but the operator is left without the kind of feedback which would confirm that the task has been completed. For example, there is a fairly complicated dialogue which is necessary in order to remove a widget from the acid bath. However, upon completion of this dialogue, the user is led into a new, unrelated dialogue, without being informed that the widget removal dialogue has been completed.

4.5 Offer simple error handling

The system seems to be designed to make the user regret any erroneous input. Not only did the system allow numerous opportunities for error, but when an error actually occurred, it was something that was not likely to be repeated for some time. This is because the user interface made recovery from an error a tedious, frustrating and at times infuriating ordeal. Some of the error messages were downright offensive and condescending.

4.6 Permit easy reversal of actions

As mentioned in the previous paragraph, the user interface made it very difficult to recover from erroneous inputs. In general, the menu system did allow easy reversal of actions, but this philosophy was not carried through to the design of dialogue boxes or to the handling of exceptional conditions. From a practical (as opposed to theoretical) point of view, most actions were irreversible when the system was in an exceptional state, and this helped lead to the "killer robot" tragedy.

4.7 Support internal locus of control

Many of the deficiencies discussed in the previous paragraphs diminished the feeling of an "internal locus of control," for example: not providing feedback, not bringing interactions to closure, and not allowing easy reversal of actions when exceptions arose. All of these things acted to diminish the user's feeling of being in control of the robot. There were many features of this interface that could make the user feel as though there was an enormous gap between the operator console and the robot itself, whereas a good interface design would have made the user interface transparent and would have given the robot operator a feeling of being in direct contact with the robot. In one case, I commanded the robot to move a widget from the acid bath to the drying chamber and it took 20 seconds before the robot seemed to respond. Thus, I did not feel as though I was controlling the robot. The robot's delayed response, along with the lack of informative feedback at the computer screen, made me feel that the robot was an autonomous agent -- an unsettling feeling, to say the least.

4.8 Reduce short-term memory load

A menu driven system is generally good in terms of the memory burden it places upon users. However, there is great variation among particular implementations of menu systems. The Robbie CX30 user interface had very large menus without any obvious internal organization. These placed a great burden upon the operator in terms of memory, and also in terms of scan time, the time it takes the operator to locate a particular menu choice.

In addition, many dialogue boxes required the user to enter part numbers, file names, and other information from the keyboard. This greatly increased the memory burden upon the user. The system could easily have been designed to present the user with part numbers, and so forth, so the user would not have to recall these things from his or her own memory.

Finally, and this is really unforgivable, incredible as it may seem, there was no on-line, context-sensitive help facility! Although I was taken through the training course offered by Silicon Techtronics, I often found myself leafing through the reference manuals in order to find the answer to even the most basic questions, such as: "What does this menu choice mean? What will happen if I make this choice?"

5. A reconstruction of the "killer robot" tragedy

Police photographs of the accident scene are not a pleasant sight. The operator console was splattered with a considerable amount of blood. However, the photographs are of exceptional quality and using blow-up techniques, I was able to ascertain the following important facts about the moment when Bart Matthews was decapitated:

The NUM LOCK light was on.
The IBM keyboard contains a calculator pad that can operate in two modes. When the NUM LOCK light is on, it behaves like a calculator. Otherwise, the keys can be used to move the cursor at the screen.
Blood was smeared on the calculator pad.
Bloody fingerprints indicate that Bart Matthews was using the calculator pad when he was struck and killed.
A green error message was flashing.
This tells us the error situation in force when the tragedy occurred. The error message said, "ROBOT DYNAMICS INTEGRITY ERROR - 45 ".
A reference manual was open and laid flat in the workstation reading/writing area.
This one volume of the four volume reference manual was open to the index page that contained the entry "ERRORS/MESSAGES."
A message giving operator instructions was also showing on the screen.
This message was displayed in yellow at the bottom of the screen. It read "PLEASE ENTER DYNAMICAL ERROR ROBOT ABORT COMMAND SEQUENCE PROMPTLY!!!"

On the basis of this physical evidence, plus other evidence contained in the system log, and based upon the nature of the error that occurred ("robot dynamics integrity error - 45," an error caused by Randy Samuels' program), I have concluded that the following sequence of events occurred on the fateful morning of the "killer robot" tragedy:

10:22.30. "ROBOT DYNAMICS INTEGRITY ERROR - 45" appears on the screen. Bart Matthews does not notice this because there is no beep or audio effect such as occurs with every other error situation. Also, the error message appears in green, which in all other contexts means that some process is proceeding normally.
10:24.00. Robot enters state violent enough for Bart Matthews to notice.
10:24.05. Bart Matthews notices the error message, but does not know what it means and does not know what to do. He tries the "emergency abort" submenu, a general purpose submenu for turning off the robot. This involves SIX separate menu choices, but Mr. Matthews does not notice that the NUM LOCK light is lit. The menu choices aren't registering, because the cursor keys are operating as calculator keys.
10:24.45. Robot turns from acid bath and begins sweep towards the operator console, its jagged robot arms flailing wildly. No one anticipated when the console was designed that the operator might have to flee a runaway robot, so Bart Matthews is cornered in his work area by the advancing robot. At about this time, Bart Matthews retrieves the reference manual and starts looking for a reference to ³robot dynamics integrity error - 45² in the index. He successfully locates an entry for error messages in the index.
10:25.00. Robot enters the operator area. Bart Matthews gives up on trying to find the operator procedure for the robot dynamics integrity error. Instead, he tries once again to enter the "emergency abort" sequence from the calculator keypad. As he does this, the robot strikes him.

6. Summary and conclusions

While the software module written by Randy Samuels did cause the Robbie CX30 robot to oscillate out of control and attack its human operator, a good interface design would have allowed the operator to terminate the erratic robot behavior. Based upon an analysis of the robot user interface using Shneiderman's eight golden rules, this interface design expert has come to the conclusion that the interface designer, and not the programmer, was the more guilty party in this unfortunate fiasco.

7. Footnotes

The media were misled to believe that Bart Matthews was crushed by the robot, but the photographic evidence given to this author shows otherwise. Perhaps authorities were attempting to protect public sensibilities.
Gritty, Horace. "The Only User Interface Book You'll Ever Need." Oshkosh, WI: Vanity Press, 1990. pg 212
Gritty, Horace. "What We Can Learn from the Killer Robot." 1992. Invited talk given at the Silicon Valley University International Symposium on Robot Safety and User Interfaces, March 1991. Also to appear in Silicon Valley University Alumni Notes.
Gritty, Horace. "CODEPENDENCY: How Computer Users Enable Poor User Interfaces." New "York: Angst Press, expected 1993.
Shneiderman, Ben. "Designing the User Interface." Reading, Massachusetts: Addison-Wesley, 1987. pg 448 (see footnote 9)
Robbie CX30 INTELLIGENT INDUSTRIAL ROBOT REQUIREMENTS DOCUMENT : Cybernetics Inc. Version, Technical Document Number 91-0023XA, Silicon Techntronics Corporation, Silicon Valley, USA, pg 1245
Foley, J. P., L.V. Wallace, and P. Chan. "The Human Factors of Computer Graphics Interaction Techniques." IEEE Computer Graphics and Applications, 4(11). 1984. pg 13-48 (see footnote 9)
Gritty, Horace (expected 1993). CODEPENDENCY: How Computer Users Enable Poor User Interfaces, Angst Press, New York. This book presents a radically new theory concerning the relationship between people and their machines. Essentially, some people need a poor interface in order to avoid some unresolved psychological problems in their lives.
Foley and Shneiderman are actual authors. The other references are fictitious.

The Case of the Killer Robot is a fictional scenario for ethics teaching and discussion purposes.

Continue to Part 8: Software Engineer Challenges Authenticity of "Killer Robot" Software Tests

Citation

Richard G. Epstein. . The Killer Robot Interface. Online Ethics Center. DOI:. https://onlineethics.org/cases/killer-robot-interface.

Site Search