---

Did HAL Commit Murder?

Daniel C. Dennett

The first robot homicide was committed in 1981, according to my files. I have a yellowed clipping dated 12/9/81 from the Philadelphia Inquirer--not the National Enquirer--with the headline:

Robot killed repairman, Japan reports

to have performed a legally prohibited action, such as killing another human being; one must have done so with a culpable state of mind, or mens rea. Such culpable mental states are of three kinds: they are either motivational states of purpose, cognitive states of belief, or the nonmental state of negligence. [Cambridge Dictionary of Philosophy 1995, p.482]

When IBM's computer Deep Blue beat world chess champion Garry Kasparov in the first game of their recent championship match, it did so by discovering and executing, with exquisite timing, a withering attack, the purposes of which were all too evident in retrospect to Kasparov and his handlers. It was Deep Blue's sensitivity to those purposes, a cognitive capacity to recognize and exploit a subtle flaw in Kasparov's game, that explains Deep Blue's success. Feng-hsiung Hsu, the principle designer of Deep Blue, didn't beat Kasparov; Deep Blue did. Hsu didn't discover the winning sequence of moves, Deep Blue did. At one point, while Kasparov was mounting a ferocious attack on Deep Blue's king, it was nobody but Deep Blue that figured out that it had the time and security it needed to knock off a pesky pawn of Kasparov's that was out of the action, but almost invisibly vulnerable. Hsu, like the human grand masters watching the game, would never have dared consider such a calm mopping-up operation under such pressure.

Deep Blue, like many other computers equipped with AI programs, is what I call an intentional system: its behavior is predictable and explainable by attributing to it beliefs and desires--"cognitive states" and "motivational states"--and the rationality required to figure out what it ought to do in the light of those beliefs and desires (Dennett, 1971, 1987) Are these skeletal versions of human beliefs and desires sufficient to meet the mens rea requirement of legal culpability? Not quite, but it is hard to see what is missing if we restrict our gaze to the limited world of the chess board. Since cheating is literally unthinkable to a chess playing computer such as Deep Blue, and since there are really no other culpable actions available to an agent restricted to playing chess, there are no ready examples of cases where we might want to blame a chess playing computer for anything, let alone convict it of a crime. But we also assign responsibility to agents in order to praise or honor the appropriate agent. Who or what deserves the credit for beating Kasparov? Deep Blue is clearly the best candidate. Yes, we may join in congratulating Feng-hsiung Hsu and the IBM team on the success of their handiwork, but in the same spirit we might congratulate Kasparov's teachers, handlers, and even his parents. But no matter how assiduously they may have trained him, drumming into his head the importance of one strategic principle or another, they didn't beat Deep Blue in the series; Kasparov did.

Deep Blue is the best candidate for the role of responsible opponent of Kasparov, but this is not good enough, surely, for full moral responsibility. If we expanded Deep Blue's horizons somewhat, it could move out into the arenas of injury and benefit that we human beings operate in. It's not hard to imagine a touching scenario in which a grand master deliberately (but oh so subtly) throws a game to an opponent, in order to save a life, or to avoid humiliating a loved one, or to keep a promise, or . . . . (make up your own O'Henry story here). Failure to rise to such an occasion might well be grounds for blaming a human chess player. Winning or throwing a chess match might even amount to the commission of a heinous crime (make up your own Agatha Christie story here). Could Deep Blue's horizons be so widened? (For more on this theme, see Haugeland 1993)

Deep Blue is an intentional system, with beliefs and desires about its activities and predicaments on the chess board, but in order to expand its horizons to the wider world of which chess is a relatively trivial part, it would have to be given vastly richer sources of "perceptual" input--and the means of coping with this barrage in real time. Time pressure is of course already a familiar feature of Deep Blue's world. As it hustles through the multi-dimensional search tree of chess, it has to "keep one eye" on the clock, but the problems of optimizing its use of time would increase by orders of magnitude when it had to juggle all these new concurrent projects (of simple perception and self-maintenance in the world, to say nothing of more devious schemes and opportunities). For this hugely expanded task of resource management, it would need extra layers of control--above and below its chess playing software. Below, it would need to be "innately" equipped with a set of rigid traffic control policies embedded in its underlying operating system, just to keep its perceptuo-locomotor projects in basic coordination. Above, it would have to be able to pay more attention to features of its own expanded resources, always on the lookout for inefficient habits of thought, strange loops (Hofstadter, 1979), obsessive ruts, oversights, and dead-ends. It would have to become a higher-order intentional system, in other words, capable of framing beliefs about its own beliefs, desires about its desires, beliefs about its fears about its thoughts about its hopes, . . .

Higher-order intentionality is a necessary condition for moral responsibility (Dennett, 1976), and Deep Blue exhibits little sign of such capabilities. There is of course some self-monitoring implicated in any well-controlled search: Deep Blue doesn't make the mistake of re-exploring branches it has already explored, for instance, but this is innate policy designed into the underlying computational architecture, not anything under flexible control. Deep Blue can't converse with you--or with itself--about the themes discernible in its own play; it's not equipped (so far as I know--David, correct me if you know better; I haven't found a good source of detailed information on the structure and capabilities of Deep Blue!) to notice--and analyze, criticize, analyze, manipulate--the fundamental parameters that determine its policies of heuristic search or evaluation. Adding the layers of software that would permit Deep Blue to become self-monitoring and self-critical, and hence teachable, in all these ways would dwarf the already huge Deep Blue programming project--and turn Deep Blue into a radically different sort of agent.

HAL purports to be just such a higher-order intentional system--and he even plays a game of chess with Dave. HAL is an enhancement of Deep Blue equipped with eyes and ears, and a large array of sensors and effectors distributed around in Discovery One, the space ship. A familiar factoid about HAL is the curious observation that its name is IBM-minus-one--just roll back each letter one place in the alphabet. According to Arthur C. Clarke's book (p.92), however, "HAL" stands for "Heuristically programmed ALgorithmic computer," a feature Hal and Deep Blue have in common in any case. HAL is not at all garrulous or self-absorbed, but in his few speeches he expresses an interesting variety of higher-order intentional states, from the most simple to the most devious:

"Yes, it's puzzling. I don't think I've ever seen anything quite like this before."

HAL doesn't just respond to novelty with a novel reaction; he notices that he is encountering novelty, a feat that requires his memory to have an organization far beyond that required for simple conditioning to novel stimuli. (See Dennett, 1993, 1996, for more on the differences between conditioning or "ABC" learning and its fancier descendants.)

"I can't rid myself of the suspicion that there are some extremely odd things about this mission."

"I never gave these stories much credence, but particularly in view of some of the other things that have happened, I find them difficult to put out of my mind."

HAL has problems of resource management not unlike our own. Obtrusive thoughts can get in the way of other activities. The price you pay for adding layers of flexible monitoring in order to keep better track of your own mental activities is . . . more mental activities to keep track of!

"I've still got the greatest enthusiasm and confidence in the mission. I want to help you."

Another price you pay for higher-order intentionality is the opportunity for duplicity, and it comes in two flavors: self-deception and other-deception. This layering of the mind is recognized by Friedrich Nietzsche as the key ingredient in making a moral animal, and in his overheated prose it becomes the "priestly" form of life:

For with the priests everything becomes more dangerous, not only cures and remedies, but also arrogance, revenge, acuteness, profligacy, love, lust to rule, virtue, disease--but it is only fair to add that it was on the soil of this essentially dangerous form of human existence, the priestly form, that man first became an interesting animal, that only here did the human soul in a higher sense acquire depth and become evil--and these are the two basic respects in which man has hitherto been superior to other beasts! (1887, First Essay, 6, p.33)

HAL's declaration of enthusiasm is nicely poised somewhere between sincerity and cheap, desperate, canned ploy--just like some of the most important declarations we make to each other. Does HAL mean it? Could he mean it? The cost of being the sort of being that could mean it is the chance that he might not mean it. HAL is indeed an interesting "animal."

But is HAL even remotely possible? In the book, Clarke has Dave reflect on the fact that HAL, whom he is disconnecting, "is the only conscious creature in my universe," and Clarke writes, from the omniscient-author perspective, about what it is like to be HAL:

He was only aware of the conflict that was slowly destroying his integrity--the conflict between truth, and concealment of truth. He had begun to make mistakes, although, like a neurotic who could not observe his own symptoms, he would have denied it. (p.148)

Is Clarke helping himself here to more than we should allow? Could something of HAL's description--a conscious, computer-bodied intelligent agent--be brought into existence by any history of design, construction, training, learning, and activity? The different possibilities have all been explored in familiar fiction, and can be neatly nested in order of descending "humanity."

1. The Wizard of Oz: HAL isn't a computer at all; HAL is actually an ordinary flesh-and-blood man, hiding behind a techno-facade, the ultimate homunculus, pushing buttons with ordinary fingers, pulling levers with ordinary hands, looking at internal screens and listening to internal alarm buzzers. (A variation on this theme is the busy-fingered John Searle [1980], hand-simulating the Chinese Room by following billions of instructions written on slips of paper.)

2. William (from "William and Mary," in Kiss Kiss, by Roald Dahl): HAL is a human brain, kept alive in a "vat" by a life-support system, detached from its former body, in which it acquired a lifetime of human memory, hankerings, attitudes, and so forth, and now harnessed to huge banks of prosthetic sense organs and effectors. (A variation on this theme is poor Yorick, Dennett's brain in the vat, in my story, "Where Am I?" in Dennett, 1978)

3. Robocop disembodied and living in a "vat": Rococop is part human brain, part computer. After a gruesome accident, the brain-part (vehicle of some of the memory and personal identity, one gathers, of the flesh-and-blood cop who was Robocop's youth) was re-embodied with robotic arms and legs, but also (apparently) partly replaced or enhanced with special-purpose software and computer hardware. We can imagine that HAL spent some transitional time as Robocop, before becoming a limbless agent.

4. Max Headroom: A virtual machine, a software duplicate of a real person's brain (or mind) that has somehow been created by a brilliant hacker; it has the memories and personality traits acquired in a normally embodied human lifetime, but has been off-loaded from all carbon-based hardware into a silicon-chip implementation. (A variation on this theme is poor Hubert, the software duplicate of Yorick, in "Where Am I?")

5.The real-life but still-in-the-future--and hence still strictly science-fictional--Cog, the humanoid robot being constructed by Rodney Brooks and the Cog team at MIT (Brooks and Stein, 1994, Dennett, 1994). Cog's brain is silicon chips from the outset, and all its body parts are inorganic artifacts, but it is designed to go through an embodied infancy and childhood, reacting to people it sees with its video eyes, making friends, learning about the world by playing with real things with its real hands, acquiring memory. If Cog ever grows up, it could surely abandon its body and make the transition described in the fictional cases. It would be easier for Cog, who has always been a silicon based, digitally encoded intelligence, to move into a silicon-based vat than for Max Headroom or Robocop, whose early years were spent in wetware. Many important details of Cog's degree of humanoidness (humanoidity?) have not yet been settled, but the scope is wide. For instance, it is now planned that Cog will have a virtual neuro-endocrine system, with virtual hormones spreading and dissipating through its logical spaces.

6. Blade Runner in a vat: HAL has never had a real humanoid body, but has the memory hallucinations of having had one. This entirely bogus past life has been constructed by some preposterously complex and detailed authoring process.

7. Clarke's own scenario, as best it can be extrapolated from the book and the movie: HAL has never had a body, and has no illusions about his past; what he knows of human life he knows as either part of his innate heritage (coded, one gathers, by the labors of many programmers, after the fashion of the real-world CYC project of Douglas Lenat [Lenat and Guha, 1990]) or as a result of his subsequent training--a sort of bed-ridden infancy, one gathers, in which he was both observer and, eventually, participant. (In the book, Clarke speaks of "the perfect idiomatic English he had learned during the fleeting weeks of his electronic childhood." [p14x])

The extreme cases at both poles are impossible for relatively boring reasons. At one end, neither the Wizard of Oz or John Searle could do the necessary hand-work fast enough to sustain HAL's quick-witted round of activities. At the other end, hand-coding enough world knowledge into a disembodied agent to create HAL's dazzlingly humanoid competence, getting it to the point where it could then benefit from an "electronic childhood," is a programming task to be measured in hundreds of efficiently organized person-centuries. In other words, the daunting difficulties observable at both ends of this spectrum highlight the fact that there is a colossal design job to be done, and the only practical way of doing it is one version or another of Mother Nature's way: years of embodied learning. The trade-offs between various combinations of flesh-and-blood and silicon-and-metal bodies are anybody's guess, but I am putting my bet on Cog, as the most likely developmental platform for a future HAL. (See also Waltz, 1988, for further supporting arguments.)

Notice that the requirement that HAL once have had a humanoid body and have lived concretely in the human world is only a practical requirement, not a metaphysical one. Once all the R and D had been accomplished in the prototype, by the odyssey of a single embodied agent, the standard duplicating techniques of the computer industry could clone HALs by the thousands, as readily as compact disks. The finished product could thus be captured in some number of terabytes of information, so "in principle" the information that fixes the design of all those chips and hard-wired connections, and configures all the RAM and ROM, could be created by hand. There is no finite bit-string, however long, that is officially off-limits to human authorship. In principle, then, Blade-Runner-like entities could be created with ersatz biographies. They would have exactly the capabilities, dispositions, strengths and weaknesses of a being whose biography had been real, not virtual, so whatever moral standing the latter deserved should belong to the former as well.

The main point of giving HAL a humanoid past is to give him the world knowledge required to be a moral agent--a modicum of understanding or empathy about the human condition. A modicum will do nicely; we don't want to hold out for too much commonality of experience. After all, among the people we know are many who have moral responsibility in spite of their obtuseness when it comes to imagining themselves into the predicaments of others. We certainly don't exculpate male chauvinist pigs who can't see women as people!

When do we exculpate people? We should look carefully at the answers to this question, because HAL shows signs of fitting into one or another of the excusing conditions in spite of his being a conscious agent.

We exculpate people when they are insane. Might HAL have gone insane? The question of HAL's capacity for emotion--and hence vulnerability to emotional disorder--is tantalizingly raised by Frank's answer to Mr. Amer:

Well, he acts like he has genuine emotions. Of course, he's programmed that way, to make it easier for us to talk to him. But as to whether he has real feelings is something I don't think anyone can truthfully answer.

Certainly HAL proclaims his emotional state at the end: "I'm afraid. I'm afraid." HAL is "programmed that way"--but what does that mean? It could mean that HAL's verbal capacity is "enhanced" with lots of canned expressions of emotional response that get grafted into his discourse at pragmatically appropriate opportunities (and of course many of our avowals of emotion are like that--insincere moments of socially lubricating ceremony), or it could mean that HAL's underlying computational architecture has been provided, as Cog's will be, with virtual emotional states, powerful attention-shifters, galvanizers, prioritizers, and the like--realized not in neuromodulator and hormone molecules actually floating in a bodily fluid, but in global variables modulating dozens of concurrent processes, while dissipating on some timetable (or something much more complex).

In the latter, more interesting, case, "I don't think anyone can truthfully answer" the question of whether HAL has emotions. He has something very much like emotions--enough like emotions, one may imagine, to mimic the pathologies of human emotional breakdown. Whether that is enough to call them real emotions, well, who's to say? There are good reasons for HAL to have such states in any case, since their role in enabling real-time practical thinking has recently been dramatically revealed by experiments involving human beings with brain damage (Damasio, 1994). This would be a profound difference between HAL and Deep Blue, by the way. Deep Blue, basking in the strictly limited search space of chess, can handle its real time decision-making without any emotional crutches. In Time Magazine's story [Feb 26, p.61] on the Kasparov match, the grand master Yasser Seirawan is quoted as saying "The machine has no fear" and the story goes on to note that expert commentators responded to some of Deep Blue's moves (the icily calm pawn capture described earlier) as taking "crazy chances" and "insane." In the tight world of chess, it appears, the very imperturbability that cripples the brain-damaged human decision-makers Damasio describes can be a blessing--but only if you have the brute force analytic speed of a Deep Blue.

HAL may then have suffered from some emotional imbalance of much the same sort as those that leads human beings astray. Whether this was the result of some sudden trauma--a blown fuse, a dislodged connector, a microchip disordered by cosmic rays--or of some gradual drift into emotional misalignment provoked by the stresses of the mission, confirming such a diagnosis should justify a verdict of diminished responsibility for HAL, just as it does in human malfeasance.

Another possible source of exculpation, more familiar in fiction than in the real world, is "brainwashing" or hypnosis.

(The Manchurian Candidate is a standard model: the prisoner of war who is turned by evil scientists into a walking time bomb, returned to his homeland to assassinate the President.) The closest real world cases are probably the "programmed" and subsequently "deprogrammed" members of cults. Is HAL like a cult member? It's hard to say. According to Clarke, HAL was "trained for his mission," not just programmed for his mission. At what point does benign, responsibility-enhancing training turn into malign, responsibility-diminishing brainwashing--in the human case? The intuitive turning point is captured, I think, in the answer to the question of whether the agent can still "think for himself" after the period of indoctrination. And what is it to be able to think for yourself? You must be capable of being "moved by reasons"--you must be reasonable, accessible to rational persuasion, the introduction of new evidence and further considerations. If you are more or less impervious to experiences that ought to influence you, your capacity has been diminished.

The only evidence that HAL might be in such a partially disabled state is the much-remarked fact that he has actually made a mistake, and the series 9000 computer is supposedly utterly invulnerable to error. This is, to my mind, the weakest point in Clarke's science fiction. The suggestion that a computer could be both a "Heuristically programmed ALgorithmic" computer and "by any practical definition of the words, fool-proof and incapable of error" verges on self-contradiction. The whole point of heuristic programming is that it defies the problem of combinatorial explosion (which mathematically cannot be solved by sheer increase in computing speed and size) by taking risky chances, truncating its searches in ways that must leave it open to error, however low the probability. The saving clause, "by any practical definition of the words," restores sanity: HAL may indeed be ultra-reliable without being literally fool-proof, a fact whose importance Alan Turing pointed out fifty years ago at the dawn of the computer age (thereby "prefuting" Endnote 1 Roger Penrose's more recent criticisms (1989) of Artificial Intelligence (see Dennett, 1995, chapter 15, for the details).):

In other words then, if a machine is expected to be infallible, it cannot also be intelligent. There are several theorems which say almost exactly that. But these theorems say nothing about how much intelligence may be displayed if a machine makes no pretence at infallibility. (Turing, 1946, p.124)

 This is just the opposite of the other conditions; it is precisely because the agent is rational, and is faced with an overwhelmingly good reason for perform an injurious deed--to kill in self-defense, in the clearest case--that the agent is excused or at least partly exonerated. These are the forced moves of life: all alternatives to them are suicidal--and that is too much to ask, isn't it?

Well is it? We sometimes call upon people to sacrifice their lives, and blame them for failure to do so, but that is typically not seen to be murder. If I could prevent your death, but out of fear for my own life I let you die, that is not murder. If HAL were brought into court and I were called upon to defend him, I would argue that Frank's decision to disable HAL was a morally loaded one, but it wasn't murder--it was assault: rendering Hal indefinitely comatose against his will. Those memory boxes were not smashed, just removed to a place where HAL could not retrieve them.

If HAL couldn't comprehend this, then this might be excusable ignorance; we might blame his trainers for not briefing him sufficiently about the existence and reversibility of the comatose state. In the book, Clarke looks into HAL's mind and says "He had been threatened with disconnection; he would be deprived of all his inputs, and thrown into an unimaginable state of unconsciousness." [p.148] That might be grounds enough to justify HAL's course of self-defense, but there is one final theme for counsel to present to the jury:

If HAL believed (we can't be sure on what grounds) that his being so rendered comatose would jeopardize the whole mission, then he would be in exactly the same moral dilemma a human being in the same predicament would face. Not surprisingly, we figure out the answer to our question by figuring out what would be true if we put ourselves in Hal's place. If you believed the mission to which your life was devoted was more important, in the last analysis, than anything else, what would you do?

"So he would protect himself, with all the weapons at his command. Without rancor--but without pity--he would remove the source of his frustrations. And then, following the orders that had been given to him in case of the ultimate emergency, he would continue the mission--unhindered, and alone." [p149]

Brooks, Rodney, and Stein, Lynn Andrea, 1994, "Building Brains for Bodies," Autonomous Robots, 1, pp.7-25.

 Damasio, Antonio, 1994, Descartes' Error: Emotion, Reason, and the Human Brain, New York: Grosset/Putnam.

 Dennett, Daniel , 1971, "Intentional Systems," Journal of Philosophy, LXVIII, 87-106.

----1976, "Conditions of Personhood," in A. Rorty, (ed.), The Identities of Persons, Berkeley: University of California Press, pp.175-196.

 ----1978, Brainstorms: Philosophical Essays on Mind and Psychology, Montgomery, VT: Bradford Books and Hassocks, Sussex: Harvester.

 ---1987, The Intentional Stance, 1987, Cambridge, MA: Bradford Books/MIT Press.

 ----1993, "Learning and Labeling" (commentary on A. Clark and A. Karmiloff-Smith, "The Cognizer's Innards"), Mind and Language, 8, (4), pp540-547.

 ---1994, "The Practical Requirements for Making a Conscious Robot" Philosophical Transactions of the Royal Society, A, 349, pp.133-46.

 ----1995, Darwin's Dangerous Idea, New York: Simon & Schuster.

 ----1996, Kinds of Minds, New York: Basic Books.

 Haugeland, John, 1993, "Pattern and Being," in Bo Dahlbom, ed., Dennett and his Critics, Oxford: Blackwell, pp.53-69.

 Hofstadter, Douglas R., 1979, Gödel Escher Bach: an Eternal Golden Braid, New York: Basic Books.

 Lenat, Douglas, and Guha, R. V., 1990, Building Large Knowledge-Based Systems: Representation and Inference in the CYC Project, Reading,MA: Addison-Wesley.

 Nietzsche, Friedrich, 1887, On the Genealogy of Morals, (translated by Walter Kaufmann, Vintage Books, 1967).

 Penrose, Roger, 1989, The Emperor's New Mind: Concerning Computers, Minds, and the Laws of Physics, Oxford: Oxford Univ. Press.

 Searle, John, 1980, "Minds, Brains and Programs," Behavioral and Brain Sciences, 3, pp.417-58.

 Turing, Alan, 1946, ACE Reports of 1946 and Other Papers, Ed. B. E. Carpenter and R. W. Doran, Cambridge, MA:MIT Press.

 Waltz, David, 1988, "The Prospects for Building Truly Intelligent Machines," Daedalus, 117, pp.191-212.

 Young, Andrew, 1994, "The Neuropsychology of Awareness," in Revonsuo, A. and Kamppinen, M., Consciousness in Philosophy and Cognitive Neuroscience, Erlbaum, pp.173-203

1. The verb "prefute" was coined in 1990, inspired by the endearing tendency of the psychologist, Tony Marcel, to interrupt conference talks by leaping to his feet and exclaiming "I can see where your argument is heading and here is what is wrong with what you're going to say . . . ." Marcel is the master of prefutation, but he is not the only practitioner.