.htmbrp {color: #333333; text-decoration: none; font-family: Verdana, Arial, Helv, Helvetica, sans-serif; font-size : 11px;}.htmbrp:link {color: #0000ff; text-decoration: none; font-family: Verdana, Arial, Helv, Helvetica, sans-serif}.htmbrp:visited {color: #800080;}.htmbrp:hover, .htmbrp:active {text-decoration: underline; }.htdirp { font-size : 12px; font-family: Verdana, Arial, Helv, Helvetica, sans-serif; color: #333333;}.htdira:link { color: #3333CC; }.htdira:visited { color: #990099; }.htdira:hover { color: #0066FF; }.htdirh1 { font-size : 20px; font-family: Verdana, Arial, Helv, Helvetica, sans-serif; color: #333333; font-weight: bold}/* CSS for aol hat - to hide nav use header_nonav.css or use header_nohat.css to hide completely */div#aol_hat {min-width: 728px; font-size:11px; font-weight:bold; z-index:200; height:30px; margin-bottom:8px; background-color:#CFD9E3;}div#aol_hat form {margin:0; }div#aol_hat ul {float:left; padding:4px 0 0 2px; height:16px; margin:0;}div#aol_hat ul li {padding:0 5px 0 0 !important; border-right:solid 1px #2864B4; line-height:11px; display:inline;}div#aol_hat ul li a {height:12px; display:inline; padding-left:5px; }div#aol_hat_form {padding-top:7px; _padding-top:3px; font-weight:normal; font-size:10px;}div#aol_hat_form input.aol_hat_button {font-size:11px; margin-top:-2px; _margin-top:0px;}div#aol_hat_form input.aol_hat_search {margin-top:-2px; _margin-top:0px; }div#aol_hat ul {padding-top:9px;}#aol_hat ul li.aol_hat_last {border:none;}#aol_hat a {color:#2864B4; background-image:none; text-decoration:none;}#aol_hat a:hover {text-decoration: underline;}#aol_hat div div {height:20px; }#aol_hat_sns div {display:inline;}#aol_hat_sns {width:auto; float:right; height:16px; text-align:right; margin:0 8px 0 0; padding:8px 0 0 0; _padding:7px 0 0 0;}#aol_hat_sns a {text-decoration:none;}#aol_hat_form {width: 300px; float:left; padding:0 0 0 12px; color:#666666;}#aol_hat input.aol_hat_search {width: 118px; height:21px; border:solid 1px #56ba24; vertical-align: middle; margin-right: 2px;}#aol_hat input.aol_hat_button {-moz-border-radius:2.5px; border-radius:2.5px; cursor:pointer; width:59px; height:21px; border:solid 1px #58B926; background-color: #D4F4C5; filter:progid:DXImageTransform.Microsoft.Gradient(gradientType=0,startColorStr=#FBFEF9,endColorStr=#BDEEA6); vertical-align: middle; font: 1.0em Tahoma,Arial,Sans-Serif;}#aol_hat input.aol_hat_button:hover {background-color:#83E839; filter:progid:filter:progid:DXImageTransform.Microsoft.Gradient(gradientType=0,startColorStr=#F9FEF7,endColorStr=#7BED46);}#aol_hat input.aol_hat_button:active {background-color:#C6FFA9; filter:progid:DXImageTransform.Microsoft.Gradient(gradientType=0,startColorStr=#99E576,endColorStr=#F5FCF1);}#content {width:auto;}Minds, Brains, and Programs if (navigator.userAgent.indexOf("AOL") == -1) { var sitedomain="hometown"; var siteState = "OrigUrl=" + location.href; var _sns_hostname_="my.screenname.aol.com"; // var _sns_hostname_ = "tweb33.web.aol.com"; // var _sns_hostport_ = "8000"; // var _sns_hostport_ssl_="8443"; document.write(''); _109778(); document.write('');} htmlAdWH('93212816', '728', '90');  Main _10423(); _11385(); Minds, Brains, and Programs Minds, Brains, and Programs John R. Searle ["Minds, Brains, and Programs," by John R. Searle, from The Behavioral and Brain Sciences, vol. 3. Copyright 1980 Cambridge University Press. Reprinted by permission of Cambridge University Press.]   What psychological and philosophical significance should we attach to recentefforts at computer simulations of human cognitive capacities? In answeringthis question, I find it useful to distinguish what I will call "strong"AI from "weak" or "cautious" AI (artificial intelligence). According to weakAI, the principal value of the computer in the study of the mind is thatit gives us a very powerful tool. For example, it enables us to formulateand test hypotheses in a more rigorous and precise fashion. But accordingto strong AI, the computer is not merely a tool in the study of the mind;rather, the appropriately programmed computer really is a mind, in the sensethat computers given the right programs can be literally said tounderstand and have other cognitive states. In strong AI, becausethe programmed computer has cognitive states, the programs are not mere toolsthat enable us to test psychological explanations; rather, the programs arethemselves the explanations.I have no objection to the claims of weak AI, at least as far as this articleis concerned. My discussion here will be directed at the claims I have definedas those of strong AI, specifically the claim that the appropriately programmedcomputer literally has cognitive states and that the programs thereby explainhuman cognition. When I hereafter refer to AI, I have in mind the strongversion, as expressed by these two claims.I will consider the work of Roger Schank and his colleagues at Yale (Schankand Abelson 1977), because I am more familiar with it than I am with anyother similar claims, and because it provides a very clear exampie of thesort of work I wish to examine. But nothing that follows depends upon thedetails of Schank’s programs. The same arguments would apply toWinograd’s SHRDLU (Winograd 1973), Weizenbaum’s ELIZA (Weizenbaum1965), and indeed any Turing machine simulation of human mental phenomena.[See "Further Reading" for Searle’s references.]Very briefly, and leaving out the various details, one can describeSchank’s program as follows: The aim of the program is to simulate thehuman ability to understand stories. It is characteristic of human beings’story-understanding capacity that they can answer questions about the storyeven though the information that they give was never explicitly stated inthe story. Thus, for example, suppose you are given the following story:"A man went into a restaurant and ordered a hamburger. When the hamburgerarrived it was burned to a crisp, and the man stormed out of the restaurantangrily, without paying for the hamburger or leaving a tip." Now, if youare asked "Did the man eat the hamburger?" you will presumably answer, "No,he did not." Similarly, if you are given the following story: "A man wentinto a restaurant and ordered a hamburger; when the hamburger came he wasvery pleased with it; and as he left the restaurant he gave the waitressa large tip before paying his bill," and you are asked the question, "Didthe man eat the hamburger?" you will presumably answer, "Yes, he ate thehamburger." Now Schank’s machines call similarly answer, questions aboutrestaurants in this fashion. To do this, they have a "representation" ofthe sort of information that human beings have about restaurants, which enablesthem to answer such questions as those above, given these sorts of stories.When the machine is given the story and then asked the question, the machinewill print out answers of the sort that we would expect human beings to giveif told similar stories. Partisans of strong AI claim that in this questionand answer sequence the machine is not only simulating a human ability butalso (1) that the machine can literally be said to understand thestory and provide the answers to questions, and (2) that what the machineand its program do explains the human ability to understand the storyand answer questions about it.Both claims seem to me to be totally unsupported by Schank’s work, asI will attempt to show in what follows. I am not, of course, saying thatSchank himself is to these claims.One way to test any theory of the mind is to ask oneself what it would belike if my mind actually worked on the principles that the theory says allminds work on. Let us apply this test to the Schank program with the followingGedankenexperiment. Suppose that I’m locked in a room and givena large batch of Chinese writing. Suppose furthermore (as is indeed the case)that I know no Chinese, either written or spoken, and that I’m not evenconfident that I could recognize Chinese writing as Chinese writing distinctfrom, say, Japanese writing or meaningless squiggles. To me, Chinese writingis just so many meaningless squiggles. Now suppose further that after thisfirst batch of Chinese writing I am given a second batch of Chinese scripttogether with a set of rules for correlating the second batch with the firstbatch. The rules are in English, and I understand these rules as well asany other native speaker of English. They enable me to correlate one setof formal symbols with another set of formal symbols, and all that "formal"means here is that I can identify the symbols entirely by their shapes. Nowsuppose also that I am given a third batch of Chinese symbols together withsome instructions, again in English, that enable me to correlate elementsof this third batch with the first two batches, and these rules instructme how to give back certain Chinese symbols with certain sorts of shapesin response to certain sorts of shapes given me in the third batch. Unknownto me, the people who are giving me all of these symbols call the first batcha "script," they call the second batch a "story," and they call the thirdbatch "questions." Furthermore, they call the symbols I give them back inresponse to the third batch "answers to the questions," and the set of rulesin English that they gave me, they call the "program." Now just to complicatethe story a little, imagine that these people also give me stories in English,which I understand, and they then ask me questions in English about thesestories, and I give them back answers in English. Suppose also that aftera while I get so good at following the instructions for manipulating theChinese symbols and the programmers get so good at writing the programs thatfrom the external point of view—that is, from tile point of view ofsomebody outside the room in which I am locked—my answers to the questionsare absolutely indistinguishable from those of native Chinese speakers. Nobodyjust looking at my answers can tell that I don’t speak a word of Chinese.Let us also suppose that my answers to the English questions are, as theyno doubt would be, indistinguishable from those of other native English speakers,for the simple reason that I am a native English speaker. From the externalpoint of view—from the point of view of someone reading my"answers"—the answers to the Chinese questions and the English questionsare equally good. But in the Chinese case, unlike the English case, I producethe answers by manipulating uninterpreted formal symbols. As far as the Chineseis concerned, I simply behave like a computer; I perform computational operationson formally specified elements. For the purposes of the Chinese, I am simplyan instantiation of the computer program.Now the claims made by strong AI are that the programmed computer understandsthe stories and that the program in some sense explains human understanding.But we are now in a position to examine these claims in light of our thoughtexperiment.1. As regards the first claim, it seems to me quite obvious in the examplethat I do not understand a word of the Chinese stories. I have inputs andoutputs that are indistinguishable from those of the native Chinese speaker,and I can have any formal program you like, but I still understand nothing.For the same reasons, Schank’s computer understands nothing of any stories,whether in Chinese, English, or whatever, since in the Chinese case the computeris me, and in cases where the computer is not me, the computer has nothingmore than I have in the case where I understand nothing.2. As regards the second claim, that the program explains human understanding,we can see that the computer and its program do not provide sufficient conditionsof understanding since the computer and the program are functioning, andthere is no understanding. But does it even provide a necessary conditionor a significant contribution to understanding? One of the claims made bythe supporters of strong AI is that when I understand a story in English,what I am doing is exactly the same—or perhaps more of the same—aswhat I was doing in manipulating the Chinese symbols. It is simply more formalsymbol manipulation that distinguishes the case in English, where I dounderstand, from the case in Chinese, where I don’t. I have not demonstratedthat this claim is false, but it would certainly appear an incredible claimin the example. Such plausibility as the claim has derives from the suppositionthat we can construct a program that will have the same inputs and outputsas native speakers, and in addition we assume that speakers have some levelof description where they are also instantiations of a program. On the basisof these two assumptions we assume that even if Schank’s program isn’tthe whole story about understanding, it may be part of the story. Well, Isuppose that is an empirical possibility, but not the slightest reason hasso far been given to believe that it is true, since what issuggested—though certainly not demonstrated—by the example is thatthe computer program is simply irrelevant to my understanding of the story.In the Chinese case I have everything that artificial intelligence can putinto me by way of a program, and I understand nothing; in the English caseI understand everything, and there is so far no reason at all to supposethat my understanding has anything to do with computer programs, that is,with computational operations on purely formally specified elements. As longas the program is defined in terms of computational operations on purelyformally defined elements, what the example suggests is that these by themselveshave no interesting connection with understanding. They are certainly notsufficient conditions, and not the slightest reason has been given to supposethat they are necessary conditions or even that they make a significantcontribution to understanding. Notice that the force of the argument is notsimply that different machines can have the same input and output while operatingon different formal principles—that is not the point at all. Rather,whatever purely formal principles you put into the computer, they will notbe sufficient for understanding, since a human will be able to follow theformal principles without understanding anything. No reason whatever hasbeen offered to suppose that such principles are necessary or even contributory,since no reason has been given to suppose that when I understand EnglishI am operating with any formal program at all.Well, then, what is it that I have in the case of the English sentences thatI do not have in the case of the Chinese sentences? The obvious answer isthat I know what the former mean, while I haven’t the faintest ideawhat the latter mean. But in what does this consist and why couldn’twe give it to a machine, whatever it is? I will return to this question later,but first I want to continue with the example.I have had the occasions to present this example to several workers in artificialintelligence, and, interestingly, they do not seem to agree on what the properreply to it is. I get a surprising variety of replies, and in what followsI will consider the most common of these (specified along with their geographicorigins).But first I want to block some common misunderstandings about "understanding":In many of these discussions one finds a lot of fancy footwork about theword "understanding." My critics point out that there are many differentdegrees of understanding; that "understanding" is not a simple two-placepredicate; that there are even different kinds and levels of understanding,and often the law of excluded middle doesn’t even apply in a straightforwardway to statements of the form "x understands y"; that in manycases it is a matter for decision and not a simple matter of fact whetherx understands y; and so on. To all of these points I want tosay: of course, of course. But they have nothing to do with the points atissue. There are clear cases in which "understanding" literally applies andclear cases in which it does not apply; and these two sorts of cases areall I need for this argument.1 I understand stories in English;to a lesser degree I can understand stories in French; to a still lesserdegree, stories in German; and in Chinese, not at all. My car and my addingmachine, on the other hand, understand nothing: they are not in that lineof business. We often attribute "understanding" and other cognitive predicatesby metaphor and analogy to cars, adding machines, and other artifacts, butnothing is proved by such attributions. We say, "The door knows whento open because of its photoelectric cell," "The adding machine knowshow (understands how, is able) to do addition and subtractionbut not division," and "The thermostat perceives changes in thetemperature." The reason we make these attributions is quite interesting,and it has to do with the fact that in artifacts we extend our ownintentionality;2 our tools are extensions of our purposes, andso we find it natural to make metaphorical attributions of intentionalityto them; but I take it no philosophical ice is cut by such examples. Thesense in which an automatic door "understands instructions" from itsphotoelectric cell is not at all the sense in which I understand English.If the sense in which Schank’s programmed computers understand storiesis supposed to be the metaphorical sense in which the door understands, andnot the sense in which I understand English, the issue would not be worthdiscussing. But Newell and Simon (1963) write that the kind of cognitionthey claim for computers is exactly the same as for human beings. I likethe straightforwardness of this claim, and it is the sort of claim I willbe considering. I will argue that in the literal sense the programmed computerunderstands what the car and the adding machine understand, namely, exactlynothing. The computer understanding is not just (like my understanding ofGerman) partial or incomplete; it is zero.Now to the replies:1. The Systems Reply (Berkeley). "While it is true that the individualperson who is locked in the room does not understand the story, the factis that he is merely part of a whole system, and the system does understandthe story. The person has a large ledger in front of him in which are writtenthe rules, he has a lot of scratch paper and pencils for doing calculations,he has ‘data banks’ of sets of Chinese symbols. Now, understandingis not being ascribed to the mere individual; rather it is being ascribedto this whole system of which he is a part."My response to the systems theory is quite simple: Let the individual internalizeall of these elements of the system. He memorizes the rules in the ledgerand the data banks of Chinese symbols, and he does all the calculations inhis head. The individual then incorporates the entire system. There isn’tanything at all to the system that he does not encompass. We can even getrid of the room and suppose he works outdoors. All the same, he understandsnothing of the Chinese, and a fortiori neither does the system, because thereisn’t anything in the system that isn’t in him. If he doesn’tunderstand, then there is no way the system could understand because thesystem is just a part of him.Actually I feel somewhat embarrassed to give even this answer to the systemstheory because the theory seems to me so implausible to start with. The ideais that while a person doesn’t understand Chinese, somehow theconjunction of that person and bits of paper might understand Chinese.It is not easy for me to imagine how someone who was not in the grip of anideology would find the idea at all plausible. Still, I think many peoplewho are committed to the ideology of strong AI will in the end be inclinedto say something very much like this; so let us pursue it a bit further.According to one version of this view, while the man in the internalizedsystems example doesn’t understand Chinese in the sense that a nativeChinese speaker does (because, for example, he doesn’t know that thestory refers to restaurants and hamburgers, etc.), still "the man as a formalsymbol manipulation system" really does understand Chinese. The subsystemof the man that is the formal symbol manipulation system for Chinese shouldnot be confused with the subsystem for English.So there are really two subsystems in the man; one understands English, theother Chinese, and "it’s just that the two systems have little to dowith each other." But, I want to reply, not only do they have little to dowith each other, they are not even remotely alike. The subsystem that understandsEnglish (assuming we allow ourselves to talk in this jargon of "subsystems"for a moment) knows that the stories are about restaurants and eating hamburgers,he knows that he is being asked questions about restaurants and that he isanswering questions as best he can by making various inferences from thecontent of the story, and so on. But the Chinese system knows none of this.Whereas the English subsystem knows that "hamburgers" refers to hamburgers,the Chinese subsystem knows only that "squiggle squiggle" is followed by"squoggle squoggle." All he knows is that various formal symbols are beingintroduced at one end and manipulated according to rules written in English,and other symbols are going out at the other end. The whole point of theoriginal example was to argue that such symbol manipulation by itselfcouldn’t be sufficient for understanding Chinese in any literal sensebecause the man could write "squoggle squoggle" after "squiggle squiggle"without understanding anything in Chinese. And it doesn't meet that argumentto postulate subsystems within the man, because the subsystems are no betteroff than the man was in the first place; they still don't have anything evenremotely like what the English-speaking man (or subsystem) has. Indeed, inthe case as described, the Chinese subsystem is simply a part of the Englishsubsystem, a part that engages in meaningless symbol manipulation accordingto rules in English.Let us ask ourselves what is supposed to motivate the systems reply in thefirst place; that is, what independent grounds are there supposedto be for saying that the agent must have a subsystem within him that literallyunderstands stories in Chinese? As far as I can tell the only grounds arethat in the example I have the same input and output as native Chinese speakersand a program that goes from one to the other. But the whole point of theexamples has been to try to show that that couldn't be sufficient forunderstanding, in the sense in which I understand stories in English, becausea person, and hence the set of systems that go to make up a person, couldhave the right combination of input, output, and program and still not understandanything in the relevant literal sense in which I understand English. Theonly motivation for saying there must be a subsystem in me thatunderstands Chinese is that I have a program and I can pass the Turing test;I can fool native Chinese speakers. But precisely one of the points at issueis the adequacy of the Turing test. The example shows that there could betwo "systems," both of which pass the Turing test, but only one of whichunderstands; and it is no argument against this point to say that since theyboth pass the Turing test they must both understand, since this claim failsto meet the argument that the system in me that understands English has agreat deal more than the system that merely processes Chinese. In short,the systems reply simply begs the question by insisting without argumentthat the system must understand Chinese.Furthermore, the systems reply would appear to lead to consequences thatare independently absurd. If we are to conclude that there must be cognitionin me on the grounds that I have a certain sort of input and output and aprogram in between, then it looks like all sorts of noncognitive subsystemsare going to turn out to be cognitive. For example, there is a level ofdescription at which my stomach does information processing, and it instantiatesany number of computer programs, but I take it we do not want to say thatit has any understanding (cf. Pylyshyn 1980). But if we accept the systemsreply, then it is hard to see how we avoid saying that stomach, heart, liver,and so on are all understanding subsystems, since there is no principle wayto distinguish the motivation for saying the Chinese subsystem understandsfrom saying that the stomach understands. It is, by the way, not an answerto this point to say that the Chinese system has information as input andoutput and the stomach has food and food products as input and output, sincefrom the point of view of the agent, from my point of view, there is noinformation in either the food or the Chinese—the Chinese is just somany meaningless squiggles. The information in the Chinese case is solelyin the eyes of the programmers and the interpreters, and there is nothingto prevent them from treating the input and output of my digestive organsas information if they so desire.This last point bears on some independent problems in strong AI, and it isworth digressing for a moment to explain it. If strong AI is to be a branchof psychology, then it must be able to distinguish those systems that aregenuinely mental from those that are not. It must be able to distinguishthe principles on which the mind works from those on which nonmental systemswork; otherwise it will offer us no explanations of what is specificallymental about the mental. And the mental-nonmental distinction cannot be justin the eye of the beholder but it must be intrinsic to the systems; otherwiseit would be up to any beholder to treat people as nonmental and, for example,hurricanes as mental if he likes. But quite often in the AI literature thedistinction is blurred in ways that would in the long run prove disastrousto the claim that AI is a cognitive inquiry. McCarthy, for example, writes."Machines as simple as thermostats can be said to have beliefs, and havingbeliefs seems to be a characteristic of most machines capable of problemsolving performance" (McCarthy 1979). Anyone who thinks strong AI has a chanceas a theory of the mind ought to ponder the implications of that remark.We are asked to accept it as a discovery of strong AI that the hunk of metalon the wall that we use to regulate the temperature has beliefs in exactlythe same sense that we, our spouses, and our children have beliefs, andfurthermore that "most" of the other machines in the room—telephone,tape recorder, adding machine, electric fight switch—also have beliefsin this literal sense. It is not the aim of this article to argue againstMcCarthy's point, so I will simply assert the following without argument.The study of the mind starts with such facts as that humans have beliefs,while thermostats, telephones, and adding machines don't. If you get a theorythat denies this point you have produced a counterexample to the theory andthe theory is false. One gets the impression that people in AI who writethis sort of thing think they can get away with it because they don't reallytake it seriously, and they don't think anyone else will either. I propose,for a moment at least, to take it seriously. Think hard for one minute aboutwhat would be necessary to establish that that hunk of metal on the wallover there had real beliefs, beliefs with direction of fit, propositionalcontent, and conditions of satisfaction; beliefs that had the possibilityof being strong beliefs or weak beliefs; nervous, anxious, or secure beliefs;dogmatic, rational, or superstitious beliefs; blind faiths or hesitantcogitations; any kind of beliefs. The thermostat is not a candidate. Neitheris stomach, liver, adding machine, or telephone. However, since we are takingthe idea seriously, notice that its truth would be fatal to strong AI's claimto be a science of the mind. For now the mind is everywhere. What we wantedto know is what distinguishes the mind from thermostats and livers. And ifMcCarthy were right, strong AI wouldn't have a hope of telling us that.2. The Robot Reply (Yale). "Suppose we wrote a different kind of programfrom Schank's program. Suppose we put a computer inside a robot, and thiscomputer would not just take in formal symbols as input and give out formalsymbols as output, but rather would actually operate the robot in such away that the robot does something very much like perceiving, walking, movingabout, hammering nails, eating, drinking—anything you like. The robotwould, for example, have a television camera attached to it that enabledit to see, it would have arms and legs that enabled it to 'act,' and allof this would be controlled by its computer 'brain.' Such a robot would,unlike Schank's computer, have genuine understanding and other mental states."The first thing to notice about the robot reply is that it tacitly concedesthat cognition is not solely a matter of formal symbol manipulation, sincethis reply adds a set of causal relations with the outside world (cf. Fodor1980). But the answer to the robot reply is that the addition of such"perceptual" and "motor" capacities adds nothing by way of understanding,in particular, or intentionality, in general, to Schank's original program.To see this, notice that the same thought experiment applies to the robotcase. Suppose that instead of the computer inside the robot, you put me insidethe room and, as in the original Chinese case, you give me more Chinese symbolswith more instructions in English for matching Chinese symbols to Chinesesymbols and feeding back Chinese symbols to the outside. Suppose, unknownto me, some of the Chinese symbols that come to me come from a televisioncamera attached to the robot and other Chinese symbols that I am giving outserve to make the motors inside the robot move the robot's legs or arms.It is important to emphasize that all I am doing is manipulating formal symbols:I know none of these other facts. I am receiving "Information" from the robot'sperceptual" apparatus and I am giving out "instructions" to its motor apparatuswithout knowing either of these facts. I am the robot's homunculus, but unlikethe traditional homunculus, I don't know what's going on. I don't understandanything except the rules for symbol manipulation. Now in this case I wantto say that the robot has no intentional states at all; it is simply movingabout as a result of its electrical wiring and its program. And furthermore,by instantiating the program I have no intentional states of the relevanttype. All I do is follow formal instructions about manipulating formal symbols.3. The Brain Simulator Reply (Berkeley and M.I.T.). "Suppose we designa program that doesn't represent information that we have about the world,such as the information in Schank's scripts, but simulates the actual sequenceof neuron firings at the synapses of the brain of a native Chinese speakerwhen he understands stories in Chinese and gives answers to them. The machinetakes in Chinese stories and questions about them as input, it simulatesthe formal structure of actual Chinese brains in processing these stories,and it gives out Chinese answers as outputs. We can even imagine that themachine operates, not with a single serial program, but with a whole setof programs operating in parallel, in the manner that actual human brainspresumably operate when they process natural language. Now surely in sucha case we would have to say that the machine understood the stories; andif we refuse to say that, wouldn't we also have to deny that native Chinesespeakers understood the stories? At the level of the synapses, what wouldor could be different about the program of the computer and the program ofthe Chinese brain?"Before countering this reply I want to digress to note that it is an oddreply for any partisan of artificial intelligence (or functionalism, etc.)to make: I thought the whole idea of strong AI is that we don't need to knowhow the brain works to know how the mind works. The basic hypothesis, orso I had supposed, was that there is a level of mental operations consistingof computational processes over formal elements that constitute the essenceof the mental and can be realized in all sorts of different brain processes,in the same way that any computer program can be realized in different computerhardwares: On the assumptions of strong AI, the mind is to the brain as theprogram is to the hardware, and thus we can understand the mind without doingneurophysiology. If we had to know how the brain worked to do AI, we wouldn'tbother with AI. However, even getting this close to the operation of thebrain is still not sufficient to produce understanding. To see this, imaginethat instead of a monolingual man in a room shuffling symbols we have theman operate an elaborate set of water pipes with valves connecting them.When the man receives the Chinese symbols, he looks up in the program, writtenin English, which valves he has to turn on and off. Each water connectioncorresponds to a synapse in the Chinese brain, and the whole system is riggedup so that after doing all the right firings, that is after turning on allthe right faucets, the Chinese answers pop out at the output end of the seriesof pipes.Now where is the understanding in this system? It takes Chinese as input,it simulates the formal structure of the synapses of the Chinese brain, andit gives Chinese as output. But the man certainly doesn't understand Chinese,and neither do the water pipes, and if we are tempted to adopt what I thinkis the absurd view that somehow the conjunction of man and waterpipes understands, remember that in principle the man can internalize theformal structure of the water pipes and do all the "neuron firings" in hisimagination. The problem with the brain simulator is that it is simulatingthe wrong things about the brain. As long as it simulates only the formalstructure of the sequence of neuron firings at the synapses, it won't havesimulated what matters about the brain, namely its causal properties, itsability to produce intentional states. And that the formal properties arenot sufficient for the causal properties is shown by the water pipe example:we can have all the formal properties carved off from the relevantneurobiological causal properties.4. The Combination Reply (Berkeley and Stanford). "While each of theprevious three replies might not be completely convincing by itself as arefutation of the Chinese room counterexample, if you take all three togetherthey are collectively much more convincing and even decisive. Imagine a robotwith a brain-shaped computer lodged in its cranial cavity, imagine the computerprogrammed with all the synapses of a human brain, imagine the whole behaviorof the robot is indistinguishable from human behavior, and now think of thewhole thing as a unified system and not just as a computer with inputs andoutputs. Surely in such a case we would have to ascribe intentionality tothe system."I entirely agree that in such a case we would find it rational and indeedirresistible to accept the hypothesis that the robot had intentionality,as long as we knew nothing more about it. Indeed, besides appearance andbehavior, the other elements of the combination are really irrelevant. Ifwe could build a robot whose behavior was indistinguishable over a largerange from human behavior, we would attribute intentionality to it, pendingsome reason not to. We wouldn't need to know in advance that its computerbrain was a formal analogue of the human brain.But I really don't see that this is any help to the claims of strong AI,and here's why: According to strong AI, instatitiating a formal program withthe right input and output is a sufficient condition of, indeed is constitutiveof, intentionality. As Newell (1979) puts it, the essence of the mental isthe operation of a physical symbol system. But the attributions of intentionalitythat we make to the robot in this example have nothing to do with formalprograms. They are simply based on the assumption that if the robot looksand behaves sufficiently like us, then we would suppose, until proven otherwise,that it must have mental states like ours that cause and are expressed byits behavior and it must have an inner mechanism capable of producing suchmental states. If we knew independently how to account for its behavior withoutsuch assumptions we would not attribute intentionality to it, especiallyif we knew it had a formal program. And this is precisely the point of myearlier reply to objection II.Suppose we knew that the robot's behavior was entirely accounted for by thefact that a man inside it was receiving uninterpreted formal symbols fromthe robot's sensory receptors and sending out uninterpreted formal symbolsto its motor mechanisms, and the man was doing this symbol manipulation inaccordance with a bunch of rules. Furthermore, suppose the man knows noneof these facts about the robot, all he knows is which operations to performon which meaningless symbols. In such a case we would regard the robot asan ingenious mechanical dummy. The hypothesis that the dummy has a mind wouldnow be unwarranted and unnecessary, for there is now no longer any reasonto ascribe intentionality to the robot or to the system of which it is apart (except of course for the man's intentionality in manipulating the symbols).The formal symbol manipulations go on, the input and output are correctlymatched, but the only real locus of intentionality is the man, and he doesn'tknow any of the relevant intentional states; he doesn't, for example,see what comes into the robot's eyes, he doesn't intend tomove the robot's arm, and he doesn't understand any of the remarksmade to or by the robot. Nor, for the reasons stated earlier, does the systemof which man and robot are a part.To see this point, contrast this case with cases in which we find it completelynatural to ascribe intentionality to members of certain other primate speciessuch as apes and monkeys and to domestic animals such as dogs. The reasonswe find it natural are, roughly, two: We can't make sense of the animal'sbehavior without the ascription of intentionality and we can see that thebeasts are made of similar stuff to ourselves—that is an eye, that anose, this is its skin, and so on. Given the coherence of the animal's behaviorand the assumption of the same causal stuff underlying it, we assume boththat the animal must have mental states underlying its behavior, and thatthe mental states intent be produced by mechanisms made out of the stuffthat is like our stuff. We would certainly make similar assumptions aboutthe robot unless we had some reasons not to, but as soon as we knew thatthe behavior was the result of a formal program, and that the actual causalproperties of the physical substance were irrelevant we would abandon theassumption of intentionality.There are two other responses to my example that come up frequently (andso are worth discussing) but really miss the point.5. The Other Minds Reply (Yale). "How do you know that other peopleunderstand Chinese or anything else? Only by their behavior. Now the computercan pass the behavioral tests as well as they can (in principle), so if youare going to attribute cognition to other people you must in principle alsoattribute it to computers."This objection really is only worth a short reply. The problem in this discussionis not about how I know that other people have cognitive states, but ratherwhat it is that I am attributing to them when I attribute cognitive statesto them. The thrust of the argument is that it couldn't be just computationalprocesses and their output because the computational processes and theiroutput can exist without the cognitive state. It is no answer to this argumentto feign anesthesia. In "cognitive sciences" one presupposes the realityand knowability of the mental in the same way that in physical sciences onehas to presuppose the reality and knowability of physical objects.6. The Many Mansions Reply (Berkeley). "Your whole argument presupposesthat AI is only about analog and digital computers. But that just happensto be the present state of technology. Whatever these causal processes arethat you say are essential for intentionality (assuming you are right),eventually we will be able to build devices that have these causal processes,and that will be artificial intelligence. So your arguments are in no waydirected at the ability of artificial intelligence to produce and explaincognition."I really have no objection to this reply save to say that it in effecttrivializes the project of strong AI by redefining it as whatever artificiallyproduces and explains cognition. The interest of the original claim madeon behalf of artificial intelligence is that it was a precise, well definedthesis: mental processes are computational processes over formally definedelements. I have been concerned to challenge that thesis. If the claim isredefined so that it is no longer that thesis, my objections no longer applybecause there is no longer a testable hypothesis for them to apply to.Let us now return to the question I promised I would try to answer: Grantedthat in my original example I understand the English and I do not understandthe Chinese, and granted therefore that the machine doesn't understand eitherEnglish or Chinese, still there must be something about me that makes itthe case that I understand English and a corresponding something lackingin me that makes it the case that I fail to understand Chinese. Now why couldn'twe give those somethings, whatever they are, to a machine?I see no reason in principle why we couldn't give a machine the capacityto understand English or Chinese, since in an important sense our bodieswith our brains are precisely such machines. But I do see very strong argumentsfor saying that we could not give such a thing to a machine where the operationof the machine is defined solely in terms of computational processes overformally defined elements; that is, where the operation of the machine isdefined as an instantiation of a computer program. It is not because I amthe instantiation of a computer program that I am able to understand Englishand have other forms of intentionality (I am, I suppose, the instantiationof any number of computer programs), but as far as we know it is becauseI am a certain sort of organism with a certain biological (i.e., chemicaland physical) structure, and this structure, under certain conditions, iscausally capable of producing perception, action, understanding, learning,and other intentional phenomena. And part of the point of the present argumentis that only something that had those causal powers could have thatintentionality. Perhaps other physical and chemical processes could produceexactly these effects; perhaps, for example, Martians also have intentionalitybut their brains are made of different stuff. That is an empirical question,rather like the question whether photosynthesis can be done by somethingwith a chemistry different from that of chlorophyll.But the main point of the present argument is that no purely formal modelwill ever be sufficient by itself for intentionality because the formalproperties are not by themselves constitutive of intentionality, and theyhave by themselves no causal powers except the power, when instantiated,to produce the next stage of the formalism when the machine is running. Andany other causal properties that particular realizations of the formal modelhave, are irrelevant to the formal model because we can always put the sameformal model in a different realization where those causal properties areobviously absent. Even if, by some miracle, Chinese speakers exactly realizeSchank's program, we can put the same program in English speakers, waterpipes, or computers, none of which understand Chinese, the programnotwithstanding.What matters about brain operations is not the formal shadow cast by thesequence of synapses but rather the actual properties of the sequences. Allthe arguments for the strong version of artificial intelligence that I haveseen insist on drawing an outline around the shadows cast by cognition andthen claiming that the shadows are the real thing.By way of concluding I want to try to state some of the general philosophicalpoints implicit in the argument. For clarity I will try to do it in aquestion-and-answer fashion, and I begin with that old chestnut of aquestion: "Could a machine think?"The answer is, obviously, yes. We are precisely suchmachines."Yes, but could an artifact, a man-made machine, think?"Assuming it is possible to produce artificially a machine with a nervoussystem, neurons with axons and dendrites, and all the rest of it, sufficientlylike ours, again the answer to the question seems to be obviously, yes. Ifyou can exactly duplicate the causes, you could duplicate the effects. Andindeed it might be possible to produce consciousness, intentionality, andall the rest of it using some other sorts of chemical principles than thosethat human beings use. It is, as I said, an empirical question."OK, but could a digital computer think?"If by "digital computer" we mean anything at all that has a level of descriptionwhere it can correctly be described as the instantiation of a computer program,then again the answer is, of course, yes, since we are the instantiationsof any number of computer programs, and we can think."But could something think, understand, and so on solely in virtueof being a computer with the right sort of program? Could instantiating aprogram, the right program of course, by itself be a sufficient conditionof understanding?"This I think is the right question to ask, though it is usually confusedwith one or more of the earlier questions, and the answer to it is no."Why not?"Because the formal symbol manipulations by themselves don't have anyintentionality; they are quite meaningless; they aren't even symbolmanipulations, since the symbols don't symbolize anything. In the linguisticjargon, they have only a syntax but no semantics. Such intentionality ascomputers appear to have is solely in the minds of those who program themand those who use them, those who send in the input and those who interpretthe output.The aim of the Chinese room example was to try to show this by showing thatas soon as we put something into the system that really does have intentionality(a man), and we program him with the formal program, you can see that theformal program carries no additional intentionality. It adds nothing, forexample, to a man's ability to understand Chinese.Precisely that feature of AI that seemed so appealing—the distinctionbetween the program and the realization—proves fatal to the claim thatsimulation could be duplication. The distinction between the program andits realization in the hardware seems to be parallel to the distinction betweenthe level of mental operations and the level of brain operations. And ifwe could describe the level of mental operations as a formal program, thenit seems we could describe what was essential about the mind without doingeither introspective psychology or neurophysiology of the brain. But theequation "mind is to brain as program is to hardware" breaks down at severalpoints, among them the following three:First, the distinction between program and realization has the consequencethat the same program could have all sorts of crazy realizations that hadno form of intentionality. Weizenbaum (1976, Ch. 2), for example, shows indetail how to construct a computer using a roll of toilet paper and a pileof small stones. Similarly, the Chinese story understanding program can beprogrammed into a sequence of water pipes, a set of wind machines, or amonolingual English speaker, none of which thereby acquires an understandingof Chinese. Stones, toilet paper, wind, and water pipes are the wrong kindof stuff to have intentionality in the first place—only something thathas the same causal powers as brains can have intentionality—and thoughthe English speaker has the right kind of stuff for intentionality you caneasily see that he doesn't get any extra intentionality by memorizing theprogram, since memorizing it won't teach him Chinese.Second, the program is purely formal, but the intentional states are notin that way formal. They are defined in terms of their content, not theirform. The belief that it is raining, for example, is not defined as a certainformal shape, but as a certain mental content with conditions of satisfaction,a direction of fit (see Searle 1979), and the like. Indeed the belief assuch hasn't even got a formal shape in this syntactic sense, since one andthe same belief can be given an indefinite number of different syntacticexpressions in different linguistic systems.Third, as I mentioned before, mental states and events are literally a productof the operation of the brain, but the program is not in that way a productof the computer."Well if programs are in no way constitutive of mental processes, why haveso many people believed the converse? That at least needs some explanation."I don't really know the answer to that one. The idea that computer simulationscould be the real thing ought to have seemed suspicious in the first placebecause the computer isn't confined to simulating mental operations, by anymeans. No one supposes that computer simulations of a five-alarm fire willburn the neighborhood down or that a computer simulation of a rainstorm willleave us all drenched. Why on earth would anyone suppose that a computersimulation of understanding actually understood anything? It is sometimessaid that it would be frightfully hard to get computers to feel pain or fallin love, but love and pain are neither harder nor easier than cognition oranything else. For simulation, all you need is the right input and outputand a program in the middle that transforms the former into the latter. Thatis all the computer has for anything it does. To confuse simulation withduplication is the same mistake, whether it is pain, love, cognition, fires,or rainstorms.Still, there are several reasons why AI must have seemed—and to manypeople perhaps still does seem—in some way to reproduce and therebyexplain mental phenomena, and I believe we will not succeed in removing theseillusions until we have fully exposed the reasons that give rise to them.First, and perhaps most important, is a confusion about the notion of"information processing": many people in cognitive science believe that thehuman brain, with its mind, does something called "information processing,"and analogously the computer with its program does information processing;but fires and rainstorms, on the other hand, don't do information processingat all. Thus, though the computer can simulate the formal features of anyprocess whatever, it stands in a special relation to the mind and brain becausewhen the computer is properly programmed, ideally with the same program asthe brain, the information processing is identical in the two cases, andthis information processing is really the essence of the mental. But thetrouble with this argument is that it rests on an ambiguity in the notionof "information." In the sense in which people "process information" whenthey reflect, say, on problems in arithmetic or when they read and answerquestions about stories, the programmed computer does not do "informationprocessing." Rather, what it does is manipulate formal symbols. The factthat the programmer and the interpreter of the computer output use the symbolsto stand for objects in the world is totally beyond the scope of the computer.The computer, to repeat, has a syntax but no semantics. Thus, if you typeinto the computer "2 plus 2 equals?" it will type out "4." But it has noidea that "4" means 4 or that it means anything at all. And the point isnot that it lacks some second-order information about the interpretationof its first-order symbols, but rather that its first-order symbols don'thave any interpretations as far as the computer is concerned. All the computerhas is more symbols. The introduction of the notion of "information processing"therefore produces a dilemma: either we construe the notion of "informationprocessing" in such a way that it implies intentionality as part of the processor we don't. If the former, then the programmed computer does not do informationprocessing, it only manipulates formal symbols. If the latter, then, thoughthe computer does information processing, it is only doing so in the sensein which adding machines, typewriters, stomachs, thermostats, rainstorms,and hurricanes do information processing; namely, they have a level ofdescription at which we can describe them as taking information in at oneend, transforming it, and producing information as output. But in this caseit is up to outside observers to interpret the input and output as informationin the ordinary sense. And no similarity is established between the computerand the brain in terms of any similarity of information processing.Second, in much of AI there is a residual behaviorism or operationalism.Since appropriately programmed computers can have input-output patterns similarto those of human beings, we are tempted to postulate mental states in thecomputer similar to human mental states. But once we see that it is bothconceptually and empirically possible for a system to have human capacitiesin some realm without having any intentionality at all, we should be ableto overcome this impulse. My desk adding machine has calculating capacities,but no intentionality, and in this paper I have tried to show that a systemcould have input and output capabilities that duplicated those of a nativeChinese speaker and still not understand Chinese, regardless of how it wasprogrammed. The Turing test is typical of the tradition in being unashamedlybehavioristic and operationalistic, and I believe that if AI workers totallyrepudiated behaviorism and operatiotialism much of the confusion betweensimulation and duplication would be eliminated.Third, this residual operationalism is joined to a residual form of dualism;indeed strong AI only makes sense given the dualistic assumption that, wherethe mind is concerned, the brain doesn't matter. In strong AI (and infunctionalism, as well) what matters are programs, and programs are independentof their realization in machines; indeed, as far as AI is concerned, thesame program could be realized by an electronic machine, a Cartesian mentalsubstance, or a Hegelian world spirit. The single most surprising discoverythat I have made in discussing these issues is that many AI workers are quiteshocked by my idea that actual human mental phenomena might be dependenton actual physical-chemical properties of actual human brains. But if youthink about it a minute you can see that I should not have been surprised;for unless you accept some form of dualism, the strong AI project hasn'tgot a chance. The project is to reproduce and explain the mental by designingprograms, but unless the mind is not only conceptually but empiricallyindependent of the brain you couldn't carry out the project, for the programis completely independent of any realization. Unless you believe that themind is separable from the brain both conceptually and empirically—dualismin a strong form—you cannot hope to reproduce the mental by writingand running programs since programs must be independent of brains or anyother particular forms of instantiation. If mental operations consist incomputational operations on formal symbols, then it follows that they haveno interesting connection with the brain; the only connection would be thatthe brain just happens to be one of the indefinitely many types of machinescapable of instantiating the program. This form of dualism is not the traditionalCartesian variety that claims there are two sorts of substances, butit is Cartesian in the sense that it insists that what is specifically mentalabout the mind has no intrinsic connection with the actual properties ofthe brain. This underlying dualism is masked from us by the fact that AIliterature contains frequent fulminations against "dualism"; what the authorsseem to be unaware of is that their position presupposes a strong versionof dualism."Could a machine think?" My own view is that only a machinecould think, and indeed only very special kinds of machines, namely brainsand machines that had the same causal powers as brains. And that is the mainreason strong AI has had little to tell us about thinking, since it has nothingto tell us about machines. By its own definition, it is about programs, andprograms are not machines. Whatever else intentionality is, it is a biologicalphenomenon, and it is as likely to be as causally dependent on the specificbiochemistry of its origins as lactation, photosynthesis, or any other biologicalphenomena. No one would suppose that we could produce milk and sugar by runninga computer simulation of the formal sequences in lactation and photosynthesis,but where the mind is concerned many people are willing to believe in sucha miracle because of a deep and abiding dualism: the mind they suppose isa matter of formal processes and is independent of quite specific materialcauses in the way that milk and sugar are not.In defense of this dualismthe hope is often expressed that the brain is a digital computer (earlycomputers, by the way, were often called "electronic brains"). But that isno help. Of course the brain is a digital computer. Since everything is adigital computer, brains are too. The point is that the brain's causal capacityto produce intentionality cannot consist in its instantiating a computerprogram, since for any program you like it is possible for something toinstantiate that program and still not have any mental states. Whatever itis that the brain does to produce intentionality, it cannot consist ininstantiating a program since no program, by itself, is sufficient forintentionality.3NOTES:1 Also, "understanding" implies both the possessionof mental (intentional) states and the truth (validity, success) of thesestates. For the purposes of this discussion we are concerned only with thepossession of the states.2 Intenionality is by definition that feature ofcertain mental states by which they are directed at or about objects andstates of affairs in the world. Thus, beliefs, desires, and intentions areintentional states; undirected forms of anxiety and depression are not.3 I am indebted to a rather large number of peoplefor discussion of these matters and for their patient attempts to overcomemy ignorance of artificial intelligence. I would especially like to thankNed Block, Hubert Dreyfus, John Haugeland, Roger Schank, Robert Wilensky,and Terry Winograd. |
|
