Science, Ecological Validity and Experimentation

SIU L. CHOW

With the advent of the information-processing approach to mental phenomena (Broadbent, 1958; Neisser, 1967; Haber, 1969), the view of science implicitly assumed by cognitive psychologists is very different from the positivistic views exemplified by Skinner (1938, 1956, 1957) and Hull (1943). Nonetheless, experimental psychology is generally being talked about or evaluated (by friends and foes alike) in terms of some composite of the Skinnerian and the Hullian views; this composite view has been called the standard view by Manicas and Secord (1983). Greenwood (1982) has made a similar observation in his discussion of the experimental approach to social psychology.

Manicas and Secord (1983) and Mook (1983) have recently highlighted some important meta-theoretical features which differentiate the contemporary experimental approach from the behaviouristic approach. The spirit of Manicas and Secord's (1983) "realist's" view can be amplified, with reference to Popper's (1962/1968) "conjectures and refutations" framework, into which some of Mook's (1983) eloquent assertions may be incorporated. This exercise is useful because the confluence of these ideas represent well the methodological characteristics of contemporary cognitive psychology.

The essence of Manicas and Secord's (1983) view is that, even though physical stimuli are being manipulated and observable behaviour is being monitored in experiments, the experimental objective is to understand the unobservable mechanisms underlying overt behaviour. This understanding is not in the form of subsuming experimental observations under some descriptive organism-environment relations or stimulusresponse regularities called "descriptive-correlative" propositions by Greenwood (1982, p. 230). Instead, it assumes the form of some theoretical statements enjoying warranted assertability. What needs discussing is how "warranted assertability" is achieved.

Experimental studies are often criticized on the grounds that conclusions based on experimental data cannot be applied to settings outside the laboratory because the laboratory is too unlike the real world (see Neisser, 1976, particularly p. 33). For this reason, contemporary cognitive psychology is found wanting because it lacks ecological validity¹ (Neisser, 1976). In support of this criticism is the contention that human behaviour outside the laboratory cannot be predicted from what has been learned in the laboratory (see, for example, Haber's 1983, prognosis of the impending demise of the icon).

Behind Neisser's (1976) "ecological validity" requirement is the assumption that observations made in the laboratory have to (or are meant to) mimic everyday life phenomena. Mook (1983) eloquently argues that this criticism of the experimental approach to psychology is incorrect because experimental studies are concerned with "what can happen," not "what does happen." Nor do experimental conclusions serve a "predict to" function. Instead, experimental conclusions come about as a result of "predicting from" theories. Mook's (1983) argument requires further elaboration.

In saying correctly that experiments data do not have to be generalized to other situations, Mook (1983) may be inadvertently conceding too much. What needs to be shown is that observations collected in an artificial setting may be what are needed to test a theory. The objective of this paper is to reinforce Manicas and Secord's (1983) and Mook's (1983) theses by describing the rationale of experimentation with reference to the Popperian framework.

Also essential to the standard view of science are (a) the meta-theoretical assumption that whatever intervenes between the observable stimulus and response events is of no consequence and (b) the methodological assumption that the laboratory setting somehow is (or meant to be) representative of the world outside the laboratory.

The objective of a scientific psychology is a utilitarian one in the standard view. A knowledge of organism-environment regularities enables the behavioural psychologists to control responses by judiciously furnishing or withholding the requisite stimuli. Consequently, empirical laws (Skinner, 1938) or empirical generalizations (Hull, 1943) are assessed in terms of their predictive power. The range of situations, subjects, and tasks over which the predictions are met determines the generality (and hence the external validity) of the empirical laws or generalizations.

Cognitive psychologists, on the other hand, assume that there are theoretically significant mechanisms (each with its unique structural and functional characteristics) underlying observable behaviour. These underlying mechanisms are assumed to have causal capabilities. Such mechanisms are called "hypothetical constructs" (MacCorquodale and Meehl, 1948) or "hypothetical mechanisms."

Within this framework, cognitive psychologists are not interested in descriptive laws of stimulus-response regularities per se, although these regularities may lead to the formulation of a theory. Rather, they are concerned with the underlying causes of behaviour. As there is no theory-neutral datum, all observations (including experimental ones) are guided by theoretical insights. At the same time, experimentation serves to corroborate theories.

A non-utilitarian objective is envisaged in the realist's view. In this view, an experimenter does not predict what will happen by appealing to some pre-determined empirical laws or generalizations. That is, the specific stimuli used in an experiment are not the bases of the subjects' responses. Moreover, whereas the conditions within the laboratory may be characterized as "conditions of complete closure" (Manicas and Secord, 1983, p. 402), the world without is an open system (See also Greenwood, 1982). Hence, conclusions from experimentation may not have any predictive power outside the laboratory (Manicas and Secord, 1983). At the same time, many experimental studies are not meant to be generalized to the world outside the laboratory (Mook, 1983). Instead of serving a utilitarian function, experimentation serves to enhance understanding at the level of human competence (Manicas and Secord, 1983).

One interpretation of Manicas and Secord's (1983) claim that the real world is an open system is that, given any mental or behavioural phenomenon of interest, it can be interpreted or explained in numerous ways. Every one of these alternative interpretations or explanations is a conjecture, however, educated. The hypothesis may be one about an ontologically real and causally important mechanism with specific properties (albeit unobservable). A conjecture is tentatively accepted as true so long as it can withstand concerted efforts to refute it while competing interpretations or explanations are being refuted. The rationale of experimentation in the "conjectures and refutations" view will be illustrated with a mundane example which serves as an analogy for discussing an example from a cognitive psychology.

Consider the case when a person explains a wet freeway with the statement, "It rained". The rainfall in the person's absence is, by definition, unobservable to that person. The wet freeway is an observable state of affairs. Moreover, there are many ways in which the freeway may become wet (e.g., someone may have washed it). That is, raining is only one of the many probable explanations of a wet freeway.

If it is true that it has rained, there must be some observations other than the wet freeway which must be present had it rained. One such candidate is the observation that leaves on top of a ten-metre tree beside the freeway are wet. In other words, in order for the statement, "It rained," to be true as an explanation of the wet freeway, not only has the freeway to be wet, but something else also has to be wet (e.g., leaves on top of tall trees). That is, in its capacity as a cause-effect statement, a theory necessarily has some causal implications other than the phenomenon for which the theory is first proposed. The relationship between a theory and one of its many causal implications is represented by the following conditional proposition:

If Theory T is correct, then Implication I is observed. (If it rained, then leaves on tree tops should be wet.)

The formal relationship between (a) a theory, (b) an implication of the theory, (c) experimental observations collected for evaluating the theory, and (d) the permissible conclusion may now be presented as follows:

(ii). Condition C is specifically set up for data collection. That is, an experiment is conducted in the way stipulated by an implication of the theory.

(iii). Observations collected are assessed as to whether they are consistent with the consequent of the conditional proposition. Call the statement descriptive of the experimental outcomes, "0," if the experimental outcomes are as stipulated and "-0" (i.e., not "0") otherwise. This statement is used as the minor premise in the syllogistic argument. Note that whether the experimental outcomes is 0 or not is an empirical matter. As an empirical event, it is not affected by what the theory implies. In other words, the theory does not predict what will happen. Instead, the theory prescribes what must happen in order for the theory to be true. This distinction between "prescription" and "prediction" is what Mook (1983) meant by the distinction between "predict from (a theory)" and "predict to (what can happen)."

(iv). The choice of the experimental conclusion is restricted to two mutually exclusive alternatives. If 0 is not observed, it can be concluded that Theory T is false (by the Modus Tollens rule).³ If 0 is observed, it is concluded that Theory T has not been refuted. A positive way of presenting this conclusion is to say that Theory T is probably true.

The Popperian rationale of experimentation just presented in general terms may be illustrated with a concrete example: Sternberg's (1969) memory-search study.⁴ The subjects in Sternberg's (1969) experiment were given one, two or four digits to memorize. A single digit (called the "probe") was presented, to which the subjects has to respond whether the probe was a member of the initial set of digits ("old") or not ("new"). The number of items to be memorized varied from trial to trial. This independent variable is called the "memory set size."

Sternberg (1969) considered three theoretical possibilities. First, the search mechanism might begin and finish searching all the elements in the memory set simultaneously (hence, called the "parallel" model). This hypothesis prescribes that the subjects' correct "old" and "new" reaction times should be the same for all memory set sizes. That is, the two functions should be flat and parallel to each other.

The second possibilities is the "serial, self-terminating" model. The search mechanism is assumed to compare the probe to the memory items one at a time. As soon as a match is found, the appropriate response will be made. This hypothesis prescribes that both correct "old" and "new" responses should increase in the memory set size. Furthermore, the slope of the "old" function should be half that of the "new" function.

The "serial, exhaustive" model is the third theoretical possibilities in which the system again searches through the memory set one after another. However, no response is made until all items in the memory set have been searched. This hypothesis prescribes that the correct "old" and "new" functions should be two parallel functions increasing with increases in the memory set sizes.

Sternberg (1969) tested the three theoretical possibilities by testing what they prescribed should happen in a well-defined task situation. Although three theoretical possibilities are being considered, the measurement of the subjects' correct reaction times and the categorization of the subjects' responses (i.e., being correct or otherwise) are independent of any one of these hypotheses. Sternberg's (1969) experimental task is atheoretical in this sense.

At a different level, Sternberg's (1969) experiments are theory-dependent. For example, the manipulation of the memory set-size is based on the theoretical properties of the hypothetical scanning mechanism. When the encoding of the probe is being investigated, the quality of the probe is manipulated. In other words, the choice of the independent variable is prescribed by a hypothetical property of the underlying mechanism. That is, experimental psychology is a theoretical science in the sense that experimental expectations depend on the theoretical properties attributed to the hypothetical mechanism in question. A psychological experiment is also theory-dependent because the choice of the experimental design and of the independent variables depend on the theoretical properties under investigation. Hence, ". . . experimental psychology is necessarily a theoretical science" (Manicas & Secord, 1983, p. 406). However, the experimental task should be theoretically neutral⁵.

It has been shown in discussing Sternberg's (1969) study that a deduction (or theoretical implication) is first derived from a theory. This theoretical implication prescribes what should happen in order for the hypothesis to be true. Moreover, this theoretical implication is made in the context of a well-defined atheoretical task.

The consequent of the major premise describes the necessary condition for the tenability of the antecedent (see Point (i) on p. 184). It is imperative to emphasize that the behavioural outcomes prescribed in the consequence are so stipulated, not just because of the choice of the stimulus, but also because of the hypothetical property envisaged in the theory. Hence, it is not appropriate to say that a prediction has been made as to what the subjects will do in the experiment. Greenwood (1982) suggests that this function is best served by a different kind of proposition, namely, the descriptive-correlative proposition, which is more commonly known as proposition about a functional relationship between two variables. The theory does not, and cannot, do this. Nor is the experimenter capable of doing that. It is more appropriate and informative to say that a theoretical expectation is stipulated by the theory before data collection. In other words ". . . we are not using what happens in the laboratory to 'predict' the real world. Prediction goes the other way; Our theory specifies what subjects should do in the laboratory" (Mook, 1983, p. 383, Mook's own emphasis). When the realist's framework is adopted, "theoretical expectation" should replace "prediction".

The realist's view of science raises several issues which require further discussion. Otherwise, the realist's view may lose the impact it deserves. First, the hypothetical mechanism intervening between a stimulus and a response is, by definition, not observable in the way a table is observable. How can one be sure that the unobservable underlying mechanism has the properties attributed to it? This is the issue of "(how) to link experience . . . with reality independent from experience . . ." raised by Leary (1984, p. 918). Secondly, it may seem an easy way out to say that experimental data are not meant to be generalized to the world outside the laboratory or that no prediction is possible because the world without is an open system. Thirdly, experimental observations are made in an artificial setting which may be very unlike the real world. How relevant are these observations to the world outside the laboratory? This is Neisser's (1976) question of ecological validity. These three questions can readily be dealt with in terms of the thesis developed so far.

The first question to consider is how objectivity can be achieved when observable behaviour is said to be caused by unobservable mechanisms. It must be admitted that this question cannot be answered in absolute terms because we cannot exclude all alternative explanations. However, an objective rule is available for deciding whether a theoretical insight is warranted or not. The rule is the modus operandi of the "conjectures and refutations" framework described above.

That is, it has to be admitted that our knowledge is only tentative. However, this tentative knowledge is constantly being tested by a continuous series of attempts to refute it. Tentative knowledge is also being continuously refined because, ideally, every unsuccessful attempt to falsify a theory also excludes at least one contending theory. It is for this reason that Popper (1962/1968) insists that a scientifically worthwhile theory is a theory which can be refuted.

The next issue to consider has two facets. On the one hand, a question may be asked about the extent to which the evidential observation used to study a phenomenon has to mimic the phenomenon. Mook (1983) suggests that such a mimicry is sometimes not necessary because some experimental conclusions are not meant to be used outside the laboratory. Be that as it may, one may still ask how relevant laboratory findings are to everyday life. For example, it has been said that experimental studies of the perceptual processes do not have any ecological validity because the experimental settings are too artificial (Gibson, 1979; Haber, 1983; Neisser, 1976)⁶. Four observations may be offered in this discussion.

Recall the mundane example about the rain and the wet freeway. The "experimental task" consists of climbing up a ten-metre tree; the unit of evidential observation is a leaf. Yet, the object of interest is the freeway. Despite the differences between a leaf and the freeway, it is readily accepted that wet leaves on tree tops are necessary for accepting that it has rained. In other words, it can be asserted that the evidential observation used to evaluate a theory about a phenomenon does not have to mimic the phenomenon. (See Berkowitz & Donnerstein, 1982, for a different way of dealing with this issue.)

Any acceptable theory for a phenomenon is necessarily consistent with the phenomenon. Consequently, if there are two contending theories, they would both be consistent with the original phenomenon. The closer the evidential observation chosen for evaluating the two theories is to the original phenomenon, the more likely the two contending theories are to account for the evidential observation. Hence, the less likely that one can choose between the two contending theories on the basis of the evidential observation. This point may be illustrated by extending the mundane example.

An alternative explanation of the wet freeway may be that someone has washed the freeway. Another potential evidential observation is whether the streets in a nearby town are wet. Wet streets are more similar to the wet freeway than wet leaves. In terms of Neisser's (1976) notion of "ecological validity," wet streets are ecologically more valid than wet leaves as evidential observation for deciding whether it had rained or someone had washed the freeway. However, both theories can account for wet streets equally readily, whereas the theory that it had rained can account for wet leaves more readily than the theory that someone had washed the freeway. That is, in order to exclude a contending theory, it is better to use an observation which is unlike the phenomenon to be explained as the evidential observation. In other words, requiring the new evidential observation to mimic the original phenomenon so as to satisfy Neisser's (1976) insistence on ecological validity will diminish the usefulness of the evidential observation. This is obviously counter-productive (see also Meehl, 1978).

To establish the generality of a theory is generally held to be an attempt to extend the range of its applicability. Moreover, it is generally held that generality is established by running literal replications of the original experiment. Mook's (1983) discussion of the agriculture model of experimentation is instructive.

A typical experimental question in early agricultural research is whether or not a particular fertilizer would give a better yield. (The application of the fertilizer in question will be called the "substantive treatment.") The experimental manipulation is the application of the fertilizer under investigation. That is, the substantive treatment is used as the experimental manipulation. This practice of not differentiating between substantive treatment and experimental manipulation has been called the "agricultural model" of experimentation (see also Hogben, 1957; Meehl, 1978).

The notion of "generality" is raised because the experimental questions asked in agricultural research are based on (a) the assumption that the experimental setting (e.g., the types of soil, seeds, and fertilizer used) is representative of the setting of interest, (b) the realization that the experimental conclusion is based on the current experimental setting, (c) the pragmatic decision that the experimental conclusion becomes the guide for future practice or the theoretical conclusion, and (d) the realization that what is true in the experimental setting may not be true in a different setting. Hence, there is the concern with generality (see also Meehl, 1978).

However, establishing the generality of a theory from cognitive psychology's perspective is quite different from the agricultural model. The applicability of a theory to a specific situation is doubtful only when there are theoretical reasons to do so (i.e., not simply because it is different). For example, there may be reasons to believe that some features in a particular situation are incompatible with the theory. On the basis of both the relevant features in the situation in question and the theory, a unique implication is derived. Consequently, an experiment different from the original one is required to test the new implication. This is the case every time the generality of a theory is being tested. That is, instead of relying on literal replications of the original experiment, dissimilar experiments may have to be used to test the various experimental expectations. These tests constitute the "converging operations" for corroborating a theory (Garner, Hake, and Eriksen, 1956). It is important to emphasize that each of these converging operations may be an experiment quite unlike the to-be-explained phenomenon.

Neisser (1976) applies the "ecological validity" characterization to both a theory and the experimental studies designed to corroborate the theory. Is this a good practice? Recall that, although leaves are examined in the mundane example, the conclusion is about a climatic event. Likewise, although the experimental setting is artificial, and the evidential observation may be different from the phenomenon of interest, the theoretical conclusion drawn (as opposed to the statistical decision made, Tukey, (1970) is one about a theory proposed to explain a certain type of events in everyday life. If the theory is ecologically valid, this validity is not affected by the artificial nature of the experimental study. In other words, ecological validity should be a question about a theory. It should not be used to decide whether or not the choice of the evidential observation is determined by the theoretical expectation in a well-defined and (admittedly artificial) setting.

Some important meta-theoretical insights regarding experimental psychology proposed by Manicas and Secord (1983) and Mook (1983) are described and integrated into Popper's (1962/1968) "conjectures and refutations" framework. It is argued that the modus operandi of the "conjectures and refutations" framework is the objective rule which makes it possible to decide whether an unobservable mechanism underlying observable behaviour can be validly entertained. In presenting the rationale of experimentation in the Popperian framework, it is argued that (a) the evidential observation used to study a phenomenon does not have to mimic the phenomenon itself, (b) such a mimicry is counter-productive, (c) the generality of a theory is not based on literal replications of the original experiment, (d) the "ecological validity" question is one about a theory, and (e) the fact that the experimental setting is artificial is irrelevant to the ecological validity of a theory.

Acknowledgement This research was supported by a Category A grant to the author from the University of Wollongong. I wish to thank Don Mixon, Peter Collett, and four anonymous reviewers for their helpful comments.

Correspondence concerning this article should be addressed to Siu L. Chow, Department of Psychology, University of Regina, Regina, Saskatchewan, Canada S4S 0A2 (formerly of University of Wollongong, P.O. Box 1144, Wollongong, New South Wales, Australia, 2500).

NOTES

REFERENCES

BERKOWITZ, L. and DONNERSTEIN, E. (1982). External validity is more than skin deep: Some answers to criticisms of laboratory experiments. American Psychologist, 37, 245-57.

BROADBENT, D. E. (1958). Perception and communication. London: Pergamon Press.

GARNER, W.R., HAKE., H.W., and ERIKSEN, C.W. (1956). Operationism and the concept of perception. Psychological Review, 67, 149-59.

GIBSON, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.

GREENWOOD, J.D. (1982). On the relation between laboratory experiments

and social behaviour: Causal explanation and generalization. Journal for the

Theory of Social Behaviour, 12, 225-250.

HABER, R.N. (Ed.). (1969). Information-processing approaches to visual perception. New York: Holt, Rinehart and Winston.

HABER, R.N. (1983). The impending demise of the icon: A critique of the concept of iconic storage in visual information processing. Behavioural and Brain Sciences, 6, 1-54.

HOGBEN, L. (1957). Statistical theory. London: Allen and Unwin.

HULL, C.L. (1943). Principles of behaviour: An introduction to behaviour theory. New York: Appleton-Century.

LEARY, D.E. (1984). Philosophy, psychology, and reality. American Psychologist, 39, 917-19.

LOFTUS, G.R. (1983). The continuing persistence of the icon. Behavioural and Brain Sciences, 6, 28.

MACCORQUODALE, K., and MEEHL, P.E. (1948). On a distinction between hypothetical constructs and intervening variables. Psychological Review, 55, 95-107.

MANICAS, P.T., and SECORD, P.F. (1983). Implications for psychology of the new philosophy of science. American Psychologist, 38, 399-413.

MEEHL, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of social psychology. Journal of Consulting and Clinical Psychology, 46, 806-34.

MILGRAM, S. (1963). Behavioural study of obedience. Journal of Abnormal and Social Psychology, 67, 371-78.

MOOK, D.G. (1983). In defense of external invalidity. American Psychologist, 38, 379-87.

NEISSER, U. (1967). Cognitive psychology. New York: Appleton-Century.

NFISSER, U. (1976). Cognition and reality. San Francisco: Freeman.

POPPER, K.R. (1962/1968). Conjectures and refutations (originally published in 1962). New York: Harper & Row.

SKINNER, B.F. (1938). The behaviour of organism: An experimental analysis. New York: Appleton-Century.

SKINNER, B.F. (1947). Experimental psychology. In W. Dennis (Ed.), Current trends in psychology. Pittsburgh, Pennsylvania: University of Pittsburgh Press.

SKINNER, B.F. (1950). Are theories of learning necessary? Psychological Review, 57, 13-216.

SKINNER, B.F. (1956). A case history in scientific method. American Psychologist, 11, 221-33.

SKINNER, B.F. (1957). The experimental analysis of behavior. American Scientist, 45, 343-371.

STERNBERG, S. (1969). Memory-scanning: Mental processes revealed by reaction-time experiments. American Scientist, 57, 421-57.

TUKEY, J.W. (1970). Conclusion vs. decision. In P. Badia, A. Haber, & R.P. Runyon (Eds.), Research problems in psychology. Massachusetts, Reading: Addison-Wesley.

TURNER, M.B. (1967). Psychology and the philosophy of science. New York: Appleton-Century.