Verplanck, W.S. (1992) A brief introduction to the Word Associate Test. The Analysis of Verbal Behavior, 10, 97-123.
A Brief Introduction to
THE WORD ASSOCIATE TEST
William S. Verplanck
University of Tennessee, Knoxville
An examination format assessing the intraverbal repertoire
of individuals in psychology is described and results using it reported.
The Associate Test is easy to prepare, to take, and to grade. Its reliability
measures are satisfactory; its ability to predict later behavior is reported
upon. The Associate Test is computer friendly, and its methods can be applied
for examination in any field, and at any level.
The writer would like to acknowledge gratefully the work
of J.R. Caplan, R. Fleischer, A. Gordon, W.N. Jennings, R.N. Krueger, R.
A. Ramos, W. Saeger, and J. Shelnutt. P.J. Lazzara and R.D. Kinser assisted
in summarizing this research and in assembling the appendices.
The Word Associate Test (WAT) (or Short-Short Answer Test) is a format
for preparing brief tests designed to demonstrate the student's command
of the critical terms of specific areas of study. It provides a measure
of the student's vocabulary in specialized and technical areas. That is,
it samples the minimal intraverbal repertoires of the individual
The Word Associate Test has been developed over some 30 years of research.
[The progressive development of the Associate Test and its
methodology into its present form have been reported at meetings of The
Psychonomic Society (6), of the Southeast Psychol. Ass'n. (6), and of the
American Education Research Ass'n. More detailed reports have appeared
in 1 Honors Paper, 2 MA Theses, and 2 Doctoral Dissertations. A listing
of these papers may be requested of the author.] This work was instigated
by dissatisfaction with the multiple-choice test, whose scores are patently
associated with specific content-independent verbal skills and with well
shaped monents for guessing.
The primary objective of the WAT is the rapid examination and grading
of large numbers of students in one or another of a broad variety of areas
(Jennings, W.N., 1968; Saeger, W., 1975), and at a number of levels
of achievement. It was not developed to find how people think, how original
[Jennings (1966) has nevertheless shown that there
is a relationship between word-associate measures and evaluation of the
"originality" shown in compositions written for the instructors
in The English Department at UTK. These instructors had no information
whatever on the WAT, much less of each student's performance.] they
are, how well they can explain something, or write, much less to predict
academic success, life achievement, or aptitude for something or other.
The command of specified vocabularies is, however, necessary for all these
performances.
The WAT asks only one question, which has many paraphrases in ordinary
language: "How successfully has he/she studied?", "What
words can the teacher or writer use and expect the students to be able
to deal with at an acceptable level?", and loosely, "How much
does the student know?". It does so without the use of the prompts
that appear in multiple-choice, true-false, matching, and fill-in-the-blanks
examination items. Incorrect prompts may indeed be added to the examinee's
intraverbal repertoire.
The WAT may be termed "robust": Instructions to guess do not
increase scores; instructions "Do NOT guess," however, decrease
them.
The WAT demonstrates the examinees minimum level of command of a selected
vocabulary. The WAT does not enable students to demonstrate their full
knowledge. For this, a further WAT, presenting a more advanced technical
vocabulary should be taken. In essence, a WAT can be prepared for any level
of study, from kindergarten through the doctorate, in any field (with the
possible exception of some branches of mathematics).
Although the test was developed before computer technology had reached
its present level of sophistication and accessibility, it was preadapted
for computer administration, grading, and presentation of results. Computer
technology will enable a student to take an Associate Test on any subject
at any level of achievement at any time at any terminal with immediate
scoring against national norms.
It may now be applied to evaluate the achievement of individual students
or specific classes and the effectiveness of various teaching methods,
or of instructors. (Jennings, 1968; Saeger, 1975; Shelnutt, J., 1969)
The description which follows gives the present status of the Word Associate
Test. The appendices present sets of data relevant to the use of the test
and to how it works in practice. An extended monograph on the WAT is now
in preparation, and is to be published by the Cambridge Center for Behavioral
Studies. This will report on the research on the WAT carried out to date,
and will indicate its potential as a research instrument as well as a method
for examination of substantive command of subject matter. Users will evaluate
its face validity on the basis of their experience with its use and with
the students who have taken it.
Given the current state of understanding of relevant criteria and of
"psychometric validity," the psychometric validity of the WAT
is no better or worse than that of any other form of examination, whether
essay, take-home, true-false, or multiple choice. It can and will be compared
with others at such time as acceptable sets of well-defined criteria can
be agreed upon.
Although the WAT has been independently developed, the readers of Rosanoff
(1918), Deese and Loftus will find it of both practical and theoretical
importance. The reader will also recognize that "Word Associate Test"
is perhaps too restricted a name. Besides words, names, and dates, one
may also use equations, curves, chemical formulas, diagrams and other visual
material as stimulus items in preparing examinations using this design,
and examinees may respond by drawing pictures and sketching graphs or diagrams,
as some have done.
When the response given to the stimulus word or term is verbal, rather
than a diagram, sketch, or graph, the associate may be considered a primary
or first-order intraverbal, one that is given directly to the stimulus.
These are the responses most likely to appear in sentences written when
the examinee is asked to write brief essays on the term. In these answers,
the first term to appear is most often the stimulus term itself--in this
context, a notate, or tact.
The possibility of the use of stimuli other than written or spoken words,
such as pictures, diagrams, sounds, and so on, is one reason why the term
"associate," rather than "intraverbal behavior" (Skinner,
B.F., 1957), is used. Certainly the specific methodology presented
deals with intraverbal behavior. But the methodology lends itself equally
well to the investigation of tacts, as distinguished from "intraverbal
responses." Word associates are intraverbal behaviors only when the
stimuli themselves are words or phrases; otherwise, they are tacts. In
Skinner's Verbal Behavior, the terms "tact" and "intraverbal
response" have extensive connotations, which may or may not be relevant
to the verbal behaviors reported on here. A second reason for omitting
these terms is that psychologists unfamiliar with or ignorant of the vocabulary
of behavior analysis may not be able to take advantage of this methodology
for examining what they would not refer to as either "verbal repertory"
or "intraverbal behavior," even though this association procedure
assesses the intraverbal repertoire. A vocabulary serves better as a bridge
between divergent systematic viewpoints than as a barrier between them.
Finally, and by no means least important, students seem to like it (Attkisson
& Snyder, 1975).
THE WORD ASSOCIATE TEST: PREPARATION, ADMINISTRATION, GRADING
The Word Associate Test has been developed to fill the need for a test
that can be easily and "objectively" graded, that can sample
broadly the content of a course, that can be readily prepared and that
avoids the multiple choice format. The format of the test is straightforward:
Stimulus terms--key words--are chosen from the material on which a class
is to be tested (e.g., terms in the lecturer's notes, in the body, index
or glossary of a text). Examinees are given the opportunity to demonstrate
their familiarity and grasp of each term by responding to it appropriately.
The stimulus terms selected are arrayed vertically; the number of responses
required is chosen by the examiner. Instructions are brief; grading is
similarly uncomplicated. Each response is graded on a scale chosen by the
instructor. The distribution of summed scores is the basis on which letter
grades can be assigned. This presentation will provide the readers with
the information they will need if they wish to use this examination format.
A monograph summarizing the findings of research on this method of examination
is in preparation. Appendices present several WAT's, completed and graded,
as well as other findings.
Introducing the WAT to Students
Before giving a WAT ("The Short-short Answer Test") as a grade
determining tool, the instructor should give the students one or more brief
practice sessions. These serve as a quick review for the students, showing
them what they need to study more, as well as giving them an opportunity
for them to know how the method works on a test.
Construction
Instructions
The instructions below have proven easy to follow. A briefer form may
be used at the top of each page of an examination. (See Appendices)
In each space provided for each item write a word, date, phrase, term,
name, or concept relevantly associated to it in the subject matter of this
course. Do not repeat. You may review your answers before passing your
paper in--you'll probably be able to fill in some of the spaces you left
blank.
References to grading need not occur in the instructions. Dry-runs,
advisable before the first administration, serve to desensitize students
to the novelty of the exam (new kinds of tests are always scary), as well
as to eliminate the possibility that a student has misconstrued the instructions.
Choice of Stimulus Terms
The number of stimulus terms in a given test will be determined by how
broad a sample the instructor wishes to examine on, how many responses
he or she wants, and on the time allotted for the test. When two responses
are required, students take approximately one minute per item to take the
test.
The instructor should choose the terms that he or she thinks the students
should know something about from lectures, discussions, or other assigned
readings. Stimulus terms can and should sample a wide range of the materials
covered in the class. Tables of contents, indexes of textbooks, and glossaries
are especially convenient sources. Boneau's paper (1990) is a useful source.
Overly general or vague terms should be avoided; anybody can give acceptable
associates to, say, "motivation," or "motherhood."
Also avoid names easily identified generically, e.g., "Schultz,"
which will elicit "German psychologist." Related terms, i.e.,
words that would be acceptable as responses to one another, may both be
included, but should be presented distantly from one another on the examination
sheet.
The wide range of sources for stimulus items allows the test to be finely
tuned to the content and level of the course. The test itself can be a
very specific probe of student knowledge in given subareas. Eventually
each instructor will be able to assemble a 'master list' of terms that
can be given to students as study guides--the concepts of which they should
be in command.
Appendix A lists, in alphabetic order, approximately 200 items from
a set of over 2,000 psychological terms that have been used in first courses
in General Psychology and Introduction to Biological Psychology at U.T.
Knoxville.
Format
Terms are arranged vertically, isolated from one another by lines. This
format discourages sentence writing, and increases the likelihood that
the second, third, and fourth associates are given to the stimulus term,
rather than to the immediately previous written response.
The student is usually required to produce four associates for each
stimulus term, although this number can vary as circumstances require.
Research indicates that four associates are ample to provide discriminatory
power for evaluating student knowledge. Fact is, two associates prove to
be enough to demonstrate basic knowledge. A test including 64 two-response
items can usually be completed by students within an hour.
Appendix B presents a number of test formats, some completed and graded.
Grading
Ideally, grading should begin with a clerical procedure tabulating together
the responses from all students with frequency counts. Responses can be
listed in order of frequency of occurrence and then graded. Appendices
D and E present such distributions. Note that unacceptable responses are
infrequent. Scores may then be transposed to the responses on each of the
original test papers. This procedure guarantees a high reliability in grading.
Lacking the computer technology, hand grading, as seen in Appendix D,
must be relied upon. It goes remarkably easily, and does not take as much
time overall as putting together an equivalent multiple choice exam.
Three methods have been used in grading, and inter-grader reliabilities
measured for each.
A. Four Point Scale for Grading
+2 - A good associate, relevant, clearly demonstrating familiarity with
the material.
+1 - Acceptable, but strained, ambiguous, overly general, or a reasonable
guess.
0 - No response, irrelevant, indeterminate or out of context.
-1 - Positively incorrect.
As an example of the four point system, scores would be assigned to
the responses below as follows:
Binet
Intelligence test +2
French +1
Stephen Vincent 0
Freud's student -1
B. Three Step Scale
The preceding system was simplified to a +2, +1, or 0 scale. The -1
score was eliminated since it seemed to measure the grader's response to
the inadequate response. Further, relative to the number of "correct"
or "acceptable" responses, few clearly incorrect rather than
irrelevant responses are given, so few associates appear that get graded
-1. Dropping the penalty for "wrong" answers proves not to affect
the grade curve.
On the 2,1,0 scale, do not limit the use of 2 to indicate the "exceptionally
good" response (corresponding to a -1, as somehow especially bad).
A score of "2" should mean that "no one could take exception"
to the associate, while a "1" indicates a response that is in
the "right direction," but not adequate to "pinpoint"
the stimulus term. With experience in grading, the grader will develop
both speed and confidence in his or her objectivity and consistency in
grading, especially if the paper graded is identified by student number,
rather than by name.
C. Simplest Scale: Two Steps
The third scale is the simplest; responses are graded 1 and 0, acceptable
or not.
Saeger (1975) states, "Review of the problem of item weighting
in general and the magnitude of the correlations between weighted versus
unweighted scoring of responses on the WAT in particular indicates that
a simplified scoring system of +1 for an acceptable associate (encompassing
+2 and +1) and 0 for an unacceptable associate (encompassing 0 and -1)
will provide sufficient estimates of test characteristics."
D. Count Score
Those who have had considerable experience reviewing WAT scores are
not surprised to find that simple counts of responses, without grading,
correlate highly with scores.
PSYCHOMETRIC PROPERTIES
Reliability
Variability distributions of scores made in a class will show a broad
range of performances--from those who fill in almost every blank correctly,
to those whose answer sheets are less than half full.
Research on the Tennessee WAT indicates that the procedure is acceptably
reliable. Various coefficients of reliability range from +.592 to +.899.
See appendix F.
Validity
The measurement of the validity of any examination of a student's achievement
in a course is a matter of faith. The question of the criterion or criteria
against which the validity of any one test is to be evaluated is very much
open to debate, the question arising just as critically with essay, short
answer, take-home, multiple choice exams as well as with the WAT. Appendix
F presents validity coefficients of WAT scores with the essay, short answer,
and GPA. "Validity" estimates range from .447 (WAT and overall
GPA) to +.773 (WAT and Short Answer Test) depending on the criterion. Most
coefficients are between +.45 and +.77 and are thus well within the acceptable
range.
Those who developed the test have done so in many smallish (10-20) classes
in both beginning and advanced Honors Psychology groups, in which it is
possible to get to know the students and their performances reasonably
well. One is very seldom surprised by scores on WAT, and then it is by
the score of the shy, quiet student who performs very well indeed on the
Associate Test.
References
Attkisson, C. C., & Snyder, C. R. (1975) Student evaluation of mulitple
choice and word association exams. Journal of Instructional Psychology,
2, 9-15.
Boneau, C. A. (1990). Psychological literacy: A first approximation.
American Psychologist, 45, 891-900.
Jennings, W. N. (1966). Originality as a function of uncommon associations
to common stimulus words. Unpublished master's thesis. University of
Tennessee.
Jennings, W. N. (1968). The assessment of instructor differences
using the Word-association examination. Unpublished doctoral dissertation.
University of Tennessee.
Rosanoff, A. J., Martin, H. E. and Rosanoff, I. R. (1918). A higher
scale of mental measurement and its application to cases of insanity. Psychological
Monographs. Princeton, N.J.
Shelnutt, J. B. (1969). Some characteristics of the Word-Association
Test used as a measure of academic knowledge with respect to individual
differences in student's patterns of responding, difficulty of stimulus
items, and heterogenity of the response distributions. Unpublished
honors thesis. University of Tennessee.
Saeger, W. (1975) An associative evaluation of selected interdisciplinary
courses. Unpublished doctoral dissertation. University of Tennessee.
Skinner, B. F. (1957) Verbal behavior. New York: Appleton-Century-Crofts.
Appendix A
List of 192 words of (approx.) 2500 on which students in a beginning
General Psychology course have been examined using the Word Associate Test.
ablation aversive stimulus cell differentiation consciousness
abscissa axon hillock central limit theorem consistency
accountants Babinski central tendency consolidation theory
Ach balance theory centrality index consonance dissonance
acquired drive balloon cephalocaudal constancy
acquisition Bartlett cerebrotonic constant method
activation syndrome basic anxiety cerebrum constitutional types
active vocabulary Beach chained schedule construct validity
actuarial behavior therapy chance errors consummatory act
adaptation behaviorism character consummatory behavior
adaptation level Bekesy Charcot content analysis
adipsia Bekhterev checking force content psychology
adrenal Bell Chomsky context theory
adrenalin Berger choroid contiguity
affectional drive Berkeley chromosome contraction
affective congruency beta weights chronograph control
all or none Binet classical convergent hierarchy
alpha biosocial theory classical conditioning core and context
alpha rhythm bipolar cell clinical interaction corporal data
Ames bit closure correlation
amnesia black headed gull clustering correlation asch
ampulla blast injection cognitive change correspondence
amygdala blockings cognitive dissonance cortex
analyzer Boring cognitive maps cortical steroid
animal consciousness Bradley cognitive theory corticospinal
anticathexis brain cohesiveness counter conditioning
anticipation error brain washing collective unconscious counterbalancing
anxiety Brentano color blindness counterconformity
anxiety reduction Breuer color solid cranial nerves
aphagia brightness color vision Crespi-effect
aphalic index brightness constancy common elements cretinism
approach gradient Broadbent's model common trait criterion
apraxia Broca comparative judgement criterion analysis
Aquinas Broca's area compromise response critical period
area sampling Brunswik computers cross validation
arousal Bunsen-Roscoe law concept formation csr
association test C. Lloyd Morgan conditional achievement cultural conflict
asymptote canalization conditioned elements cumulative record
atomism Cannon conditioned inhibition cumulative recorder
atropine case history conditioned response curare
attensity castes cones curiosity
attention castration complex conflict dark adaptation
attenuation catatonic conformity dat
authoritarian catharsis confounding ddd syndrome
autism cathexis congruity incongruity De La Mettrie
autokinetic Cattell conjunction decerebrate rigidity
automated teaching cell assembly connector neuron decibel
autonomic cell body connotative meaning decision theory
Appendix B-1
Completed Test
Psychology 251-B
Honors General Psychology
COURSE EVALUATION EXAM
Short-Answer Score 115
affective disorder
|
1
|
affects behavior
|
|
group therapy
|
2
|
method used for treatment for abnormal people
|
|
personalized systems of instruction
|
0
|
small instructor student ratio
|
|
socialization
|
1
|
sociology
|
|
affective disorder
|
1
|
noticeable disorder
|
|
group therapy
|
1
|
psychiatrist
|
|
personalized systems of instruction
|
0
|
better environment for learning
|
|
socialization
|
2
|
behavior with other organisms
|
|
aptitude
|
2
|
ability to do something
|
|
|
phenomenology
|
2
|
naturalism
|
|
statistical inference
|
2
|
conclusion drawn from statistics
|
|
aptitude
|
2
|
intelligence test
|
|
|
phenomenology
|
2
|
type of psychology
|
|
statistical inference
|
1
|
human error
|
|
authoritarian personality
|
0
|
someone in charge
|
|
|
prejudice
|
2
|
discrimination
|
|
|
authoritarian personality
|
0
|
instructor
|
|
|
|
test battery
|
2
|
Binnel test
|
|
behavior therapy
|
2
|
Watson
|
|
introvert
|
0
|
examining self
|
|
|
unconditioned stimulus (USCS)
|
1
|
automatic response
|
|
behavior therapy
|
2
|
Skinner
|
|
introvert
|
0
|
looking into self
|
|
|
unconditioned stimulus (USCS)
|
0
|
|
|
claustrophobia
|
2
|
fear of closed area
|
|
|
|
|
claustrophobia
|
0
|
psycosic
|
|
|
|
|
computer simulation
|
2
|
CAL
|
|
|
rapid eye movements (REMS)
|
2
|
sleep
|
|
|
computer simulation
|
2
|
learning
|
|
|
rapid eye movements (REMS)
|
2
|
dreams
|
|
wire mother
|
2
|
monkey development
|
|
Down's syndrome
|
2
|
genetic disorder
|
|
negative reinforcer
|
2
|
extinction
|
|
reinforcing stimulus
|
2
|
positive reward
|
|
species repertory
|
2
|
common for particular species
|
|
Down's syndrome
|
2
|
mongoloid
|
|
negative reinforcer
|
0
|
punishment
|
|
reinforcing stimulus
|
2
|
increase particular behavior
|
|
|
experimental design
|
1
|
experimental setup
|
|
obsessive-compulsive reaction
|
0
|
psycosis
|
|
|
|
experimental design
|
1
|
maze
|
|
obsessive-compulsive reaction
|
2
|
compulsive eater
|
|
|
|
|
paired-associate learning
|
2
|
form of learning
|
|
self-regulation
|
2
|
nervous system
|
|
unconditioned response
|
2
|
natural response
|
|
frustration
|
2
|
defense mechanisms
|
|
paired-associate learning
|
0
|
Skinner's box
|
|
self-regulation
|
2
|
automatic
|
|
unconditioned response
|
1
|
automatic
|
|
|
extinction
|
2
|
negative reinforcement
|
|
|
behavior repertory
|
0
|
common behavior
|
|
|
extinction
|
2
|
end of particular trait
|
|
|
|
extrasensory perception
|
2
|
ESP
|
|
|
Retroactive Interference
|
0
|
|
|
|
extrasensory perception
|
2
|
parapsychology
|
|
nonsense-syllable
|
1
|
short term memory
|
|
Retroactive Interference
|
0
|
|
|
|
ethology
|
2
|
study of behaviour in organisms other than man
|
|
modeling
|
2
|
way to change behavior
|
|
"Black Box"
|
2
|
Stimulus Response
|
|
|
|
modeling
|
2
|
used in treatment of phobia
|
|
"Black Box"
|
2
|
Behaviorist
|
|
|
Appendix D-1
Response distribution to the Stimulus Term 'Cone'.
N students = 166; N possible responses = 664
Associates with F less than 5 ommitted
Associate: F RF Associate: F RF
Primary Intraverbal Primary Intraverbal
Response Response
rods 279 .42 3 types 11 .02
eye 254 .38 vis. acuity 9 .01
fovea 247 .37 green 9 .01
color 175 .26 night 9 .01
retina 156 .24 night vision 9 .01
vision 87 .13 in fovea 9 .01
color vision 75 .11 Young-Helmholtz 9 .01
light 42 .06 part of eye 8 .01
daylight 36 .05 yellow 8 .01
day vision 27 .04 daytime 8 .01
sight 26 .04 blind spot 8 .01
iodopsin 26 .04 optic nerve 8 .01
lt. sensitive 18 .03 cells 7 .01
3 colors 17 .03 in eyes 7 .01
receptors 16 .02 dark 7 .01
conc. in fovea 15 .02 rhodopsin 7 .01
three 14 .02 ice-cream 7 .01
day lt. vision 13 .02 Hering theory 7 .01
color blindness 13 .02 red 6 .01
yellow, green, blue 12 .02 sensitive 5 .01
blue 11 .02 blanks & lower 257 .39
perception 11 .02 frequency responses
Appendix D-2
How concepts change
Group A
Response distributions (four associates) to items appearing on
final examination of one class in General Psychology, and on "orientation"
pretest of another. Number of blanks and low-frequency responses omitted.
N = number of students.
1. Stimulus word: CONTROL
Pretest (N=87) Final
(N=115)
Response F RF Response F RF
Power (over) 31 .36 Group 58 .50
Experiment 21 .24 Experimental 54 .47
Sociological Factors 15 .17 Experi(ments, tation) 35 .30
Variables 13 .15 Independent Variable 20 .17
Self 13 .15 Variables 16 .14
Emotions 12 .14 No Special Treatment 15 .13
Stimulus (&/or) Response 12 .14 Generic 11 .09
Mind 11 .12 Baseline (basis) 9 .08
Behavior 8 .09 Validity 7 .06
Groups 7 .08 Necessary 6 .05
Reward 4 .05
Test 4 .05
Biological 4 .05
2. Stimulus word: CENTRAL TENDENCY
Pretest (N=87) Final
(N=115)
Response F RF Response F RF
Habits, Drives, etc. 15 .17 Mode 96 .83
X, Median, Mode 12 .13 Median 96 .83
Stimulus (&/or) Response 9 .10 Mean 84 .73
Cause 7 .08 Average 14 .12
Most Likely 7 .08 Standard Deviation (S.D.) 11 .10
Physiology Frequency Distribution 9 .08
(brain, nervous system) 6 .07 Graph 6 .05
Inward Feelings 6 .07
Emotional
(love, hate, anger) 5 .06
Standard Deviation 4 .05
Average 4 .05
Behavior 4 .05
Group B
Response distribution (four associates) to items appearing on the
pretest and final examination of the same class.
1. Stimulus Word: HABITUATION
Pretest (N=68) Final
(N=60)
Response F RF Response F RF
Environment 10 .15 Conditioning or Learning 33 .55
Habit 5 .07 Clothes 13 .21
Continue 3 .04 Adaptation 11 .18
Stimulus 2 .03 Perception 9 .15
Desires 2 .03 Getting used to 7 .12
Need 2 .03 Accustomed 7 .12
Home 2 .03 Unaware(ness) 6 .10
Life 2 .03 Behavior 5 .08
Condition 2 .03 Adjustment 5 .08
Response 2 .03 Response 5 .08
Exhaustion 2 .03 Environment 4 .07
Reflex 2 .03 Motivation 4 .07
Habit Formation 4 .07
Repeated Stimuli 4 .07
Experience 3 .05
Babinski 3 .05
2. Stimulus Word: HOMEOSTASIS
Pretest (N=68) Final
(N=60)
Response F RF Response F RF
Stable 3 .04 Balance 34 .57
Internal Environment 3 .04 Autonomic 17 .28
Map 2 .03 Optimal 12 .20
Dynamic 2 .03 Equilibrium 11 .18
Tension 9 .15
Behavior 6 .10
Temperature 6 .10
Internal 4 .07
3. Stimulus Term: FUNCTIONAL AUTONOMY
Pretest (N=68) Final
(N=60)
Response F RF Response F RF
Movement 4 .06 Nervous System 17 .28
Muscles 2 .03 Motivation 8 .13
Self 2 .03 Miser 7 .11
Involuntary 2 .03 Homeostasis 3 .05
Nerves 2 .03
4. Stimulus Term: ROD-CONE BREAK
Pretest (N=68) Final
(N=60)
Response F RF Response F RF
Eye 15 .22 Dark Adaptation 23 .38
Color 5 .07 Rods-cones 15 .25
Sight 2 .03 Rhodopsin 14 .23
Nerve 2 .03 Day-night vision 13 .22
Back of Eyeball 2 .03 Twilight vision 12 .20
Purkinje (shift, effect
or phenonomenon) 11 .18
Eyes 7 .12
Red (lenses, vision,
goggles) 7 .12
Vision(ual) 6 .10
Threshold 6 .10
Appendix D-3
Psychology-Zoology 472. Winter Quarter 1963-1964
Item Difficulty List - Final Exam
Maximum points per item (8 X 20) = 160
Four responses; graded 2, 1, 0, -1
Total Total
Score Score
on on
item RF Stimulus word item RF Stimulus word
157 .98 peck-order 101 .63 consummatory act
149 .93 threat 100 .63 habituation
148 .93 stickleback 99 .62 mobbing
146 .91 queen substance 97 .61 population homeostasis
145 .91 jackdaw 96 .60 spacing
144 .90 paleontology 95 .59 cortical steroid
144 .90 gonadotropins 93 .58 trophallaxis
143 .89 innate 92 .58 sensitization
141 .88 pheromones 88 .55 "local dialects"
140 .88 castes 87 .54 Tribolium confusum
139 .87 estrogen 87 .54 intra-specific fighting
139 .87 waggle dance after mutual recognition
138 .86 population pressure 84 .53 discriminative stimulus
138 .86 freezing 81 .51 interspecific signals
138 .86 instinct 77 .48 general adaptation syndrome
138 .86 dominance-subordination 76 .48 vasopression
137 .86 immobilization 76 .48 ecological niche
135 .84 comparative method 73 .46 MacDougall
134 .84 sage grouse 67 .42 ecosystem
134 .84 imprinting 67 .42 king
134 .84 hormones of anterior pituitary 66 .41 vacuum activity
132 .83 classical conditioning 59 .37 cockatoo
131 .82 hypothalamus 58 .36 personal distance
129 .81 linear organization 54 .34 individual distance
127 .79 reciprocal preening 53 .33 social reflex no. 1
126 .79 baboon harem 52 .33 internal releasing mechanisms
125 .78 permanent mateship 51 .32 paradigm
120 .75 Mesozoic 50 .31 genetic control
119 .74 supplementary reproductives 48 .30 activation syndrome
118 .74 rate of response 37 .23 epideictic
118 .74 hypertrophy of sex glands 35 .22 ecdyson
117 .73 master gland 32 .20 genetic drift
115 .72 scouts 28 .18 setting operations
114 .71 interspecific recognition 28 .18 corpora alata
of calls 28 .18 subterritory
112 .70 phylogeny 25 .16 French corvids
111 .69 LH 24 .15 repletes
105 .66 operant 18 .16 cycling in population
104 .65 home range 10 .06 statary phase
104 .65 mood 6 .04 lek
104 .65 response repetoire 2 .01 local enhancement
101 .63 S-R; S 0 .00 environmental continum
Appendix D-4
The recoverability of
Stimulus terms
The left-hand column gives the response-distributions from which
the "key words" were chosen. The right-hand column gives the
response-distributions obtained when the subjects were given one blank
to fill in, with an instruction; find the "key word" that links
together the four words presented.
For Group I, the key-words were selected from the response-distributions
given by the same class on a previous exam.
For Group II, the key-words were selected from the response-distributions
given by a class other than the one seeking the key-word.
I (a) 212-2a N=74 212-2a N=74
4 Associates to Stimulus Word: "Key Word" to "G.&S., bumps,
Phrenology localization, pseudo science."
Response N RF Response N RF
Gall and/or Spurz. 59 .80 phrenology 62 .84
localization 48 .65 repetition of S 2 .03
pseudo science 48 .65 others 10 .13
bumps 37 .50
size 20 .27
brain 16 .22
trait char. 13 .18
I (b) 212-2a N=74 212-2a N=74
4 Associates to Stimulus Word: "Key Word" to "ablation, removal,
Extirpation localization, lesions."
Response N RF Response N RF
ablation 44 .60 extirpation 19 .36
removal 38 .51 brain damage 11 .31
study-investigate 26 .35 brain 10 .13
lesions 23 .31 lobotomy 7 .09
localization 21 .27 brain functions 6 .08
brain 13 .18 repetition 2 .03
drugs 6 .08 others 19 .36
Names-(Lashley, Magoun, etc.) 6 .08
II (a) 212-1 N=33 212-2b N=81
4 Associates to Stimulus Word: "Key Word" to "spatial, temporal,
Summation threshold, firing of neuron."
Response N RF Response N RF
spatial 18 .55 summation 35 .43
temporal 18 .55 stimulus 10 .20
threshold 15 .45 others 40 .37
compound stimuli 8 .24
firing of neurons 5 .15
synapse 5 .15
addition of output 3 .09
intensity 3 .09
neuron 3 .09
stimulation 2 .06
II (b) 212-1 N=33 212-2b N=81
4 Associates to Stimulus Term: "Key Word" to "neuron, nodes of
Myelinated Fibers Ranvier, insulation, faster trans."
Response N RF Response N RF
neuron 26 .70 myelin(ated) 57 .70
faster transmission 20 .61 (covering, sheath, tissue,
axon covering, shield 20 .61 fibers, insulation) others 21 .30
sheath 19 .58
insulation 6 .18
nodes of Ranvier 4 .12
around axon 2 .06
not always present 2 .06
Appendix E-1
Frequency Relative to Total Possible
Number of Associates to Each of Four Related Terms
Stimulus Term Harlow Cloth Wire Affectional
mother mother drive
(n = 153) (n = 149) (n = 181) (n = 179)
Harlow 1.00 .28 .19 .03
cloth mother .28 - .33 .08
wire mother .22 .30 - .07
experiment .06 .06 .07 .00
Sci. American .06 .01 .00 .00
behavior .05 .00 .00 .01
imprinting .04 .08 .08 .07
instinct .04 .03 .05 .09
rats .04 .00 .01 .02
motivation .03 .00 .03 .20
monkey baby .03 .03 .04 .00
wire .03 .06 .00 .00
cloth .03 .00 .04 .00
maternal .02 .03 .00 .14
surrogate .02 .03 .06 .00
maternal instinct .02 .02 .03 .03
feeding .01 .03 .06 .00
Hebb .01 .03 .02 .01
love .00 .03 .02 .28
sex .00 .00 .00 .12
security .00 .09 .02 .02
emotion .00 .00 .03 .08
innate .00 .00 .00 .10
fear .00 .01 .06 .02
affection .00 .04 .04 .00
hypothalamus .00 .00 .00 .07
protection .00 .03 .03 .00
Each stimulus term was given to approximately 1/4 of the same
class on a final examination. Four responses were possible.
Appendix E-2
The probabilistic relationship of
primary intraverbals in a group
133 students took this WAT; all stimulus terms were presented four (4)
times. The terms in squares were presented as stimuli in the WAT. The encircled
terms appeared as responses. Arrows connecting S and R give relative frequency
of occurrences of each response relative to the maximum possible number
(532). The careful reader will note that one response that occurred relatively
frequently is not appropriate ("wrong"). Investigation showed
that this word appeared in a paragraph of the text book that included the
stimulus term and two or three "correct" associates. (Draw your
own conclusions.)
Appendix F-1
Some reliability and validity measures of the WAT and
other tests
Introductory Psychology, Honors Section (not an exhaustive set)
- Reliability
- Split half
- First half-second half: r = +.724
- Odd-Even: r = +.686
- Ranked difficulty of terms: r = +.683
- Kuder-Richardson forumula 20
- Original test (36 items): r = +.724
- Original test (1st 20 items only): r = +.592
- Retest (20 items): r = +.736
- Test-retest (20 items)
- Students' scores: r = +.624
- item difficulty: r = +.899
- Validity
- Rank-order correlation between test part scores
- Essay and short answer: +.804
- Association and short answer: +.733
- Association and essay: +.602
- Correlation between GPA and scores of:
- Association part: r = +.447
- Entire test: r = +.382
- Short answer part: r = +.265
- Essay part: r = +.254
- Correlation between position on Dean's List and rank of score on:
- Association part: r = +.678
- Entire test: r = +.677
- Short answer part: r = +.609
- Essay part: r = +.409
Appendix F-2
Some correlations on
five examinations
N = No. students n = No. st terms 4 resp; 2, 1, 0, -1 grading
1. General Psych., weekly quiz 2. General Psych., Honors Final Exam
N = 133 n = 25 N = 25 n = 36
WAT: Mean score 137.0 SD = 23.3 M = 180.2 SD = 28.2
% of max score possible = 68.5 % of max score = 62.6
GPA: Mean 2.30 SD = 0.54 M = 3.44 SD = 0.28
int. consist (KR) r = 0.816 (KR) r = 0.724
WAT with GPA r = 0.356 WAT with GPA rho = 0.447
WAT with M.C. r = 0.459 WAT with short ans. rho = 0.773
M.C. with GPA r = 0.359 WAT with essay rho = 0.602
Short ans. with GPA rho = 0.265
Essay with GPA rho = 0.254
3. Comparative Psych., Upper Div. 4. History and Systems, Upper Div.
N = 20 n = 81 N = 28 n = 49
WAT: Mean score 388.1 SD = 69.3 M = 218.39 SD = 78.1
% of max score possible = 59.9 % of max score = 65.8
GPA: Mean score 2.52 SD = 0.44 M = 2.73 SD = 0.87
WAT with GPA rho = 0.647 WAT with GPA rho = 0.455
Appendix G-1 Research on Honors Comprehensive Examination.
Research on Undergraduate Honors Comprehensive examinations given
at the end of the first year of the Honors program. If a student does not
pass at that time he or she is examined the following year. The Comprehensive
Examination has ten subsections, of which History and Systems is one. Different
samples of terms are prepared for each administration. The Comprehensive
Exam was taken by Advanced Honors Students at the end of the Junior Year,
which had been devoted to intensive readings, with weekly seminars.
The same examination was given experimentally to graduate students
in Psychology at the Master's level to replace a departmentally prepared
ten-section multiple-choice examination. It was replaced by the appropriate
Graduate Record Examination, which placed no demands on faculty time.