Verplanck, W.S. (1992) A brief introduction to the Word Associate Test. The Analysis of Verbal Behavior, 10, 97-123.


A Brief Introduction to
THE WORD ASSOCIATE TEST

William S. Verplanck

University of Tennessee, Knoxville

An examination format assessing the intraverbal repertoire of individuals in psychology is described and results using it reported. The Associate Test is easy to prepare, to take, and to grade. Its reliability measures are satisfactory; its ability to predict later behavior is reported upon. The Associate Test is computer friendly, and its methods can be applied for examination in any field, and at any level.

The writer would like to acknowledge gratefully the work of J.R. Caplan, R. Fleischer, A. Gordon, W.N. Jennings, R.N. Krueger, R. A. Ramos, W. Saeger, and J. Shelnutt. P.J. Lazzara and R.D. Kinser assisted in summarizing this research and in assembling the appendices.

The Word Associate Test (WAT) (or Short-Short Answer Test) is a format for preparing brief tests designed to demonstrate the student's command of the critical terms of specific areas of study. It provides a measure of the student's vocabulary in specialized and technical areas. That is, it samples the minimal intraverbal repertoires of the individual

The Word Associate Test has been developed over some 30 years of research. [The progressive development of the Associate Test and its methodology into its present form have been reported at meetings of The Psychonomic Society (6), of the Southeast Psychol. Ass'n. (6), and of the American Education Research Ass'n. More detailed reports have appeared in 1 Honors Paper, 2 MA Theses, and 2 Doctoral Dissertations. A listing of these papers may be requested of the author.] This work was instigated by dissatisfaction with the multiple-choice test, whose scores are patently associated with specific content-independent verbal skills and with well shaped monents for guessing.

The primary objective of the WAT is the rapid examination and grading of large numbers of students in one or another of a broad variety of areas (Jennings, W.N., 1968; Saeger, W., 1975), and at a number of levels of achievement. It was not developed to find how people think, how original [Jennings (1966) has nevertheless shown that there is a relationship between word-associate measures and evaluation of the "originality" shown in compositions written for the instructors in The English Department at UTK. These instructors had no information whatever on the WAT, much less of each student's performance.] they are, how well they can explain something, or write, much less to predict academic success, life achievement, or aptitude for something or other. The command of specified vocabularies is, however, necessary for all these performances.

The WAT asks only one question, which has many paraphrases in ordinary language: "How successfully has he/she studied?", "What words can the teacher or writer use and expect the students to be able to deal with at an acceptable level?", and loosely, "How much does the student know?". It does so without the use of the prompts that appear in multiple-choice, true-false, matching, and fill-in-the-blanks examination items. Incorrect prompts may indeed be added to the examinee's intraverbal repertoire.

The WAT may be termed "robust": Instructions to guess do not increase scores; instructions "Do NOT guess," however, decrease them.

The WAT demonstrates the examinees minimum level of command of a selected vocabulary. The WAT does not enable students to demonstrate their full knowledge. For this, a further WAT, presenting a more advanced technical vocabulary should be taken. In essence, a WAT can be prepared for any level of study, from kindergarten through the doctorate, in any field (with the possible exception of some branches of mathematics).

Although the test was developed before computer technology had reached its present level of sophistication and accessibility, it was preadapted for computer administration, grading, and presentation of results. Computer technology will enable a student to take an Associate Test on any subject at any level of achievement at any time at any terminal with immediate scoring against national norms.

It may now be applied to evaluate the achievement of individual students or specific classes and the effectiveness of various teaching methods, or of instructors. (Jennings, 1968; Saeger, 1975; Shelnutt, J., 1969) The description which follows gives the present status of the Word Associate Test. The appendices present sets of data relevant to the use of the test and to how it works in practice. An extended monograph on the WAT is now in preparation, and is to be published by the Cambridge Center for Behavioral Studies. This will report on the research on the WAT carried out to date, and will indicate its potential as a research instrument as well as a method for examination of substantive command of subject matter. Users will evaluate its face validity on the basis of their experience with its use and with the students who have taken it.

Given the current state of understanding of relevant criteria and of "psychometric validity," the psychometric validity of the WAT is no better or worse than that of any other form of examination, whether essay, take-home, true-false, or multiple choice. It can and will be compared with others at such time as acceptable sets of well-defined criteria can be agreed upon.

Although the WAT has been independently developed, the readers of Rosanoff (1918), Deese and Loftus will find it of both practical and theoretical importance. The reader will also recognize that "Word Associate Test" is perhaps too restricted a name. Besides words, names, and dates, one may also use equations, curves, chemical formulas, diagrams and other visual material as stimulus items in preparing examinations using this design, and examinees may respond by drawing pictures and sketching graphs or diagrams, as some have done.

When the response given to the stimulus word or term is verbal, rather than a diagram, sketch, or graph, the associate may be considered a primary or first-order intraverbal, one that is given directly to the stimulus. These are the responses most likely to appear in sentences written when the examinee is asked to write brief essays on the term. In these answers, the first term to appear is most often the stimulus term itself--in this context, a notate, or tact.

The possibility of the use of stimuli other than written or spoken words, such as pictures, diagrams, sounds, and so on, is one reason why the term "associate," rather than "intraverbal behavior" (Skinner, B.F., 1957), is used. Certainly the specific methodology presented deals with intraverbal behavior. But the methodology lends itself equally well to the investigation of tacts, as distinguished from "intraverbal responses." Word associates are intraverbal behaviors only when the stimuli themselves are words or phrases; otherwise, they are tacts. In Skinner's Verbal Behavior, the terms "tact" and "intraverbal response" have extensive connotations, which may or may not be relevant to the verbal behaviors reported on here. A second reason for omitting these terms is that psychologists unfamiliar with or ignorant of the vocabulary of behavior analysis may not be able to take advantage of this methodology for examining what they would not refer to as either "verbal repertory" or "intraverbal behavior," even though this association procedure assesses the intraverbal repertoire. A vocabulary serves better as a bridge between divergent systematic viewpoints than as a barrier between them.

Finally, and by no means least important, students seem to like it (Attkisson & Snyder, 1975).

THE WORD ASSOCIATE TEST: PREPARATION, ADMINISTRATION, GRADING

The Word Associate Test has been developed to fill the need for a test that can be easily and "objectively" graded, that can sample broadly the content of a course, that can be readily prepared and that avoids the multiple choice format. The format of the test is straightforward: Stimulus terms--key words--are chosen from the material on which a class is to be tested (e.g., terms in the lecturer's notes, in the body, index or glossary of a text). Examinees are given the opportunity to demonstrate their familiarity and grasp of each term by responding to it appropriately. The stimulus terms selected are arrayed vertically; the number of responses required is chosen by the examiner. Instructions are brief; grading is similarly uncomplicated. Each response is graded on a scale chosen by the instructor. The distribution of summed scores is the basis on which letter grades can be assigned. This presentation will provide the readers with the information they will need if they wish to use this examination format. A monograph summarizing the findings of research on this method of examination is in preparation. Appendices present several WAT's, completed and graded, as well as other findings.

Introducing the WAT to Students

Before giving a WAT ("The Short-short Answer Test") as a grade determining tool, the instructor should give the students one or more brief practice sessions. These serve as a quick review for the students, showing them what they need to study more, as well as giving them an opportunity for them to know how the method works on a test.

Construction

Instructions

The instructions below have proven easy to follow. A briefer form may be used at the top of each page of an examination. (See Appendices)

In each space provided for each item write a word, date, phrase, term, name, or concept relevantly associated to it in the subject matter of this course. Do not repeat. You may review your answers before passing your paper in--you'll probably be able to fill in some of the spaces you left blank.

References to grading need not occur in the instructions. Dry-runs, advisable before the first administration, serve to desensitize students to the novelty of the exam (new kinds of tests are always scary), as well as to eliminate the possibility that a student has misconstrued the instructions.

Choice of Stimulus Terms

The number of stimulus terms in a given test will be determined by how broad a sample the instructor wishes to examine on, how many responses he or she wants, and on the time allotted for the test. When two responses are required, students take approximately one minute per item to take the test.

The instructor should choose the terms that he or she thinks the students should know something about from lectures, discussions, or other assigned readings. Stimulus terms can and should sample a wide range of the materials covered in the class. Tables of contents, indexes of textbooks, and glossaries are especially convenient sources. Boneau's paper (1990) is a useful source.

Overly general or vague terms should be avoided; anybody can give acceptable associates to, say, "motivation," or "motherhood." Also avoid names easily identified generically, e.g., "Schultz," which will elicit "German psychologist." Related terms, i.e., words that would be acceptable as responses to one another, may both be included, but should be presented distantly from one another on the examination sheet.

The wide range of sources for stimulus items allows the test to be finely tuned to the content and level of the course. The test itself can be a very specific probe of student knowledge in given subareas. Eventually each instructor will be able to assemble a 'master list' of terms that can be given to students as study guides--the concepts of which they should be in command.

Appendix A lists, in alphabetic order, approximately 200 items from a set of over 2,000 psychological terms that have been used in first courses in General Psychology and Introduction to Biological Psychology at U.T. Knoxville.

Format

Terms are arranged vertically, isolated from one another by lines. This format discourages sentence writing, and increases the likelihood that the second, third, and fourth associates are given to the stimulus term, rather than to the immediately previous written response.

The student is usually required to produce four associates for each stimulus term, although this number can vary as circumstances require. Research indicates that four associates are ample to provide discriminatory power for evaluating student knowledge. Fact is, two associates prove to be enough to demonstrate basic knowledge. A test including 64 two-response items can usually be completed by students within an hour.

Appendix B presents a number of test formats, some completed and graded.

Grading

Ideally, grading should begin with a clerical procedure tabulating together the responses from all students with frequency counts. Responses can be listed in order of frequency of occurrence and then graded. Appendices D and E present such distributions. Note that unacceptable responses are infrequent. Scores may then be transposed to the responses on each of the original test papers. This procedure guarantees a high reliability in grading.

Lacking the computer technology, hand grading, as seen in Appendix D, must be relied upon. It goes remarkably easily, and does not take as much time overall as putting together an equivalent multiple choice exam.

Three methods have been used in grading, and inter-grader reliabilities measured for each.

A. Four Point Scale for Grading

+2 - A good associate, relevant, clearly demonstrating familiarity with the material.
+1 - Acceptable, but strained, ambiguous, overly general, or a reasonable guess.
0 - No response, irrelevant, indeterminate or out of context.
-1 - Positively incorrect.

As an example of the four point system, scores would be assigned to the responses below as follows:

Binet

  Intelligence test  +2
  French             +1
  Stephen Vincent     0
  Freud's student    -1

B. Three Step Scale

The preceding system was simplified to a +2, +1, or 0 scale. The -1 score was eliminated since it seemed to measure the grader's response to the inadequate response. Further, relative to the number of "correct" or "acceptable" responses, few clearly incorrect rather than irrelevant responses are given, so few associates appear that get graded -1. Dropping the penalty for "wrong" answers proves not to affect the grade curve.

On the 2,1,0 scale, do not limit the use of 2 to indicate the "exceptionally good" response (corresponding to a -1, as somehow especially bad). A score of "2" should mean that "no one could take exception" to the associate, while a "1" indicates a response that is in the "right direction," but not adequate to "pinpoint" the stimulus term. With experience in grading, the grader will develop both speed and confidence in his or her objectivity and consistency in grading, especially if the paper graded is identified by student number, rather than by name.

C. Simplest Scale: Two Steps

The third scale is the simplest; responses are graded 1 and 0, acceptable or not.

Saeger (1975) states, "Review of the problem of item weighting in general and the magnitude of the correlations between weighted versus unweighted scoring of responses on the WAT in particular indicates that a simplified scoring system of +1 for an acceptable associate (encompassing +2 and +1) and 0 for an unacceptable associate (encompassing 0 and -1) will provide sufficient estimates of test characteristics."

D. Count Score

Those who have had considerable experience reviewing WAT scores are not surprised to find that simple counts of responses, without grading, correlate highly with scores.

PSYCHOMETRIC PROPERTIES

Reliability

Variability distributions of scores made in a class will show a broad range of performances--from those who fill in almost every blank correctly, to those whose answer sheets are less than half full.

Research on the Tennessee WAT indicates that the procedure is acceptably reliable. Various coefficients of reliability range from +.592 to +.899. See appendix F.

Validity

The measurement of the validity of any examination of a student's achievement in a course is a matter of faith. The question of the criterion or criteria against which the validity of any one test is to be evaluated is very much open to debate, the question arising just as critically with essay, short answer, take-home, multiple choice exams as well as with the WAT. Appendix F presents validity coefficients of WAT scores with the essay, short answer, and GPA. "Validity" estimates range from .447 (WAT and overall GPA) to +.773 (WAT and Short Answer Test) depending on the criterion. Most coefficients are between +.45 and +.77 and are thus well within the acceptable range.

Those who developed the test have done so in many smallish (10-20) classes in both beginning and advanced Honors Psychology groups, in which it is possible to get to know the students and their performances reasonably well. One is very seldom surprised by scores on WAT, and then it is by the score of the shy, quiet student who performs very well indeed on the Associate Test.


References

Attkisson, C. C., & Snyder, C. R. (1975) Student evaluation of mulitple choice and word association exams. Journal of Instructional Psychology, 2, 9-15.

Boneau, C. A. (1990). Psychological literacy: A first approximation. American Psychologist, 45, 891-900.

Jennings, W. N. (1966). Originality as a function of uncommon associations to common stimulus words. Unpublished master's thesis. University of Tennessee.

Jennings, W. N. (1968). The assessment of instructor differences using the Word-association examination. Unpublished doctoral dissertation. University of Tennessee.

Rosanoff, A. J., Martin, H. E. and Rosanoff, I. R. (1918). A higher scale of mental measurement and its application to cases of insanity. Psychological Monographs. Princeton, N.J.

Shelnutt, J. B. (1969). Some characteristics of the Word-Association Test used as a measure of academic knowledge with respect to individual differences in student's patterns of responding, difficulty of stimulus items, and heterogenity of the response distributions. Unpublished honors thesis. University of Tennessee.

Saeger, W. (1975) An associative evaluation of selected interdisciplinary courses. Unpublished doctoral dissertation. University of Tennessee.

Skinner, B. F. (1957) Verbal behavior. New York: Appleton-Century-Crofts.


Appendix A

List of 192 words of (approx.) 2500 on which students in a beginning General Psychology course have been examined using the Word Associate Test.

ablation                aversive stimulus       cell differentiation    consciousness
abscissa                axon hillock            central limit theorem   consistency
accountants             Babinski                central tendency        consolidation theory
Ach                     balance theory          centrality index        consonance dissonance
acquired drive          balloon                 cephalocaudal           constancy
acquisition             Bartlett                cerebrotonic            constant method
activation syndrome     basic anxiety           cerebrum                constitutional types
active vocabulary       Beach                   chained schedule        construct validity
actuarial               behavior therapy        chance errors           consummatory act
adaptation              behaviorism             character               consummatory behavior
adaptation level        Bekesy                  Charcot                 content analysis
adipsia                 Bekhterev               checking force          content psychology
adrenal                 Bell                    Chomsky                 context theory
adrenalin               Berger                  choroid                 contiguity
affectional drive       Berkeley                chromosome              contraction
affective congruency    beta weights            chronograph             control
all or none             Binet                   classical               convergent hierarchy
alpha                   biosocial theory        classical conditioning  core and context
alpha rhythm            bipolar cell            clinical interaction    corporal data
Ames                    bit                     closure                 correlation
amnesia                 black headed gull       clustering              correlation asch
ampulla                 blast injection         cognitive change        correspondence
amygdala                blockings               cognitive dissonance    cortex
analyzer                Boring                  cognitive maps          cortical steroid
animal consciousness    Bradley                 cognitive theory        corticospinal
anticathexis            brain                   cohesiveness            counter conditioning
anticipation error      brain washing           collective unconscious  counterbalancing
anxiety                 Brentano                color blindness         counterconformity
anxiety reduction       Breuer                  color solid             cranial nerves
aphagia                 brightness              color vision            Crespi-effect
aphalic index           brightness constancy    common elements         cretinism
approach gradient       Broadbent's model       common trait            criterion
apraxia                 Broca                   comparative judgement   criterion analysis
Aquinas                 Broca's area            compromise response     critical period
area sampling           Brunswik                computers               cross validation
arousal                 Bunsen-Roscoe law       concept formation       csr
association test        C. Lloyd Morgan         conditional achievement cultural conflict
asymptote               canalization            conditioned elements    cumulative record
atomism                 Cannon                  conditioned inhibition  cumulative recorder
atropine                case history            conditioned response    curare
attensity               castes                  cones                   curiosity
attention               castration complex      conflict                dark adaptation
attenuation             catatonic               conformity              dat
authoritarian           catharsis               confounding             ddd syndrome
autism                  cathexis                congruity incongruity   De La Mettrie
autokinetic             Cattell                 conjunction             decerebrate rigidity
automated teaching      cell assembly           connector neuron        decibel
autonomic               cell body               connotative meaning     decision theory


Appendix B-1

Completed Test


Psychology 251-B
Honors General Psychology
COURSE EVALUATION EXAM
Short-Answer     Score 115

affective disorder

1

affects behavior

group therapy

2

method used for treatment for abnormal people

personalized systems of instruction

0

small instructor student ratio

socialization

1

sociology

affective disorder

1

noticeable disorder

group therapy

1

psychiatrist

personalized systems of instruction

0

better environment for learning

socialization

2

behavior with other organisms

aptitude

2

ability to do something

homeostasis

0

same level

phenomenology

2

naturalism

statistical inference

2

conclusion drawn from statistics

aptitude

2

intelligence test

homeostasis

1

stationary

phenomenology

2

type of psychology

statistical inference

1

human error

authoritarian personality

0

someone in charge

insight

2

aha!

prejudice

2

discrimination

test battery

1

dumb test

authoritarian personality

0

instructor

insight

2

thinking

prejudice

2

oppression

test battery

2

Binnel test

behavior therapy

2

Watson

introvert

0

examining self

psi

2

parapsychology

unconditioned stimulus (USCS)

1

automatic response

behavior therapy

2

Skinner

introvert

0

looking into self

psi

2

ESP

unconditioned stimulus (USCS)

0

claustrophobia

2

fear of closed area

LSD

2

drug

psychogenic

0

Weber's law

0

claustrophobia

0

psycosic

LSD

2

hallucinate

psychogenic

0

Weber's law

0

computer simulation

2

CAL

mesomorph

0

fat

rapid eye movements (REMS)

2

sleep

wire mother

2

monkey

computer simulation

2

learning

mesomorph

0

slow

rapid eye movements (REMS)

2

dreams

wire mother

2

monkey development

Down's syndrome

2

genetic disorder

negative reinforcer

2

extinction

reinforcing stimulus

2

positive reward

species repertory

2

common for particular species

Down's syndrome

2

mongoloid

negative reinforcer

0

punishment

reinforcing stimulus

2

increase particular behavior

species repertory

0

experimental design

1

experimental setup

obsessive-compulsive reaction

0

psycosis

Schema

0

V.I. schedule

0

experimental design

1

maze

obsessive-compulsive reaction

2

compulsive eater

Schema

0

V.I. schedule

0

frustration

2

stress

paired-associate learning

2

form of learning

self-regulation

2

nervous system

unconditioned response

2

natural response

frustration

2

defense mechanisms

paired-associate learning

0

Skinner's box

self-regulation

2

automatic

unconditioned response

1

automatic

Gestalt

0

extinction

2

negative reinforcement

Id

2

Frued

behavior repertory

0

common behavior

Gestalt

0

extinction

2

end of particular trait

Id

2

animal

behavior repertory

0

extrasensory perception

2

ESP

nonsense-syllable

0

7+2

Retroactive Interference

0

washoe

2

chimp

extrasensory perception

2

parapsychology

nonsense-syllable

1

short term memory

Retroactive Interference

0

washoe

2

sign language

ethology

2

study of behaviour in organisms other than man

modeling

2

way to change behavior

"Black Box"

2

Stimulus Response

ethology

1

monkeys

modeling

2

used in treatment of phobia

"Black Box"

2

Behaviorist



Appendix D-1

Response distribution to the Stimulus Term 'Cone'.
N students = 166; N possible responses = 664
Associates with F less than 5 ommitted
Associate:             F       RF    Associate:             F       RF
Primary Intraverbal                  Primary Intraverbal              
Response                             Response                         

rods                  279     .42    3 types                11     .02
eye                   254     .38    vis. acuity             9     .01
fovea                 247     .37    green                   9     .01
color                 175     .26    night                   9     .01
retina                156     .24    night vision            9     .01
vision                 87     .13    in fovea                9     .01
color vision           75     .11    Young-Helmholtz         9     .01
light                  42     .06    part of eye             8     .01
daylight               36     .05    yellow                  8     .01
day vision             27     .04    daytime                 8     .01
sight                  26     .04    blind spot              8     .01
iodopsin               26     .04    optic nerve             8     .01
lt. sensitive          18     .03    cells                   7     .01
3 colors               17     .03    in eyes                 7     .01
receptors              16     .02    dark                    7     .01
conc. in fovea         15     .02    rhodopsin               7     .01
three                  14     .02    ice-cream               7     .01
day lt. vision         13     .02    Hering theory           7     .01
color blindness        13     .02    red                     6     .01
yellow, green, blue    12     .02    sensitive               5     .01
blue                   11     .02    blanks & lower        257     .39
perception             11     .02      frequency responses


Appendix D-2

How concepts change

Group A
Response distributions (four associates) to items appearing on final examination of one class in General Psychology, and on "orientation" pretest of another. Number of blanks and low-frequency responses omitted. N = number of students.

1. Stimulus word: CONTROL
    Pretest (N=87)      Final (N=115)

Response                  F   RF       Response                   F    RF

Power (over)              31  .36      Group                      58  .50
Experiment                21  .24      Experimental               54  .47
Sociological Factors      15  .17      Experi(ments, tation)      35  .30
Variables                 13  .15      Independent Variable       20  .17
Self                      13  .15      Variables                  16  .14
Emotions                  12  .14      No Special Treatment       15  .13
Stimulus (&/or) Response  12  .14      Generic                    11  .09
Mind                      11  .12      Baseline (basis)            9  .08
Behavior                   8  .09      Validity                    7  .06
Groups                     7  .08      Necessary                   6  .05
Reward                     4  .05
Test                       4  .05
Biological                 4  .05

2. Stimulus word: CENTRAL TENDENCY
    Pretest (N=87)      Final (N=115)

Response                  F   RF       Response                   F    RF

Habits, Drives, etc.     15  .17       Mode                       96  .83
X, Median, Mode          12  .13       Median                     96  .83
Stimulus (&/or) Response  9  .10       Mean                       84  .73
Cause                     7  .08       Average                    14  .12
Most Likely               7  .08       Standard Deviation (S.D.)  11  .10
Physiology                             Frequency Distribution      9  .08
 (brain, nervous system)  6  .07       Graph                       6  .05
Inward Feelings           6  .07
Emotional
  (love, hate, anger)     5  .06
Standard Deviation        4  .05
Average                   4  .05
Behavior                  4  .05

Group B
Response distribution (four associates) to items appearing on the pretest and final examination of the same class.

1. Stimulus Word: HABITUATION
    Pretest (N=68)      Final (N=60)

Response                  F   RF       Response                   F    RF

Environment              10  .15       Conditioning or Learning  33   .55
Habit                     5  .07       Clothes                   13   .21
Continue                  3  .04       Adaptation                11   .18
Stimulus                  2  .03       Perception                 9   .15
Desires                   2  .03       Getting used to            7   .12
Need                      2  .03       Accustomed                 7   .12
Home                      2  .03       Unaware(ness)              6   .10
Life                      2  .03       Behavior                   5   .08
Condition                 2  .03       Adjustment                 5   .08
Response                  2  .03       Response                   5   .08
Exhaustion                2  .03       Environment                4   .07
Reflex                    2  .03       Motivation                 4   .07
                                       Habit Formation            4   .07
                                       Repeated Stimuli           4   .07
                                       Experience                 3   .05
                                       Babinski                   3   .05

2. Stimulus Word: HOMEOSTASIS
    Pretest (N=68)      Final (N=60)

Response                  F   RF       Response                   F    RF

Stable                    3  .04       Balance                    34  .57
Internal Environment      3  .04       Autonomic                  17  .28
Map                       2  .03       Optimal                    12  .20
Dynamic                   2  .03       Equilibrium                11  .18
                                       Tension                     9  .15
                                       Behavior                    6  .10
                                       Temperature                 6  .10
                                       Internal                    4  .07

3. Stimulus Term: FUNCTIONAL AUTONOMY
    Pretest (N=68)      Final (N=60)

Response                  F   RF       Response                   F    RF

Movement                  4  .06       Nervous System            17   .28
Muscles                   2  .03       Motivation                 8   .13
Self                      2  .03       Miser                      7   .11
Involuntary               2  .03       Homeostasis                3   .05
Nerves                    2  .03

4. Stimulus Term: ROD-CONE BREAK
    Pretest (N=68)      Final (N=60)

Response                  F   RF       Response                   F    RF

Eye                      15  .22       Dark Adaptation           23   .38
Color                     5  .07       Rods-cones                15   .25
Sight                     2  .03       Rhodopsin                 14   .23
Nerve                     2  .03       Day-night vision          13   .22
Back of Eyeball           2  .03       Twilight vision           12   .20
                                       Purkinje (shift, effect
                                             or phenonomenon)    11   .18
                                       Eyes                       7   .12
                                       Red (lenses, vision,
                                                 goggles)         7   .12
                                       Vision(ual)                6   .10
                                       Threshold                  6   .10


Appendix D-3

Psychology-Zoology 472. Winter Quarter 1963-1964
Item Difficulty List - Final Exam
Maximum points per item (8 X 20) = 160
Four responses; graded 2, 1, 0, -1
Total                                   Total
Score                                   Score
 on                                      on
item  RF  Stimulus word                 item  RF  Stimulus word
157  .98  peck-order                    101  .63  consummatory act
149  .93  threat                        100  .63  habituation
148  .93  stickleback                    99  .62  mobbing
146  .91  queen substance                97  .61  population homeostasis
145  .91  jackdaw                        96  .60  spacing
144  .90  paleontology                   95  .59  cortical steroid
144  .90  gonadotropins                  93  .58  trophallaxis
143  .89  innate                         92  .58  sensitization
141  .88  pheromones                     88  .55  "local dialects"
140  .88  castes                         87  .54  Tribolium confusum
139  .87  estrogen                       87  .54  intra-specific fighting
139  .87  waggle dance                              after mutual recognition
138  .86  population pressure            84  .53  discriminative stimulus
138  .86  freezing                       81  .51  interspecific signals
138  .86  instinct                       77  .48  general adaptation syndrome
138  .86  dominance-subordination        76  .48  vasopression
137  .86  immobilization                 76  .48  ecological niche
135  .84  comparative method             73  .46  MacDougall
134  .84  sage grouse                    67  .42  ecosystem
134  .84  imprinting                     67  .42  king
134  .84  hormones of anterior pituitary 66  .41  vacuum activity
132  .83  classical conditioning         59  .37  cockatoo
131  .82  hypothalamus                   58  .36  personal distance
129  .81  linear organization            54  .34  individual distance
127  .79  reciprocal preening            53  .33  social reflex no. 1
126  .79  baboon harem                   52  .33  internal releasing mechanisms
125  .78  permanent mateship             51  .32  paradigm
120  .75  Mesozoic                       50  .31  genetic control
119  .74  supplementary reproductives    48  .30  activation syndrome
118  .74  rate of response               37  .23  epideictic
118  .74  hypertrophy of sex glands      35  .22  ecdyson
117  .73  master gland                   32  .20  genetic drift
115  .72  scouts                         28  .18  setting operations
114  .71  interspecific recognition      28  .18  corpora alata
            of calls                     28  .18  subterritory
112  .70  phylogeny                      25  .16  French corvids
111  .69  LH                             24  .15  repletes
105  .66  operant                        18  .16  cycling in population
104  .65  home range                     10  .06  statary phase
104  .65  mood                            6  .04  lek
104  .65  response repetoire              2  .01  local enhancement
101  .63  S-R; S                          0  .00  environmental continum


Appendix D-4

The recoverability of
Stimulus terms

The left-hand column gives the response-distributions from which the "key words" were chosen. The right-hand column gives the response-distributions obtained when the subjects were given one blank to fill in, with an instruction; find the "key word" that links together the four words presented.

For Group I, the key-words were selected from the response-distributions given by the same class on a previous exam.

For Group II, the key-words were selected from the response-distributions given by a class other than the one seeking the key-word.

I (a)  212-2a  N=74                    212-2a  N=74

4 Associates to Stimulus Word:         "Key Word" to "G.&S., bumps,
     Phrenology                        localization, pseudo science."

Response                      N   RF   Response                    N   RF

Gall and/or Spurz.            59 .80   phrenology                  62  .84
localization                  48 .65   repetition of S              2  .03
pseudo science                48 .65   others                      10  .13
bumps                         37 .50
size                          20 .27
brain                         16 .22
trait char.                   13 .18


I (b)  212-2a  N=74                    212-2a  N=74

4 Associates to Stimulus Word:         "Key Word" to "ablation, removal,
     Extirpation                       localization, lesions."

Response                      N   RF   Response                    N   RF

ablation                      44 .60   extirpation                 19 .36
removal                       38 .51   brain damage                11 .31
study-investigate             26 .35   brain                       10 .13
lesions                       23 .31   lobotomy                     7 .09
localization                  21 .27   brain functions              6 .08
brain                         13 .18   repetition                   2 .03
drugs                          6 .08   others                      19 .36
Names-(Lashley, Magoun, etc.)  6 .08


II (a)  212-1  N=33                    212-2b  N=81

4 Associates to Stimulus Word:         "Key Word" to "spatial, temporal,
     Summation                         threshold, firing of neuron."

Response                      N   RF   Response                    N   RF

spatial                      18  .55   summation                   35  .43
temporal                     18  .55   stimulus                    10  .20
threshold                    15  .45   others                      40  .37
compound stimuli              8  .24
firing of neurons             5  .15
synapse                       5  .15
addition of output            3  .09
intensity                     3  .09
neuron                        3  .09
stimulation                   2  .06


II (b)  212-1  N=33                    212-2b  N=81

4 Associates to Stimulus Term:         "Key Word" to "neuron, nodes of
     Myelinated Fibers                 Ranvier, insulation, faster trans."

Response                      N   RF   Response                    N   RF

neuron                        26 .70   myelin(ated)                57 .70
faster transmission           20 .61   (covering, sheath, tissue,
axon covering, shield         20 .61   fibers, insulation) others  21 .30
sheath                        19 .58
insulation                     6 .18
nodes of Ranvier               4 .12
around axon                    2 .06
not always present             2 .06


Appendix E-1

Frequency Relative to Total Possible
Number of Associates to Each of Four Related Terms
Stimulus Term      Harlow       Cloth          Wire      Affectional
                                mother        mother       drive

                 (n = 153)     (n = 149)      (n = 181)   (n = 179)

Harlow              1.00         .28            .19         .03
cloth mother         .28          -             .33         .08
wire mother          .22         .30             -          .07
experiment           .06         .06            .07         .00
Sci. American        .06         .01            .00         .00
behavior             .05         .00            .00         .01
imprinting           .04         .08            .08         .07
instinct             .04         .03            .05         .09
rats                 .04         .00            .01         .02
motivation           .03         .00            .03         .20
monkey baby          .03         .03            .04         .00
wire                 .03         .06            .00         .00
cloth                .03         .00            .04         .00
maternal             .02         .03            .00         .14
surrogate            .02         .03            .06         .00
maternal instinct    .02         .02            .03         .03
feeding              .01         .03            .06         .00
Hebb                 .01         .03            .02         .01
love                 .00         .03            .02         .28
sex                  .00         .00            .00         .12
security             .00         .09            .02         .02
emotion              .00         .00            .03         .08
innate               .00         .00            .00         .10
fear                 .00         .01            .06         .02
affection            .00         .04            .04         .00
hypothalamus         .00         .00            .00         .07
protection           .00         .03            .03         .00

Each stimulus term was given to approximately 1/4 of the same
class on a final examination. Four responses were possible.


Appendix E-2

The probabilistic relationship of
primary intraverbals in a group

133 students took this WAT; all stimulus terms were presented four (4) times. The terms in squares were presented as stimuli in the WAT. The encircled terms appeared as responses. Arrows connecting S and R give relative frequency of occurrences of each response relative to the maximum possible number (532). The careful reader will note that one response that occurred relatively frequently is not appropriate ("wrong"). Investigation showed that this word appeared in a paragraph of the text book that included the stimulus term and two or three "correct" associates. (Draw your own conclusions.)


Appendix F-1

Some reliability and validity measures of the WAT and other tests

Introductory Psychology, Honors Section (not an exhaustive set)

  1. Reliability
    1. Split half
      1. First half-second half: r = +.724
      2. Odd-Even: r = +.686
      3. Ranked difficulty of terms: r = +.683
    2. Kuder-Richardson forumula 20
      1. Original test (36 items): r = +.724
      2. Original test (1st 20 items only): r = +.592
      3. Retest (20 items): r = +.736
    3. Test-retest (20 items)
      1. Students' scores: r = +.624
      2. item difficulty: r = +.899
  2. Validity
    1. Rank-order correlation between test part scores
      1. Essay and short answer: +.804
      2. Association and short answer: +.733
      3. Association and essay: +.602
    2. Correlation between GPA and scores of:
      1. Association part: r = +.447
      2. Entire test: r = +.382
      3. Short answer part: r = +.265
      4. Essay part: r = +.254
    3. Correlation between position on Dean's List and rank of score on:
      1. Association part: r = +.678
      2. Entire test: r = +.677
      3. Short answer part: r = +.609
      4. Essay part: r = +.409


Appendix F-2

Some correlations on
five examinations
   N = No. students  n = No. st terms      4 resp; 2, 1, 0, -1 grading

1. General Psych., weekly quiz         2. General Psych., Honors Final Exam
     N = 133                 n = 25        N = 25                n = 36
     WAT: Mean score 137.0   SD = 23.3     M = 180.2             SD = 28.2
     % of max score possible = 68.5        % of max score = 62.6
     GPA: Mean  2.30         SD = 0.54     M = 3.44              SD = 0.28
     int. consist (KR)       r = 0.816     (KR)                  r = 0.724
     WAT with GPA            r = 0.356     WAT with GPA          rho = 0.447
     WAT with M.C.           r = 0.459     WAT with short ans.   rho = 0.773
     M.C. with GPA           r = 0.359     WAT with essay        rho = 0.602
                                           Short ans. with GPA   rho = 0.265
                                           Essay with GPA        rho = 0.254

3. Comparative Psych., Upper Div.      4.  History and Systems, Upper Div.
     N = 20                 n = 81         N = 28                n = 49
     WAT: Mean score 388.1  SD = 69.3      M = 218.39            SD = 78.1
     % of max score possible = 59.9        % of max score = 65.8
     GPA: Mean score 2.52   SD = 0.44      M = 2.73              SD = 0.87
     WAT with GPA           rho = 0.647    WAT with GPA          rho = 0.455


Appendix G-1 Research on Honors Comprehensive Examination.

Research on Undergraduate Honors Comprehensive examinations given at the end of the first year of the Honors program. If a student does not pass at that time he or she is examined the following year. The Comprehensive Examination has ten subsections, of which History and Systems is one. Different samples of terms are prepared for each administration. The Comprehensive Exam was taken by Advanced Honors Students at the end of the Junior Year, which had been devoted to intensive readings, with weekly seminars.

The same examination was given experimentally to graduate students in Psychology at the Master's level to replace a departmentally prepared ten-section multiple-choice examination. It was replaced by the appropriate Graduate Record Examination, which placed no demands on faculty time.