Principles of assessment

© Dr Caroline Clapham

Table of contents

• 1. Introduction
• 2. Validity
• 3. Reliability
• 4. Washback
• 5. Pre-testing
• 6. Concluding remarks
• Bibliography
• Related links

1. Introduction

Assessment has profited from the advances that have taken place in statistical analysis over the last century and it is now considered important for assessors to check their tests empirically as well as rationally. In addition, the upsurge in interest in the results of tests and examinations by governments, parents and candidates has led to a multi-million pound testing industry and contemporaneously there is a huge increase in the numbers of scholars researching different aspects of testing and assessment.

Although trends in testing, as in other fields, change over time, some principles of assessment are permanent and are not overly affected by current fashions. They need to be held in mind by assessors whenever they construct a test, whether this be a class quiz, a class essay or an end-of-year examination. (Here the terms 'test' and 'assessment' are used interchangeably.) The most important of these principles fall under three heads: validity, reliability and washback. Whilst the following discussion focuses specifically upon language assessment, the principles of testing apply more generally.

2. Validity

'Validity' is an all-encompassing term which is related to questions about what the test is actually assessing. Is the test telling you what you want to know? Does it measure what it is intended to measure? A test is not valid, for example, if it is intended to test a student's level of reading comprehension in a foreign language but instead tests intelligence or background knowledge.

When a new test is constructed it should be assessed for validity in as many ways as possible. The aspects of validity which are looked at will, of course, depend on the purpose for which the test has been designed and will partly depend on the importance of the assessment. A teacher writing a classroom quiz will not have the time or the inclination to carry out many different investigations of validity, but the constructors of an examination which will affect candidates' futures are duty-bound to examine as many aspects of validity as possible. (It should be remembered that it is not the test itself, but the use of the test for a particular purpose, that should be examined; a test of oral French, for example, may be perfectly valid for 16-year-olds in the British examinations system but might not be valid for doctors needing medical French in order to be able to cope in Parisian hospitals.)

There are different views on the best ways of assessing validity, but there are some key aspects, and it is good practice to investigate as many of these as possible:

2.1 Construct validity

The term 'construct validity' refers to the overall construct or trait being measured. It is an inclusive term which, according to some testing practitioners, covers all aspects of validity, and is therefore a synonym for 'validity'. If a test is supposed to be testing the construct of listening, it should indeed be testing listening, rather than reading, writing and/or memory. To assess construct validity the test constructor can use a combination of internal and external quantitative and qualitative methods. For more about this, see section 6. An example of a qualitative validation technique would be for the test constructors to ask test-takers to introspect while they take a test, and to say what they are doing as they do it, so that the test constructors can learn about what the test items are testing, as well as whether the instructions are clear, and so on.

Construct validation also relates to the test method, so it is often felt that the test should follow current pedagogical theories. If the current theory of language teaching emphasises a communicative approach, for example, a test containing only out-of-context, single-sentence, multiple-choice items, which test only one linguistic point at a time, is unlikely to be considered to have construct validity.

2.2 Content validity

The content validity of a test is sometimes checked by subject specialists who compare test items with the test specifications to see whether the items are actually testing what they are supposed to be testing, and whether the items are testing what the designers say they are. (On specifications as a whole, see Davidson & Lynch 2002, and Alderson, Clapham & Wall 1995: Chapter 2.) In the case of a classroom quiz, of course, there will be no test specifications, and the deviser of the quiz may simply need to check the teaching syllabus or the course textbook to see whether each item is appropriate for that quiz.

One of the advantages of even the most rudimentary content validation is that it identifies those items which are easy to test but which add nothing to our knowledge of what the students know; it is tempting for a test writer to write easy-to-test items, and to ignore essential aspects of a foreign language, for example, because they are difficult to assess.

2.3 Face validity

Face validity is an important aspect of a test; it relates to the question of whether non-professional testers such as parents and students think the test is appropriate. If these non-specialists do not think the test is testing candidates' knowledge in a suitable manner, they may, for example, complain vociferously and the candidates may not tackle the test with the required zeal. If the test lacks face validity, it may not work as it should, and may have to be redesigned. (See Alderson, Clapham & Wall 1995: 172-73.)

2.4 Criterion-related validity

The aspects of test validity described so far relate to the 'internal' validity of the test, but some methods, and these are widely used for 'high-stakes' tests, also assess the 'external', 'criterion-related' validity of a test. To assess criterion-related validity, the students' test scores may, for example, be correlated with other measures of the students' language ability such as teachers' rankings of the students, or with the scores on a similar test. Such measures assess the concurrent validity of the measure. Similarly the future ability of the students can be assessed (the test's predictive validity) to see if the test can accurately foretell how the candidates will fare in the future. For example, if a test is supposed to assess whether students have a high enough level of a foreign language to be able to teach that language to secondary school children, the test should be validated, perhaps by classroom observation, to see whether students who have passed the test do actually have enough of the foreign language to be able to teach it in the classroom.

On validity as a whole, and especially predictive validity, see Alderson, Clapham & Wall 1995 and ILTA 2002.

3. Reliability

The reliability of a test is an estimate of the consistency of its marks; a reliable test is one where, for example, a student will get the same mark if he or she takes the test, possibly with a different examiner, on a Monday morning or a Tuesday afternoon. A test must be reliable, as a test cannot be valid unless it is reliable. However, the converse is not true: it is perfectly possible to have a reliable test which is not valid. For example, a multiple-choice test of grammatical structures may be wonderfully reliable, but it is not valid if teachers are not interested in the grammatical abilities of their students and/or if grammar is not taught in the related language course.

If the test consists of right/wrong items such as multiple-choice items or some sorts of short answer questions, a reliability estimate such as the Alpha Coefficient or Kuder Richardson 21 may be calculated (see Alderson, Clapham & Wall 1995: 87-89); but if the test consists of an essay or an oral interview, for example, then other forms of test reliability must be estimated. A statistic which can be used by the statistically sophisticated is based on Generalizability Theory (see Crocker & Algina 1986: Chapter 8.), but more simple measures such as correlations between the scores a marker gives on Day 1 and Day 5 (intra-rater reliability), and correlations between two different markers' scores (inter-rater reliability) can be estimated, along with calculations of whether the levels of raters' marks, as well as the order of the scores, are similar (see Alderson, Clapham & Wall 1995: Chapter 6).

4. Washback

Any language test or piece of assessment must have positive washback (backwash), by which I mean that the effect of the test on the teaching must be beneficial. This should be held in mind by the test constructors; it is only too easy to construct a test which leads, for example, to candidates learning material by heart or achieving high marks by simply applying test-taking skills rather than genuine language skills (see Wall 1997).

5. Pre-testing

If test writers wish to follow good practice in assessment, they need to make sure that their tests or examinations are valid and reliable. Both validity and reliability depend on the test items/tasks working well, and for this purpose most tests are pre-tested. Of course there is no time to do this for a class quiz, but for 'high stakes' tests, if it is possible to have 20 or more in the pre-testing sample, it is essential (see Alderson, Clapham & Wall 1995: Chapter 4). If a test contains some kind of right/wrong items, where the student gets a mark for each correct answer, an important part of test construction is that the test should undergo item analysis. That is each item should first be examined to see how difficult it is for the group tested (facility value) and scores on that item should also be correlated with the total scores on the test in order to see whether the highest proficiency students tend to get an item right whilst the lowest proficiency ones get it wrong (discrimination index). If the discrimination index of an item is poor, the item may be easier for the linguistically weak students than for the linguistically strong, and there may be something seriously wrong with the item or its answer key. It is probable that that item should be changed or dropped. An excellent program for producing the statistics relating to the evaluation of right/wrong test items such as multiple-choice items and short-answer questions is ITEMAN 1989.

If students' futures are likely to be affected in any way by the test results, performance tests such as essays and oral interviews should also be tried out, though on fewer candidates. Such tests and tasks should be pre-tested to see, for example, whether they produce the expected kinds of language, and whether the marking criteria are appropriate. (For more about marking criteria see Alderson, Clapham & Wall 1995: Chapter 5, and Weigle 2002.)

6. Concluding remarks

Testing and assessment take up large slices of teachers' time and, perhaps partly because of this, there is currently great interest in validity issues, washback and in the ethics of testing. More and more tests now have a pre-testing element built in to their construction and, in addition, there is much research into the construction, marking and validation of 'subjective tests'. Readers who wish to read more about Language Testing than just introductions to testing are recommended to consult Bachman 1991. They should also, if they want to read short articles on various aspects of language assessment for first and second language learners, look at some of the chapters in Clapham & Corson 1997 . (This publication has 29 chapters on key language testing aspects in first and second language assessment, and each chapter is accompanied by approximately 30 references for further reading.)

7. Glossary of testing terms (with special reference to Language Testing)

For further information on language testing terms refer to the Multilingual Glossary of Language Testing Terms (1998) and Davies et al (1999).

Bibliography

Alderson, J. C. (2000). Assessing Reading. Cambridge: Cambridge University Press.

Alderson, J. C., C. M. Clapham & D. Wall (1995). Language Test Construction and Evaluation. Cambridge: Cambridge University Press.

Bachman, L. (1991). Fundamental Considerations in Language Testing, Oxford: Oxford University Press.

Buck, G. (2001). Assessing Listening. Cambridge: Cambridge University Press

Clapham, C. & D. Corson (eds) (1997). Language Testing and Assessment, Volume 7 of Encyclopaedia of Language and Education edited by D. Corson.. Dordrecht: Kluwer Academic.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth.

Davidson, F. & B. K. Lynch (2002). Testcraft: A Teacher's Guide to Writing and Using Language Test Specifications. London: Yale University Press.

Davies, A., A. Brown, C. Elder et al (eds) (1999). Dictionary of Language Testing. Cambridge: Cambridge University Press

International Language Testing Association (ILTA) (2002). http://www.surrey.ac.uk/ELI/ILTA/faqs

ITEMAN (1989). Assessment Systems Corporation, 2233 University Avenue, Suite 200, St. Paul, MN, USA.

Multilingual Glossary of Language Testing Terms (1998). Edited by ALTE members.

Wall, D. (1997). Impact and Washback in Language Testing. In C. Clapham & D. Corson (eds), Encyclopaedia of Language and Education 7: Language Testing and Assessment, 291-302. Dordrecht: Kluwer Academic.

Weigle, S. C. (2002). Assessing Writing. Cambridge: Cambridge University Press.

Related links

There are an increasing number of test sites on line, but as a start readers are encouraged to look at:

The International Language Testing Association (ILTA) 2002 Web Site. This has 10-minute videos on various topics relating to Language Testing: http://www.surrey.ac.uk/ELI

The reader could also look at the following two sites, the first of which gives examples of many tests of English and the second of which describes the DIALANG project in Europe:
http://www.cambridge-efl.org
http://www.dialang.org/english

Referencing this article

Below are the possible formats for citing Good Practice Guide articles. If you are writing for a journal, please check the author instructions for full details before submitting your article.

• MLA style:
  Canning, John. "Disability and Residence Abroad". Southampton, 2004. Subject Centre for Languages, Linguistics and Area Studies Guide to Good Practice. 7 October 2008. http://www.llas.ac.uk/resources/gpg/2241.
• Author (Date) style:
  Canning, J. (2004). "Disability and residence abroad." Subject Centre for Languages, Linguistics and Area Studies Good Practice Guide. Retrieved 7 October 2008, from http://www.llas.ac.uk/resources/gpg/2241.