---
abstract: |
Objective: To assess the sources of measurement error in an electrocardiogram (ECG) interpretation examination given in a third-year internal medicine clerkship.
Design: Three successive generalizability studies were conducted. 1) Multiple faculty rated student responses to a previously administered exam. 2) The rating criteria were revised and study 1 was repeated. 3) The examination was converted into an extended matching format including multiple cases with the same underlying cardiac problem.
Results: The discrepancies among raters (main effects and interactions) were dwarfed by the error associated with case specificity. The largest source of the differences among raters was in rating student errors of commission rather than student errors of omission. Revisions in the rating criteria may have helped increase inter-rater reliability slightly however, due to case specificity, it had little impact on the overall reliability of the exam. The third study indicated the majority of the variability in student performance across cases was in performance across cases within the same type of cardiac problem rather than between different types of cardiac problems.
Conclusions: Case specificity was the overwhelming source of measurement error. The variation among cases came mainly from discrepancies in performance between examples of the same cardiac problem rather than from differences in performance across different types of cardiac problems. This suggests it is necessary to include a large number of cases even if the goal is to assess performance on only a few types of cardiac problems.
altloc: []
chapter: ~
commentary: ~
commref: ~
confdates: ~
conference: ~
confloc: ~
contact_email: ~
creators_id: []
creators_name:
- family: Solomon
given: David J
honourific: ''
lineage: ''
- family: Frerenchick
given: Gary
honourific: ''
lineage: ''
date: 2004-12
date_type: published
datestamp: 2006-01-06
department: ~
dir: disk0/00/00/46/71
edit_lock_since: ~
edit_lock_until: ~
edit_lock_user: ~
editors_id: []
editors_name: []
eprint_status: archive
eprintid: 4671
fileinfo: /style/images/fileicons/application_pdf.png;/4671/1/ECG_ARticle.pdf
full_text_status: public
importid: ~
institution: ~
isbn: ~
ispublished: pub
issn: ~
item_issues_comment: []
item_issues_count: 0
item_issues_description: []
item_issues_id: []
item_issues_reported_by: []
item_issues_resolved_by: []
item_issues_status: []
item_issues_timestamp: []
item_issues_type: []
keywords: |-
electrocardiogram, educational, measurement, generalizability, performance based
assessment, reliability
lastmod: 2011-03-11 08:56:15
latitude: ~
longitude: ~
metadata_visibility: show
note: ~
number: ~
pagerange: 283-290
pubdom: FALSE
publication: Advances in Health Sciences Education
publisher: Kluwer Academic Publishers.
refereed: TRUE
referencetext: |-
Brennan, R.L. (2001). Generalizability Theory. St. Paul Mn: Assessment Systems Corporation.
Crick, J.E. & Brennan, R.L. (1984). A General Purpose Analysis of Variance System, Version 2.2.
American College Testing Service.
Downing, S. (2000). Assessment of knowledge with written test forms. In G.R. Norman C.P.M. van
der Vleuten & D.I. Newble (eds.), International Handbook of Research in Medical Education.
Dordrecht/Boston/London: Kluwer Academic Publishers, pp. 647–672.
Hancock, E.W., Norcini, J.J. & Webster, G.D. (1987). A standardized exam in the interpretation of
electrocardiograms. JACC 10(4): 882–886.
Mavis, B.E., Henry, R.C., Ogle, K.S. & Hoppe, R.B. (1996). The emperor’s new clothes: the OSCE
reassessed. Academic Medicine 71(5) (May): 447–453.
Norman, G.R., Tugwell, P., Feightner, J.W., Muzzin, L.J. & Jacoby, L.L. (1985). Knowledge and
clinical problem-solving. Medical Education 19: 344–35
relation_type: []
relation_uri: []
reportno: ~
rev_number: 12
series: ~
source: ~
status_changed: 2007-09-12 17:01:30
subjects:
- cog-psy
succeeds: ~
suggestions: ~
sword_depositor: ~
sword_slug: ~
thesistype: ~
title: 'Sources of Measurement Error in an ECG Examination: Implications for Performance-Based Assessments'
type: journalp
userid: 6075
volume: 9