Pedagogical research methods (26 Feb 2003)

Event report

by Dawn Ebbrell

A research methods workshop was organised as part of the Pedagogical Research Fund to support funded projects in developing their research methodologies and as part of that process, providing opportunities for projects to network and share expertise. A number of places were also offered to the wider subject involving a greater diversity of projects and expertise in that process.

The workshop was led by Professor Rosamund Mitchell from the Research and Graduate School of Education at the University of Southampton and covered the following areas:

• project Design and Management;
• collecting and interpreting interview data;
• collecting and interpreting questionnaire data.

Planning and managing research projects

The first session looked at issues in planning and managing a pedagogical research workshop. Participants discussed issues related to the planning of their own projects focusing on the following areas:

• Plan for focus – research questions drive the rest!.
• Use the work of others.
• Plan for theory-building.
• Plan for explanation.
• Plan for methodological innovation.
• Plan for dissemination.

Designing interviews and analysing interview protocols

A range of methods for devising interviews were discussed including

• structured;
• semi- structured;
• unstructured/ ethnographic;
• group versus individual interviews;
• The place of stimulated recall (use of artefacts as prompts).

Some of the methods identified here were related to the issue of inevitable subjectivity in the interview process, particularly, as may often be the case in small-scale and perhaps solo research projects, if the interviewer is researching an aspect of their own teaching situation. Even if this is not the case, both parties will bring with them to the interview their own concerns about how they come across for a number of reasons.

The advantages of group versus individual interviews were discussed. Some of the perceived advantages of group interviews were that you could draw on a wide range of opinions in a short time-frame; ‘safety in numbers’ may make otherwise reticent interviewees more vocal; another person’s opinion may inspire comment in another member of group, raising an issue that they may not otherwise have thought of. Conversely, it was felt by some that peer pressure may increase reticence and other people’s views may skew the opinions of others ! Problem-solving activities could be set up to make the situation less intimidating and in a sense, depersonalise the situation whilst still eliciting personal viewpoints. For example, a group of school children might be asked to articulate what good teaching means to them by giving them the following scenario:

imagine a student teacher is coming to school next week, make a list of tips that would be useful in helping them motivate your class.

It was advised that whatever type of interview is chosen, researchers should review the interview paying attention to what worked and what did not. A number of methods for analysing interview data were given including the use of software packages such as NUD*IST and Ethnograph. Both of these are tools for analysing electronic transcripts allowing the researcher to code texts and build up theories by pulling together examples of similar themes, behaviours or concepts.

Designing and conducting questionnaire surveys

The final session addressed the use of questionnaires in pedagogical research which are heavily used in pedagogical research and all of the projects represented at this workshop were using them in one way or another.

Key issues to bear in mind when designing questionnaires include being aware of the level of analysis that the questionnaires will require. e.g. open ended responses will inevitably require more analysis but will lend themselves to both theory building and theory testing. In questionnaire series, it is useful to use a numbering system for anonymous questionnaires as different responses can be related back to subjects and answers to different questions from the same person compared. This technique is also useful in keeping track of which questionnaires have been inputted for later analysis. Regardless of the questionnaire type, the need for careful piloting of the questionnaire is key.

A number of different examples from the group were discussed including a tightly structured needs analysis for language training questionnaire, a largely open-ended on-line survey of why students choose to study a particular discipline, in this case, Australian studies and a tool called the Ideal *** Inventory. The latter has been used in pedagogical research to explore differences between tutors’ and students’ perceptions e.g. the ideal lecture or the ideal distance learning task. In the Liverpool Hope University College project, it is being used with students to develop oral assessment criteria in the light of students’ use of the Inventory. In this project, students are asked to develop just five ‘Ideal’ and five ‘Not Ideal’ criteria in order to focus them on the most important points. The examples that are given to illustrate the task are deliberately not related to the assessment criteria so as not to lead their thinking. The project will cover years 1 to 3 to establish whether different criteria are needed depending on the level of student experience. e.g. confidence in a presentation might be a criteria at level one but not at level three.

The projects funded through the Subject Centre Pedagogical Research Fund will come together again at the end of their one-year project to present their findings in a workshop which will be open to the wider subject community. Further details will be circulated when available.

Further information

Locating Research (rtf, 461Kb)
www.cilt.org.uk/infos/rtf/26to50/InformationSheet35.rtf
CILT Information sheet 35: Locating Research in Language Teaching

The Ideal*** Inventory
http://ctiwebct.york.ac.uk/LTSNPsych/PLAT2002/abstracts/session_15.htm
Information and references about this tool

Data Analysis

NUD*IST software
www.qsrinternational.com/products/productoverview/product_overview.htm
NUD*IST software (latest version is known as N6)

Ethnograph software
www.scolari.co.uk/ethnograph/ethnograph.htm

Approaches to data analysis
www.scilt.stir.ac.uk/SLR/Issue5/SCOTLANG.htm
Mitchell, Ros, (2002), Approaches to data analysis. Article published in SCILT’s Scottish Languages Review, Issue 5, 2002.