E3AN Evaluation Plan

Liz McDowell

Introduction

The Project Plan indicates that evaluation will include the following processes:

·	Summative – providing evidence of project achievements at formal interim and final stages
·	Formative – providing evidence to be used to monitor and adapt the project during its lifetime
·	Reflective – providing evidence to inform longer –term developments and broader issues beyond the framework of the project itself.

This Evaluation Plan satisfies these three purposes and is structured according to three output-oriented evaluation levels:

•	Performance 	What has the Project done?
•	Impact 		With what effects?
•	Synthesis 		What can be learnt from the Project?

For convenience, a brief summary of project activity in each Work Period is appended to this plan.

Performance

Performance evaluation is the responsibility of the Project Team & Steering Group.  Evidence of Project activities and achievements is the main focus.  Associated quality assurance procedures are required to give stakeholders confidence in the Project.  The role of the External Evaluator is advisory.

In Work Periods 1-4:
•	Project activities will be expressed as objectives or targets (based on Appendix B of the Project bid).  There will be a variety of kinds of targets and not all will be numerical.  Agreed evidence will be gathered on the attainment of targets.  For example, in Work Period 1, initial contacts with the user community will be undertaken and evidence will be collated to indicate who has been contacted, how and how many contacts have been made.

In Work Period 1:
•	Outline quality assurance procedures will be drafted for the establishment of the testbanks (eg issues of security, access, data formats, metadata) and for the creation of questions (eg selection of consultants, peer review, question trialling).  

In Work Periods 2 and 3:
•	Quality assurance procedures will be implemented and reviewed in practice.  A working handbook will be developed.

In Work Period 4: 
•	Quality assurance procedures will be summatively reviewed and a definitive handbook produced for continued use. 

Impact 

Impact evaluation is the joint responsibility of the Project Team & Steering Group, and the External Evaluator. It focuses on the responses of stakeholders and impacts on the user communities.  The External Evaluator will assist the Project Team  to devise appropriate ways of determining stakeholder/user response and impact.  The External Evaluator will also make direct contact with some groups of stakeholders/users where the external perspective is seen to be particularly valuable.

In Work Period 1:
•	Data will be gathered from initial contacts in the user community to enable an assessment to be made of the baseline ‘state of play’ with respect to assessment.  This will complement what is already known from previous work and published sources and enable change over the life of the project to be evaluated. 

In Work Periods 2 and 3:
•	Data will be gathered as part of dissemination and consultation activities and analysed to indicate user responses, needs and requirements.
•	Student impact data will be gathered as part of question trialling.

In Work Period 2:
•	External evaluation of the experiences and views of the Project consultants will be undertaken

In Work Period 3:
•	Feedback data will be gathered on the impacts on teaching and learning of testbank use from initial users and the extended group of consultants.
•	External evaluation of the experiences and views of the Project consultants and pilot users of the testbank will be undertaken

In Work Period 4:
•	Impact evaluation analyses will be drawn together for the final project report and will include evidence of change and development over the life of the project. It will be used to inform continuation plans.

Synthesis 

Synthesis evaluation is the joint responsibility of the Project Team & Steering Group, and the External Evaluator.  There will be joint engagement in reflective consideration of wider project implications.  Wider groups of stakeholders may also be involved in these activities.  The focus is on implications for future developments in assessment in EEE education, and for learning, teaching & assessment in HE more generally

In Work Period 1:
•	An initial review of key issues relevant to the project will be conducted.  This will enable the work of the project to be located in its broader context right from the start.  The key issues will include the definition of good practice in assessment and models of innovation and change in HE pedagogy.

In Work Periods 2 and 3:
•	A reflective review will be undertaken of emerging issues concerning assessment in EEE and pedagogical change.  
•	Interim reflective reports will be included as part of annual reporting and other appropriate audiences will be identified and targeted.
•	The outcomes of reflective review will inform dissemination and consultation activities in the EEE user community.

In Work Period 4:
•	A summative reflective review will be undertaken as part of the final reporting of the project
•	Key issues in the areas of assessment, computer-assisted assessment, collaborative development, HE pedagogy, innovation and change will be identified. 
•	Target audiences will have been identified by this stage.  Outcomes will be presented in a variety of appropriate and targeted formats.

Summary of Project Activities in each Work Period

Work Period 1: Planning (Month 1 - M6)

In this period the main activities are project set-up and planning for the testbanks (format for electronic storage, working methods for question development and subject focus).  Initial contacts with target user communities will also be made

Work Period 2: Local Pilots (M7-M18)

The main activities are the development of testbanks in selected topics with a target of 1200 questions.  Dissemination and consultation activities will be undertaken and guides to assessment issues will be published.

Work Period 3: Local and national Activity (M19-M30)

The main activities are the further development of testbanks drawing upon a larger team of consultants,  with a target of an additional 1800 questions.  Dissemination and consultation activities will be undertaken and guides to assessment issues will be published.

Work Period 4: Consolidation (M31-36)

Final dissemination activities and publications will be delivered.  Arrangements for continuation of testbank developments an support for its use will be determined.