Provenance Support for Evidence Based Policy

Scenario Authors:

Edoardo Pignotti, Peter Edwards, Richard Reid, Adele Laing, Lorna Philip ( University of Aberdeen)

Brief Summary:

Evidence Based Policy Assessment is associated with all aspects of policy-making and is required, in the UK, of national and local government. Best practice principles for ex ante appraisal of policy and ex post evaluation of policy are set out in the UK Government’s Green Book (HM Treasury, 2003). Our use of the term assessment embraces both appraisal and evaluation. The Green Book’s primary concern is economic assessment whereby the costs and benefits of policy options are estimated, a process which includes formal consideration of ‘non-market’ policy impacts. Economic assessment requires that:

• Reports should provide sufficient evidence to support their conclusions and recommendations;
• There should be an easy audit trail to allow decision makers to understand the assumptions underlying conclusions and recommendations;
• There should be sufficient information to support any later evaluation.

Two generic types of policy evaluation are recognised in the UK Government’s performance management system: formative or process evaluation (how, why and under what conditions does a policy intervention ‘work’ or ‘fail to work’?) and summative or impact evaluation (what impact does a policy intervention have?). These types of policy evaluation may overlap and interact and, importantly, recognise that policy evaluation can benefit from using a wide range of evidence, not just (economic) cost-benefit analysis. Useful evidence for policy evaluation can be the data and/or the conclusions from previous research, and new material collected to supplement existing evidence, to fill in gaps in existing evidence, or to create completely new evidence. Policy makers are encouraged to critically appraise the evidence they work with. The nature of research questions, the primary data collection methods used, secondary source materials and the various analytical and interpretative techniques employed set limits on the reliability, replicability, generalisability and overall validity of conclusions that can be drawn from a piece of research that academic researchers may produce for use in policy assessment. Management of evidence is an important aspect of the work outlined above. This requires a detailed description of the digital artefacts associated with projects, people, and social interaction. Additionally, provenance must provide support for the creation of audit trails to allow evidence to be assessed/validated, and must ultimately include mechanisms for creating policy arguments and options from the evidence base. Such a provenance model would facilitate better communication and collaboration between members of a project team, while at the same time enhancing the ‘interface’ between the producers of evidence (e.g. academic researchers) and the users of that evidence (e.g. policy makers).

Scenario Diagram:

To illustrate a subset of this scenario the diagram above presents an example on how traditional artefact and process-driven provenance can be associated with social data. A set of interview notes has been produced, following an interview conducted by John Farrington; Lorna Philip, who works with John Farrington, has posted a comment on the interview notes asking where the audio recording of that interview is. This example demonstrates how social context (e.g. worksWith) and user-generated content (e.g. Post) can be integrated with the data provenance record. The effect of this is a far more definitive provenance representation of the digital artefact; the result of Lorna Philip’s post may have inspired John Farrington to upload the audio recording of the interview. Alternatively, the lack of a vital piece of supporting information may have altered the level of trust in the interview transcript.

Users:

Single and Multi discipline research groups (academic researchers)

Policy Makers

Requirement for provenance:

In order to solve the problem described above we require a representational framework for provenance which goes beyond simple metadata descriptions of artefacts and processes. Such an approach has to create a single, unified record, bridging both the digital and physical worlds; it must also provide support for reasoning capabilities, to allow it to be used for more than just descriptive purposes. The following are the characteristics of the provenance framework required:

1. It should describe and uniquely identify a range of entities: artefacts (digital & physical); processes (services & human activities); people; organisational structures/membership; social networks.
2. It should situate entities in time and space.
3. It should incorporate online communication (e.g. instant messaging, blog entries, email) into the provenance record.
4. It should allow relationships (e.g. causal, social, organisational) to be defined between entities.
5. It should make explicit goals and constraints associated with processes and associated artefacts, in order to capture the ‘why?’ aspect of provenance.
6. It should facilitate reasoning about access control; documentation policies; completeness of the provenance record; trust and reputation.

Provenance Questions:

1. How was this evidence derived? Is the derivation history complete?
2. Where and when was the artefact (created, collected, published, analysed etc.)?
3. Who contributed in creating this artefact, executing this process?
4. What online communication was associated with the creation of this artefact or execution of this process?
5. Can I allow person P to access artefact A?
6. Can I trust this artefact, provenance record, person, project ….?
7. Why was a particular scientific activity undertaken?
8. Why was a particular artefact used as part of a process?

Technologies Used:

This scenario requires ontologies in order describe different aspect of evidence: artefacts, processes, people, organisations etc.

A Web-based environment is required in order allow researched to upload and share digital artefacts and to collaborate using social networking tools such as blogging and messages.

-- EdoardoPignotti - 17 May 2010
to top