University of Southampton > ECS > FREMA > Project Journal

Project journal

FREMA OVERVIEW
SORM METHODOLOGY
SYSTEM/ONTOLOGY
GAP ANALYSIS
ASSESSMENT TOOLS AND TECHNOLOGIES
BPEL4WS
USE CASES
CONCEPT MAPS
POPULATING THE DATABASE (Domain Definition)
CETIS/JISC CONFERENCE
DISSEMINATION

FREMA OVERVIEW

The FREMA project brings together expertise from a number of people from four universities around the UK (Strathclyde, Hull, Open University, and Southampton). A number of face-to-face technical meeting were arrange so that the knowledge and experience of the group could be applied and focused on key issues in the development of a reference model for assessment in e-learning. The meetings were attended by the majority of the team, and to make the most of having all this expertise in one place very often long days too. The meetings were initially fortnightly as the project was evolving then once a month as the work became more clearly defined and tasks allocated to specific groups. The early meeting focused on the underlying data model (ontology) and database design for the site and was quickly followed by the concept and then concepts maps. The concept maps were a result of several workshops, before they were taken on the road to be validate by practitioners across the UK, not only from Universities but also other agencies for example examination boards. The later workshops have concentrated on gathering use cases (UML diagrams and Scenarios) and populating the database with these.

Dr Gary Wills

SORM METHODOLOGY

Service-Oriented Reference Models are community-driven devices for understanding how services fit together to provide functionality for a particular domain. At the simplest level this could be a static description of a a set of services (service profiles) and a common understanding of how they fit together. A more sophisticated view is that a reference model should not only describe a core set of services, but also should present a full audit trail of how those services were derived, which of them have been implemented, what standards they use, and how they fit into use cases.

To enable this more sophisticated view we have conceptualised a Service Oriented Reference Model as a number of layers (shown in Figure) where each layer is smaller in scope but more concrete than the one below it. It is imagined that as a community uses and further develops a Reference Model its higher layers will cover more and more of the lower.

The Domain Definition is an overview of the domain that the reference model covers, it contains details on existing domain resources (such as standards, people and projects) and also the ontological relationships between them. Identifying Common Usage Patterns is the process of scoping the domain into key activities. These can then be described with use cases (formalising the usage patterns). The use cases can then be used to undertake a Gap Analysis, a mapping of use cases to service descriptions and software. This shows how the key activities of the domain are currently supported with software and services. The Service Profile layer contains the descriptions of those services identified in the gap analysis (both formal, e.g. in WSDL, and informal, e.g. written description). Finally the most concrete layer is an actual reference implementation of the service profiles, although not all services will necessarily be implemented.

This multilayered approach means that the reference model can be used in a variety of ways:

• Domain Definition: This might be used to develop a context for ones own work. To understand how existing work fits together, identify standards and locate experts.
• Use Cases: These can be used to help understand usage patterns within the domain and help developers create new Service Profiles and thus Services.
• Service Profiles: Developers that wish to build new services that work within the domain framework will need to use the service profiles to ensure interoperability. They might also wish to create alternative versions of existing services, either to improve on the existing implementations, or for commercial reasons.
• Reference Implementation: Finally the actual software implementations are available to those developers that wish to build on some, or all, of the developed services.

We believe that supporting the development of Web Service frameworks requires more than a static reference model. This is because developing these frameworks is a community exercise and the effort is otherwise so difficult for individuals to comprehend, use and contribute to. It is our hope that our Reference Model for e-Learning Assessment will not only facilitate the population of the JISC Framework, but will also be a template for future Reference Models and make Web Service Frameworks more accessible and easier to develop.

SYSTEM/ONTOLOGY

The runtime of the FREMA Reference Model is built upon an ontological database that describes all the resources in the model and the relationships between them.

An ontology is simply the collection of classes and relations that are permissible for any given domain (it is called an ontology since it restricts and defines which parts of the world may be understood by entities conforming to it). The advantage of ontological modelling over database schemas is that it enforces a finely grained, and thus flexible and extensible, set of relationships. (It also means that the resources in the Reference Model could be described on the Semantic Web, and that would enable interoperability between different Reference Models, and reasoning about the described resources.)

GAP ANALYSIS

One of the main benefits of a dynamic Reference Model is that it is possible to run a Gap Analysis at any moment in time that compares the Use Cases in the model with the Service Profiles and Software so that we can see what core activities of the domain are currently supported. As an example, running a gap analysis might produce the following table:

Use Case:	Service Profile:	Software:
Check for Plagiarism	Plagiarism DB Check	N/A
Check for Plagiarism	Plagiarism Analysis	EssayCompare
Report Breach of Rules	N/A	N/A

This would tell us that for the 'Check for Plagiarism' Use Case we have two relevant Services, one of which has software that implements it, but for the 'Report Breach of Rules' Use Case there is no defined service.

The Gap Analysis therefore helps us discover which of our domain core activities are currently represented by services and supported by software, helping to direct future development efforts.

Dr Dave Millard

ASSESSMENT TOOLS AND TECHNOLOGIES

As part of my work, I looked at a range of e-learning assessment tools and technologies developed by different universities as well as commercial organizations. Some of these include but not limited to E3AN, TOIA, Samigo, Spaid, HELM (Helping Engineers Learn Mathematics), Question Mark Perception tool, UMap, Criterion, Intelligent Assessment technologies, QTI Lite, Hot potatoes, Respondus, etc

I tried to reverse engineer the functionality available in assessment systems like TOIA, E3AN, Samigo, etc to understand the implementation, underlying data models for item bank, Meta data and also to re-use some of the existing web services

Designed & implemented the following to feed in to FREMA e-learning reference model.

• A web service with search By Keyword and Search By Subject theme operations for searching against TOIA item bank
• A web service with search By Keyword and Search By Subject theme operations for searching against E3AN item bank
• A sample web application which uses the above web services to query against E3AN item bank & TOIA item bank.
• Web Services to implement various design patterns

Technologies: Java, Java Server Pages, Servlets, XML, JDBC, SOAP, WSDL, Apache Axis, Tomcat

Swapna Chennupati

BPEL4WS

Business Process Execution Language for Web Services (BPEL4WS) is used to formally specify business processes and their interaction protocols. By doing so, it extends the web services interaction model and enables it to support business transactions. BPEL4WS process represents all partners and interactions with these partners in terms of abstract WSDL interfaces (portTypes and operations); no references are made to the actual services used by a process instance. There are two ways to use BPEL4WS for orchestration.

• Each web service is a process with its own distinct role and the relationship is modelled as a partner link.
• An executable business process is defined to determine the nature and sequence of the web service interactions conducted at each business partner, and thus the interaction protocols.

USE CASES

Business use cases were created for existing software such as e3an, TOIA and the ultimate assessment engine UAE. Each use case is associated with a structured narrative, expressed in the language of the application domain and of users. These use cases reflect the functionality of the software from the users’ point-of-view and would be elemental in the gap analysis.

Mr Ehtesham-Rasheed Jam

CONCEPT MAPS

A FREMA concept map is a graphical model of the e-assessment domain. It allows the end-user to search within and navigate through the FREMA reference model data, using domain-specific terminology meaningful to e-assessment practitioners. The approach we adopted was to

• design initial concept maps
• import these maps into the FREMA reference model
• associate references with concepts

First we designed two prototypes

1. The Entity Concept Map, which takes a resource-oriented perspective of e-assessment, modelling entities such as data, specifications and reports
2. The Process Concept Map, which represents a functional perspective of the domain, modelling activities such as writing questions, scheduling tests and providing learner feedback.

Each map underwent an iterative design and review process within the FREMA team. Wherever possible, we added provenance to our components (e.g. referencing the “Ultimate Assessment Engine” paper and the Dialog+ project). Latterly the maps were exposed to the wider e-assessment community (e.g. at the JISC – CETIS Conference “The e-Framework: Priorities and Challenges for 2006” Heriot-Watt University, Edinburgh, 15 /16 November 2005) from whom we received favourable feedback. Moreover, each paper in the 2005 CAA Conference Proceedings was matched against the maps, thereby helping us refine and identify gaps in our concepts.

Next the maps were imported into the FREMA database (via XML). At this point we authored definitions for each concept (approximately 150 in total), thereby producing an internal glossary of terms. Finally, we mapped each entry (including web services) in the reference model to at least one concept in each map.

POPULATING THE DATABASE (Domain Definition)

The preliminary step was to build a “reference portfolio” of e-assessment practice - identifying inter alia its main stakeholders, projects, systems and standards. This was compiled primarily via web searching and reading the literature (e.g. Ultimate Assessment Engine Paper, IBIS Report, SQA Guidelines). Thereafter the reference model was designed and implemented as a MySQL database. A set of PHP screens was generated to facilitate populating the database. For each reference, it was necessary to specify

• Basic definitions – these comprise mainly name, short description and external links
• Associations – these interlink references (e.g. software “belongs to” a project). Descriptions can be added to associations to explain why two components relate.
• Mapping to concepts - each database entry should be mapped to one or more concepts

Some points which arose

• Both the database schema and interface were amended as a result of this exercise – e.g. allowing text files to be uploaded
• The prototype editing interface was adequate; however we have identified future enhancements to it
• Need naming convention for references (particularly software-related such as service descriptions)
• Short descriptions - creating these is not as easy as it sounds! Much enhanced by inserting images (if available)
• External links – homepages were the most frequent entries here. Also a useful way to share documents amongst references
• Mapping to concepts – useful comments from ASAP project e.g. should subnodes inherit concepts already mapped to their supernodes? Ideally real practitioners (e.g. service developers) would create and maintain their FREMA components – thereby ensuring accuracy and currency of the data.

Mr Iain Tulloch

CETIS/JISC CONFERENCE

At the JISC/Cetis Conference held at Heriot-Watt, Edinburgh, on 15 Nov 2005, a group of about 20 teachers, technical providers, and managers reviewed the FREMA reference model and provided detailed feedback on the usefulness of the reference model and how well the work of the group's individuals and organisations could be represented in and by the reference model.

The group undertook a scenario-based role play, acting as an early adopter of e-assessment, a technical developer of e-assessment tools, or a manager or planner involved in the selection or deployment of such tools. Each role was provided with a scenario that was typical of their activities, which was then enacted by using the FREMA Reference Model Web site.

Feedback was provided through a structured questionnaire, a semi-structured free-response form, and in a group debriefing session, on the topics of 'Getting to what you wanted: site use', 'What you got: site content', and 'Overview: site effectiveness'.

The group's responses were particularly positive in the areas of model layout & structure, site attractiveness, intuitive and easy navigation, accessibility, and the provision of the concept maps which were thought accurate and relevant.

The group indicated areas for improvement in widening the content coverage, enhancing the obviousness of what was on offer in the model, and extending the provision of help while using the model.

Mr Lester Gilbert

DISSEMINATION

Disseminating our ideas and work throughout the e-learning community has been a significant and integral part of the Frema project. As well as the wider e-learning community, we have built strong relationships with specialist assessment stakeholders: CETIS assessment SIG, Qualifications bodies (SQA), and academic institutions – Loughborough, Kingston ad York.

We certainly hit the ground running with a meeting at the Oxford ELF developers’ codebash in March 2005 (this was before the project had officially started! But we were able to present an overview of the project and our project time line.

CETIS assessment SIG Southampton April 2005.
This was our first meeting with the assessment SIG whose co-ordinator, Rowin Young ( Strathclyde) is also a member of the FREMA team. By this time we had developed our ideas for the structure of Service Oriented reference model, SORM, as an abstraction (see Dave Millard’s piece) and how that could be instantiated for the assessment domain. Our view that a reference model should be searchable, dynamic, evolving and accessible for all stakeholders and from many points of view was endorsed by the SIG members.

9th International CAA Conference Loughborough July 2005
Over 120 delegates attended the 9th CAA Conference. We presented a paper ‘Aggregating Assessment Tools in a Service Oriented Architecture’ and also gave a presentation about the FREMA reference model in the JISC workshop strand. This was our first opportunity to spread our ideas and work to the wider assessment community. We showed them how dynamic, searchable reference models can increase the effectiveness of e-assessment initiatives by encouraging re-use and interoperability.

CETIS assessment SIG Strathclyde July 2005
Frema presented and demonstrated concept maps   as a graphical way of navigating through the resources in the reference model (projects, services, software) using terms familiar to practitioners in the domain. The SIG members gave us encouraging feedback that we are producing a valuable way of finding resources and were keen to offer suggestions about how we might extend and improve our vision.

JISC Vocational portfolios Workshop Millenium Thinktank Birmingham November 2005 We were invited to give a talk to e-vocationalists about standards and frameworks as a means of creating an environment where interoperability and re-use can be achieved. This was an audience quite different from the Higher education area that we are positioned in, but it was clear from audience questions that we all experience the difficulties of applications that don’t work together or communicate.

JISC/Cetis Conference held at Heriot-Watt, Edinburgh, on 15 Nov 2005
We asked a group of about 20 teachers, technical providers, and managers to review the FREMA reference model and to provide detailed feedback on the usefulness of the reference model and how well the work of the group's individuals and organisations could be represented in and by the reference model.

9th International CAA Conference Loughborough July 2005
Over 120 delegates attended the 9th CAA Conference. We presented a paper and also gave a presentation in the Jisc workshop strand This was our first opportunity to spread our ideas and work to the wider assessment community.

E-learn October 2005
Over 400 papers were presented at this year’s e-Learn conference in Vancouver We presented a paper describing our technique of using graphical concept maps of the domain to enable users to orientate and navigate themselves through the resources available in the assessment domain from their own mental models.

Orca off the Gulf Islands near Vancouver

Educa Online Conference in Berlin November 2005 Jisc reference model projects joined nearly 2300 delegates and speakers and 120+ exhibitors from more than 70 countries at OEB this year, ‘to listen to thought leaders, to share experiences with colleagues, to get advice from experts, to make contacts, and to get new stimuli and fresh ideas’. Frema took part in a pre-conference seminar “Did you hear the one about the ELF, the Model and the Learning Technologist?" which attracted delegates from a number of countries. The seminar was in the form of presentations from each of the reference model projects with time for questions from the delegates. A bonus was the opportunity for the reference projects to compare notes and explore differences and similarities in approach.

JISC CETIS Conference 'e-Framework, Priorities and Challenges for 2006', due to take place on 'Tuesday 15th November & Wednesday 16th November' at the 'Edinburgh Conference Centre', Heriot-Watt University Edinburgh.
At the JISC/Cetis Conference held at Heriot-Watt, Edinburgh, on 15 Nov 2005, a group of about 20 teachers, technical providers, and managers reviewed the FREMA reference model and provided detailed feedback on the usefulness of the reference model and how well the work of the group's individuals and organisations could be represented in and by the reference model

JISC e-Learning Projects 6 - 7 December. York
FREMA presented our reference model work to joint Programme Meeting JISC e-Learning Projects including those funded by the Scottish Funding Council under the e-Learning Transformation programme. This meeting took taking place on the 6 and 7 December in York

Dr Yvonne Howard