D5.1-5 Verification Plan: Acceptance test specification

Back to D5.1 System Specification

Overview

This chapter can almost be considered a separated document and constitutes the Testing and Verification Plan for the eCHASE System specified in the previous chapters.

Purpose

The purpose of the Verification Plan is to define responsibilities, schedule, assumptions, strategy, approach and test cases needed to assess the quality of the eCHASE System.

Scope and Schedule

This paragraph will define what will be tested and when. A basic assumption of this Plan is that Unit Tests are performed by developers before releasing each Unit of code. Thus the main scope of this Plan is System Level Tests, i.e. the object of the tests is the System resulting from the integration of the various modules. An overview of the modules that constitutes the architecture of the eCHASE System can be found in par. , and a summary of the functionalities provided by each module is available in par. . The eCHASE system will be released in an incremental way: first an internal prototype, available only to eCHASE Partners, then an external prototype, to which functionalities will be added in following releases as projected in the System Roadmap (see Appendix TBD – to be provided by ITI). The plan is to incrementally test the System after each release. Each testing session will verify the new functionalities and will possibly do regression testing for the functionalities already present -and tested- in the previous releases.

Resources, Roles and Responsibilities

Both the internal prototype and the first external prototype will be hosted by the eCHASE Partner IT Innovation. Access to the System will be available through the Internet. Each Quality Assurance (QA) Engineer dedicated to testing the System will be given a personal username and password. Each eCHASE Partner will be entitled to appoint one or more QA Engineer to test the system and help ensuring its quality. It is advisable that the Partners that provided content for the eCHASE System appoint at least one QA Engineer specifically to verify that the data and metadata provided by that Partner has been correctly handled by the System. More information on this content-based approach to testing can be found in par. .

A Testing Manager will be appointed by the eCHASE Partners for each test session and will be responsible to track testing activities and collect test results from each QA Engineer. The Testing Manager will aggregate test results and will apply the Acceptance Criteria (see par. ) to state the quality of the tested release.

Defect Handling

Defect Logging and Tracking

Effective testing is nothing without an accurate means of reporting defects, monitoring their progress, and managing their resolution. A Bug Tracking a Reporting System is essential in order to ensure a formal approach to defect tracking, it must be employed to make sure the effort expended on testing is well spent by ensuring that defects detected during testing are correctly managed and corrected.

Bugs resulting from the testing phases will be managed by the Scarab Bug Tracking System which will help developers and testing team in managing the bug resolution workflow.

Using Scarab it will be possible to have a central repository able to provide a consistent and detailed view of all bugs; reporting functionalities will help in monitoring the advancements of the testing and bug resolution phases.

Scarab main functionalities are summarized by the following bulleted list.

• Issue entry: The issue entry process is designed to help users create meaningful issues and avoid duplication. Attributes are used to describe issues and to check the issue database for similar entries.
• Templates for issue entry: the system can use templates in order to speed up the entry of recurrent issues
• Issue query: Users can easily find, access and modify issues using query functionalities
• Reports: reports functionalities allows teammates to create and save reports. Reports provide customizable and meaningful metrics.
• Email notifications: when specified events occur an email notification is sent to all parties associated with the event. Alert emails contain links to directly access the event.

Defect Classification

When logging defects an initial severity classification will be proposed by the QA Engineer that discovered the problem. See the following table for a definition of the three severity classes to be used when reporting a problem.

Severity Classification	Guideline for Severity Classification
Fatal	A problem that is a showstopper (i.e. application crashes or hangs during processing) and prevents a user from further using the application; or A problem that causes (or might cause) disastrous consequences (including financial losses)/unexpected results if the system is used due to distortion of functionality.
Major	A problem in the software/system which causes serious disruption of a major business function for which a workaround is available.
Minor	Minor Defects that cause (or could cause) small or negligible consequences for the system in question. These defects are easy to rectify.

Acceptance Criteria

Given the defect classification provided in par. , the tested release of the System can be considered having acceptable quality if the following condition is satisfied:

• Zero Fatal Defects
• N Major Defects with N less than 5 for the internal prototype and less than 2 for the external prototype and following releases
• M Minor Defects with M less than 40

Assumptions

This Verification Plan is based on the following assumptions:

• The object code will be fully unit and integration tested and made available on the Test environment by the date(s) given in the schedule for executing the test scenarios
• Each System Release will be followed by a dedicated testing session
• A Testing Manager is appointed by the eCHASE Partners for each testing session
• At least one QA Engineer is appointed by the Technical Partners and at least one QA Engineer is appointed by the Content Partners for each testing session

Testing approach

eCHASE provides a huge set of functionalities that range from simple login and access control functions to complex multilingual translation features. Given this wide and not homogeneous set of features we recognize that is not feasible to use the same testing approach in all circumstances, so we classify eCHASE functionalities in two different sets.

The first set will be composed by functionalities that can be tested and objectively measured using test cases. The second is composed by functionalities that cannot be easily tested using a test case driven approach and for which usually test results cannot be summarized by an accepted / not accepted flag but require a more complex evaluation.

In the second category fall many eCHASE functionalities, usually these features heavily rely on the goodness of provided data, such as multilingual translation and gazetteer mapping that depend on the metadata provided by content providers. The accuracy of searching results is another key function that cannot be easily tested and verified, actually even in this case goodness of results depends on the metadata provided. Furthermore eCHASE provides quite complex research features, such as the find similar images and find by colour, that need more complex analysis than simple test cases in order to define the accuracy of search results.

Moreover, in eCHASE some functionality are offered by tools, such as import and processing tools, that are not completely automated, but require human involvement in order to work. For example, import phase is a process with some tasks automated and others carried out by humans. This process cannot be verified directly but only indirectly, by measuring the goodness of other functionalities, e.g. the accuracy of a keyword search will depend on how good was the mapping of keywords in the import phase. For the import phase it’s not feasible to write Test Cases, because steps to follow usually depend on input data provided and cannot be exactly foreseen without having knowledge of it.

Given this classification of functionalities, we use two methodological approaches specialized for each class. The former is applied to functionalities that can objectively measured using test cases and where it’s possible to define objective metrics for acceptance of functionalities. For this class we provide a test book in paragraph .

For functionalities belonging to the second class we will use a more informal and subjective approach, based on a questionnaire that will be filled out by accurately selected personnel. In particular, for all functionalities where results are dependant on provided data, such as multilingual translation, thesaurus or gazetteer mapping, etc, we plan the direct involvement of content providers, the only actors that a have a deep knowledge of the provided data and that can evaluate the goodness and accuracy of elaboration results.

Architectural Overview

The following picture depicts the main architectural modules of the eCHASE demonstrator prototype.

eCHASE Architectural overview

As depicted, the main system modules are:

• Metadata Importer: this module takes care of metadata import phases, it aggregates heterogeneous datasets provided by content owners into a common structured format that is stored in the Unified Metadata Repository.
• Media Engine: the Media Engine module works on multimedia collections supplied by content owners, it provides import functionalities and takes care of serving requested content to users. Moreover it allows performing Content Based Queries on the repository.
• Search/Retrieve Web Service: is a stateless Web Service that exposes interfaces to perform metadata and content based queries. This component relies on recognized standards, such as SRW 1.1 and CQL ensuring it a high reutilization potential. In fact, using these standardized interfaces, third party software, even developed outside of the project context, can harvest and integrate with eCHASE repository.
• Web application: the web application provides a web interface to the data stored in the Metadata Repository and the Media Repository by interfacing with Search Retrieve Web Service (SRW) module. It’s a lightweight module with only minimal business logic; in fact all the search and retrieval logic is encapsulated in the underlying layers. SRW and Media Engine are loosely coupled with the web application, communications by these modules are rely on HTTP and SOAP Protocols.

System Functionalities

The objective of this paragraph is to provide a list of all the functionalities provided by eCHASE demonstrator and a categorisation based on the aforementioned testing approaches.

Metadata Importer

• Metadata formats conversion: metadata provided in heterogeneous format are transformed in a common format. Multilingual translation is performed in order to have metadata available in all the supported languages.
• Dirty data cleaning: data provided in a format that is not ready to be elaborated (e.g. dirty date format) are normalized.
• Unstructured fields indexing: fields with an unstructured format, such as free text description, are indexed in order to be easily queried.
• Metadata mapping to thesauri and common structures/dictionaries: metadata fields are mapped to preconfigured structures.
• Metadata mapping to common schema/ontology
• Original legacy metadata storing: original metadata is maintained in order to be usable at a later stage.
• Processed metadata storing: metadata are stored in a centralised repository (Unified Metadata Repository).

Media Engine

• Media import functionalities: provided media are imported in the eCHASE media repository
• Media descriptors generation and indexing: descriptors are generated, indexed and stored in dedicated repository
• Media comparison functions and content base searching: media engine module provides functions that allow other modules to perform content base queries, e.g. find by similar colour, find similar images, etc.
• Media conversion: media engine is able to perform manipulation on provided digital contents, some of these are:
  • Thumbnails for images
  • Key frames for videos
  • Media conversion
  • Media Resize
• Serving Media and Media descriptors: stored digital contents and descriptors are accessible using services provided by Media Engine.

Search Retrieve Web Service

• Media based searching: it allows performing content based queries using SRW standard (for example, finding images that are similar to a provided image).
• Textual metadata searching: it allows contents to be retrieved by searching metadata (for example, finding all media that depicts a particular person).
• Searching by concept (for example, by querying with respect to people - who, art objects and representations - what, events and activities - when, places – where, and methods and techniques - how)

Web application

• Search by metadata web interface: this web interface allows users to perform metadata based queries, it provides advanced functionalities such as
  • Google-style free text searching
  • Search refining by using filtering capabilities
  • Browsing of thesauri for powerful searching of well structured data
  • Powerful date handling for searching date ranges
• Search by content web interface: provides the interface for performing content based queries.
• Displaying Media Content: allows users to visualize media content, images and video.
• Displaying Unified Metadata: the web application is able to display unified metadata to users.
• Provide display of the pre-mapped legacy record
• Ability to annotate records
• Search saving: the system will provide functionalities that let users to save their own searches and use them in a second time.
• Provide a light box to organize searches and objects found: search results can be organized in a personal structure.
• Ability to annotate content packages created with the light box: users can add annotations to light box objects stored in their own light box.
• Provide export capabilities for media and metadata: contents in the light box can be exported in a packaged format and than managed outside of the eCHASE control
• Authentication: system provides an authentication mechanism based on username and password
• Self registration: a self registration module can be used in order to allows users to create their own accounts
• Help System: a comprehensive help will illustrate eCHASE’s functions to users.

Functionalities Categorisation

In the following table listed functionalities are categorised in function of the testing approach that will be used for the verification phase.

Module	Cod	Functionality	'TB'	'Q'	'Ref'
Metadata Importer	1	Metadata formats conversion		X	16-19
	2	Dirty data cleaning		X	16-19
	3	Unstructured fields indexing		X	16
	4	Metadata mapping to thesauri and common structures/ dictionaries		X	16
	5	Metadata mapping to common schema/ontology		X	16
	6	Original legacy metadata storing		X
	7	Processed metadata storing		X	16-19
Media Engine	8	Media import functionalities		X	18
	9	Media descriptors generation and indexing		X	17
	10	Media comparison functions and Content base searching		X	17
	11	Media conversion		X	18
	12	Serving Media and Media descriptors		X	18
Search Retrieve Web Service	13	Media based searching	X	X	17
	14	Textual metadata searching	X	X	16
	15	Searching by concept	X	X	??
Web application	16	Search by metadata web interface	X
	17	Search by content web interface	X
	18	Displaying Media Content	X
	19	Displaying Unified Metadata	X
	20	Provide display of the pre-mapped legacy record	X
	21	Ability to annotate records	X
	22	Search saving	X
	23	Provide a light box to organize searches and objects found	X
	24	Ability to annotate content packages created with the light box	X
	25	Provide export capabilities for media and metadata	X
	26	Authentication	X
	27	Self registration	X
	28	Help system	X

Test Book

Web application

TC11-Help System

Back to D5.1 System Specification