Provenance Challenge

Provenance Challenge: UvA^?/VL-e

Participating Team

Team and Project Details

• Short team name: UvA/VL-e
• Participant names: Victor Guevara, Ammar Benabdelkader, Adam Belloum
• Project URL: http://www.vl-e.nl/
• Project Overview:
• Relevant Publications:

Workflow Representation

The workflow is implemented using a Grid-based workflow system developed at the the University of Amsterdam, called WS-VLAM. The provenance is, however, handled with an initial version of PLIER (Provenance Layer Infrastructure for e-Science Resources) which is being developed by the Information Management Group at PCC UvA. Essentially, PLIER hides the details of handling and managing the provenace from WS-VLAM system when it executes the workflow. The WS-VLAM engine uses PLIER, in its API form, to generate provenance data from the workflow into an OPM graph.

It is clear that the workflow has to be adapted to the WS-VLAM system for its composition and execution. When performing this adaptation, we faced already multiple ways to adjust the command line workflow to its WS-VLAM counter part. It is possible, for instance, to execute some processes by farming them or employ sequential processes instead; or to create a 'big' workflow or use composite workflows (hierarchical). Some of these initial alternatives are described in the slides contained in this powerpoint file.

The following figures present the different stages to generate the provenace. On one hand, the Figure 1 shows the WS-VLAM Composer with the design, or representation, of the workflow. On the other hand, the OPM Graph generated by PLIER can be seen as a XML format, in Figuere 2, or as a diagram, in Figure 3.

Figure 1. WS-VLAM Composer	Figure 2. PLIER (XML Tree)	Figure 3. PLIER (Diagram)

Similarly to the previous figures, the below representations show the WS-VLAM Composer, in Figure 4, and the PLIER tool, in Figure 5, with a more detailed workflow.

Figure 4. WS-VLAM Composer (detailed workflow)	Figure 5. PLIER (detailed workflow)

Open Provenance Model Output

The following table show the latest OPM Graphs generated by PLIER from WS-VLAM workflows.

Case (Job ID)	OPM Graphs (XML format)
J062941	J062941-OPM.xml
J062942	J062942-OPM.xml
J062943	J062943-OPM.xml
J062944	J062944-OPM.xml
J062945	J062945-OPM.xml

Query Results

The PLIER repository is a relational database that is accessed by scientists indirectly through the GUI tool or programatically via the API. Basically, the PLIER repository handles the data but relies on the GUI end-user application to retrieve or query the information elements. These tools, however, are still under development and they may not be able to query rather browse the data. Although PLIER does not provide any low level manipulation mechanisms, it does not restrict the user from accessing through SQL commands. Thus, for the sake of providing some clarification, the OPM data will be then queried using both OQL and SQL.

Query 1

For a given detection, which CSV files contributed to it?

Our solution first notices that the WS-VLAM system is agnostic of what the processes do. The WS-VLAM engine schedules and submits each process for execution, while monitor its progress. The module instead executes its specific task, as a black box, without being aware about the existance of WS-VLAM system. Second, PLIER is meant to provide provenance to the workflow as a whole and, at this moment, it does not consider the granulaity contained by the modules. Under these circumtances, if the given detection is treated by the any module it, the generated provenace data by itself is insufficient to answer the query.

We can interpret this query from another point of view. In our data provenance, the filenames are provided as parameters to the modules to generate the output (or the input). The parameters are expressed in the OPM Graph as Agents, having an ID linking it to the detection and a data Value. Thus, in order to retrieve the CSV files that contributed to a detection, we perform the following query:

   SQL: select Agent.Value from AGENT where Agent.ID like '%Detection%'

Which returns:

    P2_J062945_B001_P2fits0_20081115_P2Detection.csv

Another possibility is to employ with a more general query that retrieves the CSV files participating in all events.

   SQL: select * from AGENT where Agent.value like '%.csv'

Which returns:

   P2_J062945_B001_P2fits0_20081115_P2Detection.csv
   P2_J062945_B001_P2fits0_20081115_P2FrameMeta.csv
   P2_J062945_B001_P2fits0_20081115_P2ImageMeta.csv

Comment:

• In this second view, we do not see any link between the database internal operations and the generated OPM graph. In the implementation the database is receiving a bulk file that imported into its internal tables. After performing this task, the only information from the database is the number of records either inserted, updated, or deleted. We are answering the query based on the id having 'Detection' or the values having '.csv' (part of the input file name). Clearly, it is not always the case that a file name or its extension reflect its content.

Query 2

The user considers a table to contain values they do not expect. Was the range check ('IsMatchTableColumnRanges') performed for this table?

Query 3

Which operation executions were strictly necessary for the Image table to contain a particular (non-computed) value?

Suggested Workflow Variants

Suggestions for Modification of the Open Provenance Model

OPM Specification

While implementing the necessary mechanisms to import and export information using the OPM model, based on the given XML Schema (http://openprovenance.org), we faced some inconveniences while parsing the XML tags. Therefore, we modified the schema to cope with those problems as well to match the database model from the repository. These changes to the original XML schema are summarized below:

• The modified schema basically contains the the same definitions but instead of using complexType constructs, we employ element definitions.
• The main problem for our parser was to unambiguously identify the concept account since the definition for AccountId and Account share the same element (account) and attribute (id).
• The modified schema takes a generic approach for ids rather than explicitly define ProcessId, AgentId, or AgentId.

In order to better clarify our points, we attached the revised XML schema OPMv101.revised.xsd

Conclusions

-- VictorGuevara - 02 Jun 2009
to top