Face to Face Meeting, Chicago, July 20-22

Agenda

1. Registry
   • Using the registry
   • Integration with Globus and VDT
2. XDTM and VDT
3. Provenance

Detailed Agenda: Wednesday 20/7

• Grimoires and VDS
  • Demo of registry
  • Publication of services and applications
  • Publication of transformations
  • ISI catalog
  • Defining a common adapter interface

Detailed Agenda: Thursday 21/7

• Grimoires and VDS
  • Roadmap
• Provenance
  • PASOA/Eu Provenance provenance architecture
  • The ICE experiment and use of VDT
• Finalisation of sigmod paper

Detailed Agenda: Friday 22/7

• Provenance roadmap
• Breakout with Jim to share progress on provenance
• Grimoires and Globus
• DFDL vs XDTM
• Meeting, work and publication Planning

Location

The meeting will take place at the Universith of Chicago, in the Research Institutes building, Room 405, 5650 South Ellis Avenue Chicago, IL 60637 (57th and Ellis, NW corner).

Minutes

We went through the readme of the Grimoires Shell and ran it in parallel. http://twiki.grimoires.org/bin/view/Soca/ShellReadMe The shell interacts with the Grimoires registry located: http://fantasio.ecs.soton.ac.uk:8080/grimoires/services

We discussed the encoding of metadata in Grimoires. A not-fully-uptodate description can be found from the following paper http://www.ecs.soton.ac.uk/~lavm/papers/sciprog04.pdf

---

We identified several levels of integration between Grimoires and VDC:

1. Take a VDL workflow, catalog its transformations in Grimoires, reconstruct the tc.data file from Grimoires
2. Take a VDL workflow, catalog its transformations in Grimoires, build A TC API interface instance to call Grimoires directly
   • Requires us to map TC schema to Grimoires Schema
3. Perform the same but for workflows that invoke services, using Yong's command line script to invoke a Web Service
4. Do the same with an workflow invoking applications and web services

---

We looked at the TC Schema (expressed as SQL relations). For a transformation, we have:

• id is primary key
• namespace, name, version form a secondary key
• resourceid: site handle
• pfn: application path (physical name on a site)
• type: enumeration of some values (e.g., installed boolean, instalationl script, ...
• archid: foreign key to other table containing: operating system name, os version, glibc version, cpu architecture to determine binary compatibility
• lfnid, pfnid: mapping between logical and physical representation
• pfnprofile: information used to setup environment variables and specific dagman/globus configurations when running a specific transformation (env profile, globus profile, condor profile, dagman profile, vds profile, hints selector, selector profile)

Through the portal, Yong presented the "shim" capable of making web service invocations for VDL transformations.

Example: http://twiki.grimoires.org/pub/Soca/VirtualDataLanguagePreprocessing/tc.data

---

Mike, Luc and Yong discussed Luc's comments on the sigmod record paper.

Discussion about the roles of registry, provenance, vdc

Transformation: a procedure. Derivation: procedure call.

---

TC API:

• online javadoc, occasionally updated: http://www.griphyn.org/workspace/VDS/javadoc/
• TC API therein: http://www.griphyn.org/workspace/VDS/javadoc/org/griphyn/common/catalog/TransformationCatalog.html
• The full and most recent Java documentation is also in the $VDS_HOME/doc/javadoc subdir of each binary distribution.
• TC-UpdatedDB.pdf: ER diagram for the TC database

---

Roadmap

1. In the fmRI application, identify AIRsuite transformations (8 atomic, 2 workflows)
   • http://bishopw.loni.ucla.edu/AIR5/programs.html
   • http://bishopw.loni.ucla.edu/AIR5/programsabc.html
   • Yong to give us VDL for transformations and tc.data file
2. Publish their descriptions in Grimoires (description should include XDTM types)
3. Produce some command line scripts that query Grimoires to discover transformations of a given functionality, taking some input types, producing some output types, etc
   • 15 Sept 05 , lead: Soton
4. Write a command line tool that helps compose and edit compound transformations; planning support based on performance prediction; semantic type checking
   • write two typecheckers for a compound and for a workflow (exact matching of logical types or physical types)
   • 15 Oct 05 , lead: Chicago

---

Air type hierachy:

   image_coregistration
o   linear
      align_linear
o   non-linear
      align_warp
   image_realignment
   image_spatial_smoothing
o      softmean
   image_slice_time_correction

fmRI workflows and tc.data file:

• fmridc.vdl: fmri vdl workflow 1
• rundg_test.vdl: fmri vdl workflow 2
• tc.data: fmri tc.data file

---

• schema mapping:
  

• provenance.obj: schema mapping

• ptc_api.txt: ptc_api

---

Provenance roadmap

• Top down:
  • Identify fmRI provenance use cases (for information, PASOA provenance requirements http://eprints.ecs.soton.ac.uk/10269/), how to capture requirements: http://twiki.gridprovenance.org/bin/view/Provenance/HowToCaptureRequirements deadline: 01 September , lead: Chicago
  • Identify queries over respective schemas to support use cases, deadline: 15 September , lead: Chicago (for their data model) and Soton (for their data model)

• Bottom up
  • Identify the implicit information underpinning the VDL computation model that must be made explicit in some form of p-assertions, deadline: 15 September , lead: Soton interacting with Jens

• Assess situation, deadline: 30 September , AG meeting to decide what use case to implement with what level of integration between systems.

levels of integration:

• conceptual integration (i.e. just ideas)
• schema level mapping, leading to
• import from ptc to PReServ
• interface level integration, leading to
• shared software

Software:

• • Recording p-assertions
  • Generalize tool for submitting p-assertions to provenance catalog and/or PReServ

---

Grimoires and GT4 roadmap:

• see ian's email
• integration fest between GT4 and Grimoires (weijian and victor): security, aggregrator, descriptions as ws-resources(?)
• use cases: for aggregator and descriptions as resources, deadline: 30 August, lead: Ian
• use case for crypto assertions
• to discuss XACML access control assertions

---

XDTM vs DFDL

http://forge.gridforum.org/projects/dfdl-wg/

Roadmap:

• To produce an up to date spec, deadline: 30 August, lead Jim
• Use DFDL approach to describe existing data sets, e.g., fmRI, DR2
• Gap analysis
• Review October Boston

---

Planning

Timetable:

• 15 September: Produce some command line scripts that query Grimoires, lead: soton
• 15 October: write two typecheckers for a compound transformation and for a workflow, lead: chicago
• 01 September: Identify fmRI provenance use cases, lead: Chicago
• 15 September, schema mapping exercise, lead: Soton interacting with Jens
• 15 September: Identify queries over respective schemas to support use cases, lead: Chicago (for their data model) and Soton (for their data model)
• 30 August: rdf schema for provenance, lead soton
• 15 September: analysis of mapping of provenance schema to triple store, lead: NCSA
• 30 September: AG: Assess situation
• 30 Augusut: DFDL spec, lead: Jim

Meetings

• Soton, to find out about visa status of victor and Weijian, for grimoires/gt4 integration fest (sometimes in autumn)
• Integration fest for provenance (to decide after AG)

• Provenance workshop
  • Sollicit upto 6 pages position papers
  • PC selects papers for presentation at workshop
  • 20 minutes talk based on position paper
  • Publish papers on web
  • At workshop, decide if special issue on provenance is suitable, don't advertise it
  • Proposed date March, location: Chicago
  • Wednesday lunchtime till Friday lunchtime, March 8-10???
  • Name: IPAW
  • Maximum number of participants of about 50
  • Deadline for submission Dec 1 (no extension), feedback Dec 15th

Publications

-- LucMoreau - 15 Jul 2005

to top