Languages and message passing

Data parallel programming languages

Fortran-90

This is a superset of Fortran-77. Fortran-90 provides significant new facilities some of which, such as array syntax, make it easier for a compiler to determine that operations may be carried out concurrently.

Compilers available from NAG, IBM, DEC, Cray Research, and others.

Array Features

One of the most significant aspects of Fortran-90.

Arrays can be declared ALLOCATABLE. Compiler will defer allocating memory space to the array until execution.
Assignments may be made to entire arrays without use of subscripts.
Array section references - in which part of an array may be assigned to all or part of another array. The arrays must be conformable.

Array section references

INTEGER, DIMENSION(1:100) :: array1
INTEGER, DIMENSION(1:50) :: array2
INTEGER, DIMENSION(1:100) :: array3, array4

array1(1:50) = 5
array1(51:100) = 25
array2 = arr(1:50)
array3(51:100) = array2
array3(1:50) = array1(51:100)
array(1:100:2) = array2

Array Intrinsic Functions

Vector and matrix multiplication; for example, DOT_PRODUCT, MATMUL
Array reduction functions; for example, ALL, MAXVAL, SUM
Array inquiry functions; for example, SIZE, SHAPE
Array construction functions; for example, MERGE, SPREAD

Other features include

Control Structures.
Procedure interface blocks.
Modules.
Derived data types.
Pointers.

High Performance Fortran (HPF)

HPF is the latest set of extensions to the Fortran language and is still under development. Although the published specification has not yet been adopted by a Standards body almost all parallel machine vendors have announced implementation efforts.

Subset HPF

A minimal starting language defined to encourage early releases of compilers with HPF features.

Goals of HPF

Some goals of HPF, as laid down by the HPF Forum, are:

To support data parallel programming.
To enable high performance on parallel computers while not impeding performance on other machines.
To allow relatively easy conversion from existing sequential code to parallel code.

Data Parallelism

This is executing the same operations (either synchronously or asynchronously) in parallel on different sets of data. HPF expresses data parallelism in several ways.

Fortran-90 array expressions and assignments.
Array intrinsics defined in Fortran-90.
Array library functions defined in the HPF library.

There are also constructs to describe operations that can be performed in parallel if the computer has the resources:

	The FORALL statement.

	  FORALL (I = 0:9)
	    X(I+1) = X(I)
	  END FORALL

Data Mapping

HPF can describe how data is to be divided among the processors in a parallel machine:

DISTRIBUTE - is a directive that describes how an array is divided and distributed to processors in a regular way.
ALIGN - is a directive that describes how two arrays "line up".
There are other directives, including dynamic versions of the above.

Message Passing

In the message passing model, parallel processes do not have direct access to each others' local memory. Communication is achieved through exchange of data, using SEND and RECEIVE primitives.

Point-to-point communication - always involves exactly two processes. One process SENDS and the other RECEIVES.
Collective communication involves every process in a specified group.

PVM

Parallel Virtual Machine (PVM) is a software package that allows a heterogeneous network of computers (parallel, vector, and serial) to appear as a single concurrent computational resource - virtual machine.

PVM supports heterogeneity at the application, machine, and network level.
PVM has libraries that contain routines for initiating processes on other machines, for communicating between processes, and changing the configuration of the virtual machine.
PVM provides interfaces to programs in C and Fortran.
Multiple users can configure overlapping virtual machines.

Point-to-point communication

Uses system buffers at SEND and RECEIVE ends to pack and unpack data. However, recent versions e.g. Cray T3D avoid this.

	CALL PVMFSEND(TID, MSGTAG, INFO)

This is an asynchronous send - no synchronous version.

	CALL PVMFRECV(TID, MSGTAG, BUFID)

This is a blocking receive.

Collective communication

Tasks can broadcast messages to groups whether or not they belong to that group.

	CALL PVMFBCAST(GROUP, MSGTAG, INFO)

Availability of PVM

Vendors
Several vendors are supplying and supporting optimized versions of PVM for their multi-processor systems, including Cray Research, IBM, Convex, Intel, SGI, and DEC.
Public domain
- Available on netlib, which is a software distribution service set up on the Internet.
- Available by Email to: netlib@ornl.gov
  with the message: send index from pvm3
- or anonymous ftp from: cs.utk.edu pub/xnetlib

MPI

Message Passing Interface (MPI) is a new library specification for message-passing, proposed as a standard by a broadly based committee of vendors, implementors, and users. MPI provides source-code portability for message-passing programs while allowing efficient vendor implementations. Features of MPI are:

Support for separately compiled libraries.
A great deal of functionality based on experience from other message-passing systems.
To allow for implementations that can be used in heterogeneous environments.
Interfaces to programs in C and Fortran.

Point-to-point communication

There are four SEND modes.

Standard
Synchronous
Buffered
Ready

Standard Send

	CALL MPI_SEND( BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)

Receive

	CALL MPI_RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)

Collective communication

This involves every process in the specified communicator.
There exists a number of routines to distribute data amongst processes.

MPI_BCAST
MPI_SCATTER
MPI_GATHER
Plus others

Availability of MPI

Implementations from most major vendors of parallel machines.
Available public domain.

For example, the Argonne National Laboratory /Mississippi State University implementation. Available by anonymous ftp from
info.mcs.anl.gov in pub/mpi

Submitted by Mark Johnston,
last updated on 21 February 1995.