PUBLIC INTERFACE ~ PUBLIC DATA ~ PUBLIC ROUTINES ~ NAMELIST ~ DIAGNOSTIC FIELDS ~ ERROR MESSAGES ~ REFERENCES ~ NOTES

## Module mpp_mod

Contact: V. Balaji
Reviewers:
Change History: WebCVS Log
RCS Log: RCS Log

#### OVERVIEW

`mpp_mod`, is a set of simple calls to provide a uniform interface to different message-passing libraries. It currently can be implemented either in the SGI/Cray native SHMEM library or in the MPI standard. Other libraries (e.g MPI-2, Co-Array Fortran) can be incorporated as the need arises.

The data transfer between a processor and its own memory is based on `load` and `store` operations upon memory. Shared-memory systems (including distributed shared memory systems) have a single address space and any processor can acquire any data within the memory by `load` and `store`. The situation is different for distributed parallel systems. Specialized MPP systems such as the T3E can simulate shared-memory by direct data acquisition from remote memory. But if the parallel code is distributed across a cluster, or across the Net, messages must be sent and received using the protocols for long-distance communication, such as TCP/IP. This requires a ``handshaking'' between nodes of the distributed system. One can think of the two different methods as involving `put`s or `get`s (e.g the SHMEM library), or in the case of negotiated communication (e.g MPI), `send`s and `recv`s.
The difference between SHMEM and MPI is that SHMEM uses one-sided communication, which can have very low-latency high-bandwidth implementations on tightly coupled systems. MPI is a standard developed for distributed computing across loosely-coupled systems, and therefore incurs a software penalty for negotiating the communication. It is however an open industry standard whereas SHMEM is a proprietary interface. Besides, the `put`s or `get`s on which it is based cannot currently be implemented in a cluster environment (there are recent announcements from Compaq that occasion hope).
The message-passing requirements of climate and weather codes can be reduced to a fairly simple minimal set, which is easily implemented in any message-passing API. `mpp_mod` provides this API.
Features of `mpp_mod` include:
1) Simple, minimal API, with free access to underlying API for more complicated stuff.
2) Design toward typical use in climate/weather CFD codes.
3) Performance to be not significantly lower than any native API.
This module is used to develop higher-level calls for domain decomposition and parallel I/O.
Parallel computing is initially daunting, but it soon becomes second nature, much the way many of us can now write vector code without much effort. The key insight required while reading and writing parallel code is in arriving at a mental grasp of several independent parallel execution streams through the same code (the SPMD model). Each variable you examine may have different values for each stream, the processor ID being an obvious example. Subroutines and function calls are particularly subtle, since it is not always obvious from looking at a call what synchronization between execution streams it implies. An example of erroneous code would be a global barrier call (see mpp_sync below) placed within a code block that not all PEs will execute, e.g:

       if( pe.EQ.0 )call mpp_sync()

Here only PE 0 reaches the barrier, where it will wait indefinitely. While this is a particularly egregious example to illustrate the coding flaw, more subtle versions of the same are among the most common errors in parallel code.
It is therefore important to be conscious of the context of a subroutine or function call, and the implied synchronization. There are certain calls here (e.g `mpp_declare_pelist, mpp_init, mpp_malloc, mpp_set_stack_size`) which must be called by all PEs. There are others which must be called by a subset of PEs (here called a `pelist`) which must be called by all the PEs in the `pelist` (e.g `mpp_max, mpp_sum, mpp_sync`). Still others imply no synchronization at all. I will make every effort to highlight the context of each call in the MPP modules, so that the implicit synchronization is spelt out.
For performance it is necessary to keep synchronization as limited as the algorithm being implemented will allow. For instance, a single message between two PEs should only imply synchronization across the PEs in question. A global synchronization (or barrier) is likely to be slow, and is best avoided. But codes first parallelized on a Cray T3E tend to have many global syncs, as very fast barriers were implemented there in hardware.
Another reason to use pelists is to run a single program in MPMD mode, where different PE subsets work on different portions of the code. A typical example is to assign an ocean model and atmosphere model to different PE subsets, and couple them concurrently instead of running them serially. The MPP module provides the notion of a current pelist, which is set when a group of PEs branch off into a subset. Subsequent calls that omit the `pelist` optional argument (seen below in many of the individual calls) assume that the implied synchronization is across the current pelist. The calls `mpp_root_pe` and `mpp_npes` also return the values appropriate to the current pelist. The `mpp_set_current_pelist` call is provided to set the current pelist.

#### OTHER MODULES USED

      shmem_interface
                  mpi
    mpp_parameter_mod
         mpp_data_mod

#### PUBLIC INTERFACE

F90 is a strictly-typed language, and the syntax pass of the compiler requires matching of type, kind and rank (TKR). Most calls listed here use a generic type, shown here as `MPP_TYPE_`. This is resolved in the pre-processor stage to any of a variety of types. In general the MPP operations work on 4-byte and 8-byte variants of `integer, real, complex, logical` variables, of rank 0 to 5, leading to 48 specific module procedures under the same generic interface. Any of the variables below shown as `MPP_TYPE_` is treated in this way.

mpp_error:
  Error handler.
mpp_init:
  Initialize `mpp_mod`.
mpp_exit:
  Exit `mpp_mod`.
mpp_malloc:
  Symmetric memory allocation.
mpp_set_stack_size:
  Allocate module internal workspace.
mpp_max:
  Reduction operations.
mpp_sum:
  Reduction operation.
mpp_transmit:
  Basic message-passing call.
mpp_broadcast:
  Parallel broadcasts.
mpp_chksum:
  Parallel checksums.

#### PUBLIC DATA

None.

#### PUBLIC ROUTINES

1. #### mpp_error

       call mpp_error ( errortype, routine, errormsg )

   DESCRIPTION
     It is strongly recommended that all error exits pass through `mpp_error` to assure the program fails cleanly. An individual PE encountering a `STOP` statement, for instance, can cause the program to hang. The use of the `STOP` statement is strongly discouraged.
     Calling mpp_error with no arguments produces an immediate error exit, i.e:

             call mpp_error
             call mpp_error(FATAL)

     are equivalent.
     The argument order

             call mpp_error( routine, errormsg, errortype )

     is also provided to support legacy code. In this version of the call, none of the arguments may be omitted.
     The behaviour of `mpp_error` for a `WARNING` can be controlled with an additional call `mpp_set_warn_level`.

             call mpp_set_warn_level(ERROR)

     causes `mpp_error` to treat `WARNING` exactly like `FATAL`.

             call mpp_set_warn_level(WARNING)

     resets to the default behaviour described above.
     `mpp_error` also has an internal error state which maintains knowledge of whether a warning has been issued. This can be used at startup in a subroutine that checks if the model has been properly configured. You can generate a series of warnings using `mpp_error`, and then check at the end if any warnings has been issued using the function `mpp_error_state()`. If the value of this is `WARNING`, at least one warning has been issued, and the user can take appropriate action:

             if( ... )call mpp_error( WARNING, '...' )
             if( ... )call mpp_error( WARNING, '...' )
             if( ... )call mpp_error( WARNING, '...' )
             ...
             if( mpp_error_state().EQ.WARNING )call mpp_error( FATAL, '...' )

   INPUT

     `errortype` | One of `NOTE`, `WARNING` or `FATAL` (these definitions are acquired by use association). `NOTE` writes `errormsg` to `STDOUT`. `WARNING` writes `errormsg` to `STDERR`. `FATAL` writes `errormsg` to `STDERR`, and induces a clean error exit with a call stack traceback.

2. #### mpp_init

       call mpp_init ( flags )

   DESCRIPTION
     Called to initialize the `mpp_mod` package. It is recommended that this call be the first executed line in your program. It sets the number of PEs assigned to this run (acquired from the command line, or through the environment variable `NPES`), and associates an ID number to each PE. These can be accessed by calling `mpp_npes` and `mpp_pe`.
   INPUT

     `flags` | `flags` can be set to `MPP_VERBOSE` to have `mpp_mod` keep you informed of what it's up to. [integer]

3. #### mpp_exit

       call mpp_exit ()

   DESCRIPTION
     Called at the end of the run, or to re-initialize `mpp_mod`, should you require that for some odd reason.
     This call implies synchronization across all PEs.

4. #### mpp_malloc

       call mpp_malloc ( ptr, newlen, len )

   DESCRIPTION
     This routine is used on SGI systems when `mpp_mod` is invoked in the SHMEM library. It ensures that dynamically allocated memory can be used with `shmem_get` and `shmem_put`. This is called symmetric allocation and is described in the `intro_shmem` man page. `ptr` is a Cray pointer (see the section on portability). The operation can be expensive (since it requires a global barrier). We therefore attempt to re-use existing allocation whenever possible. Therefore `len` and `ptr` must have the `SAVE` attribute in the calling routine, and retain the information about the last call to `mpp_malloc`. Additional memory is symmetrically allocated if and only if `newlen` exceeds `len`.
     This is never required on Cray PVP or MPP systems. While the T3E manpages do talk about symmetric allocation, `mpp_mod` is coded to remove this restriction.
     It is never required if `mpp_mod` is invoked in MPI.
     This call implies synchronization across all PEs.
   INPUT

     `ptr` | a cray pointer, points to a dummy argument in this routine.
     `newlen` | the required allocation length for the pointer ptr [integer]
     `len` | the current allocation (0 if unallocated). [integer]

5. #### mpp_set_stack_size

       call mpp_set_stack_size (n)

   DESCRIPTION
     `mpp_mod` maintains a private internal array called `mpp_stack` for private workspace. This call sets the length, in words, of this array.
     The `mpp_init` call sets this workspace length to a default of 32768, and this call may be used if a longer workspace is needed.
     This call implies synchronization across all PEs.
     This workspace is symmetrically allocated, as required for efficient communication on SGI and Cray MPP systems. Since symmetric allocation must be performed by all PEs in a job, this call must also be called by all PEs, using the same value of `n`. Calling `mpp_set_stack_size` from a subset of PEs, or with unequal argument `n`, may cause the program to hang.
     If any MPP call using `mpp_stack` overflows the declared stack array, the program will abort with a message specifying the stack length that is required. Many users wonder why, if the required stack length can be computed, it cannot also be specified at that point. This cannot be automated because there is no way for the program to know if all PEs are present at that call, and with equal values of `n`. The program must be rerun by the user with the correct argument to `mpp_set_stack_size`, called at an appropriate point in the code where all PEs are known to be present.
   INPUT

     `n` | [integer]

6. #### mpp_max

       call mpp_max ( a, pelist )

   DESCRIPTION
     Find the max of scalar a the PEs in pelist result is also automatically broadcast to all PEs
   INPUT

     `a` | `real` or `integer`, of 4-byte of 8-byte kind.
     `pelist` | If `pelist` is omitted, the context is assumed to be the current pelist. This call implies synchronization across the PEs in `pelist`, or the current pelist if `pelist` is absent.

7. #### mpp_sum

       call mpp_sum ( a, length, pelist )

   DESCRIPTION
     `MPP_TYPE_` corresponds to any 4-byte and 8-byte variant of `integer, real, complex` variables, of rank 0 or 1. A contiguous block from a multi-dimensional array may be passed by its starting address and its length, as in `f77`.
     Library reduction operators are not required or guaranteed to be bit-reproducible. In any case, changing the processor count changes the data layout, and thus very likely the order of operations. For bit-reproducible sums of distributed arrays, consider using the `mpp_global_sum` routine provided by the `mpp_domains` module.
     The `bit_reproducible` flag provided in earlier versions of this routine has been removed.
     If `pelist` is omitted, the context is assumed to be the current pelist. This call implies synchronization across the PEs in `pelist`, or the current pelist if `pelist` is absent.
   INPUT

     `length` |
     `pelist` |

   INPUT/OUTPUT

     `a` |

8. #### mpp_transmit

       call mpp_transmit ( put_data, put_len, put_pe, get_data, get_len, get_pe )

   DESCRIPTION
     `MPP_TYPE_` corresponds to any 4-byte and 8-byte variant of `integer, real, complex, logical` variables, of rank 0 or 1. A contiguous block from a multi-dimensional array may be passed by its starting address and its length, as in `f77`.
     `mpp_transmit` is currently implemented as asynchronous outward transmission and synchronous inward transmission. This follows the behaviour of `shmem_put` and `shmem_get`. In MPI, it is implemented as `mpi_isend` and `mpi_recv`. For most applications, transmissions occur in pairs, and are here accomplished in a single call.
     The special PE designations `NULL_PE`, `ANY_PE` and `ALL_PES` are provided by use association.
     `NULL_PE`: is used to disable one of the pair of transmissions.
     `ANY_PE`: is used for unspecific remote destination. (Please note that `put_pe=ANY_PE` has no meaning in the MPI context, though it is available in the SHMEM invocation. If portability is a concern, it is best avoided).
     `ALL_PES`: is used for broadcast operations.
     It is recommended that `mpp_broadcast` be used for broadcasts.
     The following example illustrates the use of `NULL_PE` and `ALL_PES`:

             real, dimension(n) :: a
             if( pe.EQ.0 )then
                 do p = 1,npes-1
                    call mpp_transmit( a, n, p, a, n, NULL_PE )
                 end do
             else
                 call mpp_transmit( a, n, NULL_PE, a, n, 0 )
             end if

             call mpp_transmit( a, n, ALL_PES, a, n, 0 )

     The do loop and the broadcast operation above are equivalent.
     Two overloaded calls `mpp_send` and `mpp_recv` have also been provided. `mpp_send` calls `mpp_transmit` with `get_pe=NULL_PE`. `mpp_recv` calls `mpp_transmit` with `put_pe=NULL_PE`. Thus the do loop above could be written more succinctly:

             if( pe.EQ.0 )then
                 do p = 1,npes-1
                    call mpp_send( a, n, p )
                 end do
             else
                 call mpp_recv( a, n, 0 )
             end if

9. #### mpp_broadcast

       call mpp_broadcast ( data, length, from_pe, pelist )

   DESCRIPTION
     The `mpp_broadcast` call has been added because the original syntax (using `ALL_PES` in `mpp_transmit`) did not support a broadcast across a pelist.
     `MPP_TYPE_` corresponds to any 4-byte and 8-byte variant of `integer, real, complex, logical` variables, of rank 0 or 1. A contiguous block from a multi-dimensional array may be passed by its starting address and its length, as in `f77`.
     Global broadcasts through the `ALL_PES` argument to `mpp_transmit` are still provided for backward-compatibility.
     If `pelist` is omitted, the context is assumed to be the current pelist. `from_pe` must belong to the current pelist. This call implies synchronization across the PEs in `pelist`, or the current pelist if `pelist` is absent.
   INPUT

     `length` |
     `from_pe` |
     `pelist` |

   INPUT/OUTPUT

     `data(*)` |

10. #### mpp_chksum

        mpp_chksum ( var, pelist )

    DESCRIPTION
      `mpp_chksum` is a parallel checksum routine that returns an identical answer for the same array irrespective of how it has been partitioned across processors. `LONG_KIND`is the `KIND` parameter corresponding to long integers (see discussion on OS-dependent preprocessor directives) defined in the header file `fms_platform.h`. `MPP_TYPE_` corresponds to any 4-byte and 8-byte variant of `integer, real, complex, logical` variables, of rank 0 to 5.
      Integer checksums on FP data use the F90 `TRANSFER()` intrinsic.
      The serial checksum module is superseded by this function, and is no longer being actively maintained. This provides identical results on a single-processor job, and to perform serial checksums on a single processor of a parallel job, you only need to use the optional `pelist` argument.

               use mpp_mod
               integer :: pe, chksum
               real :: a(:)
               pe = mpp_pe()
               chksum = mpp_chksum( a, (/pe/) )

      The additional functionality of `mpp_chksum` over serial checksums is to compute the checksum across the PEs in `pelist`. The answer is guaranteed to be the same for the same distributed array irrespective of how it has been partitioned.
      If `pelist` is omitted, the context is assumed to be the current pelist. This call implies synchronization across the PEs in `pelist`, or the current pelist if `pelist` is absent.
    INPUT

      `pelist` |
      `var` |

#### DATA SETS

None.

#### ERROR MESSAGES

None.

top