System Programs and Libraries in the Material Science Grid Clusters

This is a report for the Material Science Grid Maintenance Project concerning the system programs, such as compilers and libraries for the clusters.

Goal

The main goal is to suggest, what system software, batch queue system, compilers and libraries are installed on the clusters and what are supported by CSC and recommend the levels of the support.

Principles

1. The primary goal is a reasonable, efficient and highly usable system environment for the clusters, common (or similar) to all sites.
2. CSC will install, maintain and support the minimum common system environment achieving the above goal.
3. CSC will document the system environment, so that both local administrators can maintain and the users can work with the environment.
4. CSC will not give direct user support (except providing for the documentation mentioned above).
5. The direct user support is the responsibility of the local support people.
6. Additional programs and modifications required on each site are the responsibility of the local support people.
7. Changes to these support policies will be discussed with the user groups and implemented according to the principles agreed upon.

Support levels

There is a three level support structure for system software support by CSC:

• Highest level 2: "Full" support, indicated by supp in the table below.
  • CSC will install the feature, upgrade it when necessary, fix the known bugs, and help the local support to pinpoint the bugs. The feature is updated within reasonable time, when the upgrade or bug fix is available.
• Medium level 1. Installation only, indicated by inst in the table below.
  • CSC will install the feature initially, but will not support it. The feature is "as is".
  • Especially this means, that all the upgrades, bug fixes, and finding the bugs and correcting them are the responsibility of the local support people (or users themselves).
  • If the feature is part of the operating system distribution (e.g. editors in the table below), the level 1 support includes installing upgrades when the operating system is normally upgraded (probably not very often) or when a security fix for the package becomes available.
  • For each feature in this category there should be one group in charge of the feature. This responsible group will install and test the new releases and bug fixes etc.
  • CSC will then add the feature to the installation system, so that it will be installed to all sites (if they wish, can be selected on a per site basis) next time a normal system maintenance is scheduled by CSC.
• Lowest level 0: No support by CSC, supported locally if required. Indicated by local in the table below.
  • Local administrators or users install, maintain and support these features completely by themselves, if these features are required.
  • The groups are encouraged to elevate the desired features to medium level 1 by taking responsibility of the feature in the sense required for that level; especially if several groups are interested in the same feature.

The System Software as a Table

Compilers

The GNU compilers are included in all Linux distributions. They are not the most suitable ones for the clusters, however. A version of them are installed with the Rocks distribution, and that version is supported at the start only "as is", medium level 1 (indicated by inst in the table). The versions of GNU compilers in Rocks are not the newest; groups can install locally more recent versions if they are needed. CSC may later consider the need for new GNU compilers too.

One (commercial) compiler package will be supported (full support level 2). By choosing the compiler package carefully, one can drastically affect the time needed for software development. The downside is that these commercial packages can cost quite a bit.

In the M-grid Cluster Administration Meeting (2004-08-13) the Portland Group Compiler package was selected as the initial choice. It was estimated that a license for 5 simultaneous users is enough, at least initially. However, due to problems detected with compiling certain codes with the PGI compiler, only 2 simultaneous users license has been bought first. CSC will bear the costs of these licenses. CSC will also host the licenses on it's license server.

Portland Group (PGI) Compiler package includes all needed compilers: Fortran 90/95, Fortran 77, HPF = High Performance Fortran, C and C++. There are also optimized versions of the LAPACK/BLAS, MPICH, ScaLAPACK libraries and support for message passing LAM/MPI.

PathScale is a new compiler, seemingly based on the GNU compiler and compatible with it. It is partly open source (GPL), partly proprietary (and commercial), and especially geared and optimized towards AMD64-architecture. It seems to produce faster code than the Portland Group and GNU compilers in most cases and is currently the recommended compiler.

Similarly Intel package consists of all needed compilers and libraries. One should also watch closely its future 64-bit features.

Absoft and NAGWare Fortran Compiler will probably be used by one group (each).

Message Passing Libraries

Both MPICH and LAM will be installed and supported on full support level 2. If there are some unforeseen difficulties for having them both, the more efficient LAM is the preferred choice. In that case, MPICH will be supported only on medium inst level 1 or even local support level 0, but this scenario is not very probable.

Subroutine Libraries

At present, none of the commercial libraries (NAG; IMSL, Linda) will be acquired. They could be very expensive for these kind of clusters, because there are so many processors. And it seems that there is no real need (nor money) for these commercial libraries. Gaussian03 uses Linda library for parallelization in Linux, but it will probably not be acquired. If some groups need these libraries they should obtain licenses and install them locally.

So CSC will support only free (open source) or vendor libraries. CSC will install, maintain and support only the most important libraries, and their parallel versions are usually at lower support level in comparison to the serial ones.

The other mentioned libraries such as NetCDF, GSL, DISLIN, SLATEC will be initially at local-support-only level 0, but later they can be elevated to the medium inst level 1 ("as is") when needed and as the case may be. This has to be agreed separately.

Some of the scientific applications, which are to be installed, could require specific libraries to be able to run, especially if they are acquired as a binary version. This will be taken into account when reconsidering the support levels for the libraries.

The libraries are made available at least for the primary compiler (at present PGI) and primarily in the 64-bit version. However, libraries for all supported compilers (at present this applies to GNU in addition of PGI) and the 32-bit versions are included in the support as needed and as available (or compilable). The support level of the library is never greater than that of the compiler, and in some cases it could be lower. Especially the 32-bit versions are not efficient on the cluster 64-bit architecture, and hence of lower importance, but sometimes they are required by a special software package.

Programming Environment and Tools

The default editor will be Emacs, which is standard in all Linux/Unix distributions. Also ed and vi are included. Debuggers, profiling tools and other programming tools are usually included in the compilers or message passing libraries.

Here we probably do not need installations or support separate from the general support already included in the support of the operating system, compilers or libraries.

Support tasks 

The task include:

• fetching of the packages and uncompressing of the packages
• configuration
• compiling
• installation
• building suitable packages for distribution into nodes
• discovering and solving bugs
• evaluating the necessary updates, upgrades and patches
• documenting the changes and installation details
• following security issues related to the packages
• following the development and features of these software packages generally
• communicating and discussing with the groups

In many cases, the need for support lessens after initial installation. However, one major problem could easily take several  several weeks or even months to solve and patch.

Changelog

2005-09-14 Version 1.1, Pathscale support added (AJT)

2004-09-27 Version 1.01, number of PGI compiler licenses fixed (AJT)

2004-09-08 Version 1.00, distributed to groups on 2004-09-24