Equipment

Systems

Consulting

RCC staff are available to provide consultation in order to facilitate and optimize the use of RCC's resources.

Equipment

Linux Cluster (rcluster)

Code Compilation

	Portland Group (PGI)	GNU	MPICH/PGI	File extension
Fortran77	pgf77	g77	mpif77	.f
Fortran90	pgf90		mpif90	.f90
Fortran95	pgf95	gfortran		.f95
C	pgcc	gcc	mpicc	.c
C++	pgCC	g++	mpicxx	.C, .cpp

NOTE
The Linux cluster is currently comprised of both AMD Opterons and Intel Xeon processors. When compiling code with the PGI compilers, by default the executable file (binary) will be optimized for the type of processor on which the compilation was carried out (note: the rcluster headnode is an Opteron). However, you can compile for the other type of processor or generate a unified binary, which is optimized for both types of processors, using the options in the table below. We recommend that users generate unified binaries; however users who prefer to compile codes directly on a Xeon can do so in one of the interactive nodes. For more information on interactive nodes, see Running an Interactive Job.

The Portland Group (PGI) compilers are installed in /usr/local/pgi/linux86-64/7.2/bin/ and they are in all users' default path. The current PGI compiler version is 7.2. If interested (it is not required for code compilation), users can source the file /usr/local/bin/use_pgi_compilers.sh (if using bash shell) or /usr/local/bin/use_pgi_compilers.csh (if using csh or tcsh) to set up environment variables for using the PGI compilers.

The GNU compilers are installed in /usr/bin and this directory is in all users' default path.

Type
g++ -v
to check the version of the GNU compilers (3.4.6). If interested (it is not required for code compilation), users can source the file /usr/local/bin/use_gnu_compilers.sh (if using bash shell) or /usr/local/bin/use_gnu_compilers.csh (if using csh or tcsh) to set up environment variables for using the GNU compilers.

GNU compilers v. 4.1.2 are also installed on the system and they can be invoked with gcc4, g++4, and gfortran.

Please note that gfortran is not yet a fully conformant Fortran 95 compiler and it does not fully replace g77.

In general the PGI compilers perform significantly better than the corresponding GNU compilers, therefore it is our recommendation that you use the PGI compilers whenever possible.

Some commonly used compilation flags for the PGI compilers:
-O0	Specifies no optimization, recommended for code debugging
-O1	Specifies local optimization
-O2	Specifies global optimization (this is the default, same as using -O)
-O3	Includes -O1, -O2 and more aggressive optimization. Use with care.
-tp k8-64	Generates a binary that is optimized for Opterons.
-tp penryn-64	Generates a binary that is optimized for the quad-core Xeons.
-tp k8-64,penryn-64	Generates a unified binary that is optimized for Opterons and quad-core Xeons.
-fast	Chooses generally good optimization options for the platform. Type `pgcc -fast -help` to see the equivalent options.
-Mbounds	Performs runtime array bound check, recommended for code debugging
-g	Produces symbolic debug information in the object files.
-r8	Interpret REAL variables as DOUBLE PRECISION.
-B	Allow C++ style comments in C source code; these begin with '//' and continue until the end of the current line. pgcc only.
-Kieee	Perform floating-point operations in strict conformance with the IEEE 754 standard. The default compilation is -Knoieee, which uses faster but very slightly less accurate methods.

NOTE
When using optimization options, please check if your code becomes more efficient (in some cases optimization options will slow the code down) and if it still generates correct results. Many other compiler options are available. For more information on the PGI compilers, please refer to the user guides in the directory /usr/local/pgi/linux86-64/7.2/doc/

MPI Libraries for parallel jobs

There are two compilations of MPICH available on the system:

1. MPICH compiled with the PGI compilers

MPICH 1.2.7 binaries (mpif77, mpif90, mpicc, mpicxx, mpirun, etc) that use the PGI compilers are installed in /usr/local/pgi/linux86-64/7.2/mpi/mpich/bin/ and this directory is in all users' default path. Therefore, if you type e.g. mpirun without any explict path, you will be using this compilation of MPICH. Because the PGI compilers are in general more efficient than the GNU ones, we recommend that you use this compilation of MPICH whenever possible.

Example

To compile a Fortran90 MPI code:

rcluster> mpif90 -o program program.f90

An executable program will be created. You can add other PGI compiler flags (e.g. optimization flags) if appropriate. This compilation is equivalent to

rcluster> pgf90 -o program program.f90 -Mmpi

To run the executable program using for example 4 processors, type

rcluster> mpirun -np 4 -machinefile=host.list ./program

NOTE:
Because the working directory is not in users' default path, it is necessary to precede the binary by ./ when running it.

For more information on the -machinefile option, please refer to Running Jobs on rcluster.

2. MPICH compiled with the GNU compilers

MPICH 1.2.7 binaries (mpif77, mpif90, mpicc, mpicxx, mpirun, etc) that use the GNU compilers are installed in /opt/mpich/gnu/bin/ and this directory is NOT in users' default path. To use this compilation of MPICH, you need to source /usr/local/bin/use_gnu_mpich.sh (if using bash shell) or /usr/local/bin/use_gnu_mpich.csh (if using csh or tcsh). When running a batch job using this version of MPICH, you need to source the above files in your job submission script.

3. MPICH2 compiled with the PGI compilers

MPICH2 is a Message Passing Interface (MPI) library that implements both MPI-1 and MPI-2 standards. For more information on MPICH2, please see http://www.mcs.anl.gov/research/projects/mpich2. The procedure for running MPI code linked to MPICH2 is different from that for running code linked to MPICH.

MPICH2 1.0.8p1 binaries that use PGI compilers are installed in /usr/local/mpich2-1.0.8p1_pgi72/bin and this directory is NOT in users' default path. To use this installation of MPICH2 you need to use the full path to the executables, e.g. /usr/local/mpich2-1.0.8p1_pgi72/bin/mpicc, /usr/local/mpich2-1.0.8p1_pgi72/bin/mpif77, etc. For information on how to run executables linked to MPICH2, please refer to Running Jobs on rcluster.

4. MPICH2 compiled with the GNU compilers

MPICH2 1.1 binaries that use GNU compilers are installed in /usr/local/mpich2-1.1_gcc346/bin and this directory is NOT in users' default path. To use this installation of MPICH2 you need to use the full path to the executables, e.g. /usr/local/mpich2-1.1_gcc346/bin/mpicc, /usr/local/mpich2-1.1_gcc346/bin/mpif77, etc. For information on how to run executables linked to MPICH2, please refer to Running Jobs on rcluster.

5. OpenMPI compiled with the PGI compilers

OpenMPI is an open source Message Passing Interface (MPI) library that implements the MPI-2 standard. For more information on OpenMPI, please see http://www.open-mpi.org.

OpenMPI 1.3 binaries that use the PGI compilers are installed in /usr/local/openmpi-1.3_pgi72_eth/bin and this directory is NOT in users' default path. To use this installation of OpenMPI you need to use the full path to the executables, e.g. /usr/local/openmpi-1.3_pgi72_eth/bin/mpicc, /usr/local/openmpi-1.3_pgi72_eth/bin/mpif77, etc. For information on how to run executables linked to OpenMPI, please refer to Running Jobs on rcluster.

Sample Makefile

Users are encouraged to compile codes using a makefile. Here is a very simple example of a makefile, which we call Makefile.sample. It is used to compile a Fortran77 program that is comprised of three parts, namely prog1.f, prog2.f and prog3.f. Note that lines 12 and 15 in Makefile.sample start with a Tab, not with blank spaces.

To compile the code:

rcluster> make -f Makefile.sample

To delete the object files:

rcluster> make -f Makefile.sample clean

Request Consultation

Please initiate requests for help or consultation via our:

Feel free to email or call us (we're in the UGA directory). The quickest way to get support is to use the RCC Support form.

For a complete list of all applications on RCC service, please check the complete list all applications page.