| Accessibility Statement

Equipment

Systems

Consulting

RCC staff are available to provide consultation in order to facilitate and optimize the use of RCC's resources.

Equipment

Using the Qlogic Infiniband on rcluster

A portion of the Intel dual processor quad-core Xeon nodes (32 nodes total) on rcluster, obtained with an NSF/MRI grant, are interconnected with a Qlogic Infiniband switch. In order to run MPI programs using the Infiniband switch you have to compile your code with the MPI libraries built for Infiniband, use appropriate batch job submission scripts and submit your job to the batch queues that specifically target the nodes interconnected with Infiniband.

Note
You need to login to node c11-1 to compile a program that will use Infiniband for internodal communication.

That is, first login to rcluster, then on an rcluster command prompt type

ssh c11-1

Once you are logged into c11-1, you can compile/link your code using the MPI libraries described below.

MPI libraries built for Infiniband

The following MPI libraries installed on rcluster support Infiniband:

1. MVAPICH with PGI compilers:

MVAPICH version 1.1, compiled with PGI 7.2, is available in /usr/local/mvapich-1.1_pgi72 and the executables (mpicc, mpiCC, mpif90, mpirun, etc) are installed in /usr/local/mvapich-1.1_pgi72/bin and this directory is not in users' default path. To compile a code and link to this version of MVAPICH, please use the full path. For example, to compile a Fortran90 code, use

/usr/local/mvapich-1.1_pgi72/bin/mpif90 prog.f90

You can also add other compilation options, as appropriate.

For information on MVAPICH, please see http://mvapich.cse.ohio-state.edu/

2. MVAPICH with GNU compilers:

MVAPICH version 1.1, compiled with GNU compilers (gcc/g++ v. 3.4.6 and gfortran v. 4.1.2, is available in /usr/local/mvapich-1.1_gcc346 and the executables (mpicc, mpiCC, mpif90, mpirun, etc) are installed in /usr/local/mvapich-1.1_gcc346/bin and this directory is not in users' default path. To compile a code and link to this version of MVAPICH, please use the full path. For example, to compile a Fortran90 code, use

/usr/local/mvapich-1.1_gcc346/bin/mpif90 prog.f90

You can also add other compilation options, as appropriate.

3. MVAPICH2 with PGI compilers:

MVAPICH2 version 1.2, compiled with PGI 7.2, is available in /usr/local/mvapich2-1.2_pgi72 and the executables (mpicc, mpiCC, mpif90, mpirun_rsh, etc) are installed in /usr/local/mvapich2-1.2_pgi72/bin and this directory is not in users' default path. To compile a code and link to this version of MVAPICH2, please use the full path. For example, to compile a Fortran90 code, use

/usr/local/mvapich2-1.2_pgi72/bin/mpif90 prog.f90

You can also add other compilation options, as appropriate.

For information on MVAPICH2, please see http://mvapich.cse.ohio-state.edu/

4. MVAPICH2 with GNU compilers:

MVAPICH2 version 1.2, compiled with GNU compilers (gcc/g++ v. 3.4.6 and gfortran v. 4.1.2), is available in /usr/local/mvapich2-1.2_gcc346 and the executables (mpicc, mpiCC, mpif90, mpirun_rsh, etc) are installed in /usr/local/mvapich2-1.2_gcc346/bin and this directory is not in users' default path. To compile a code and link to this version of MVAPICH2, please use the full path. For example, to compile a Fortran90 code, use

/usr/local/mvapich2-1.2_gcc346/bin/mpif90 prog.f90

You can also add other compilation options, as appropriate.

For information on MVAPICH2, please see http://mvapich.cse.ohio-state.edu/

5. OpenMPI with PGI compilers:

OpenMPI version 1.3.3, compiled with PGI 7.2, is available in /usr/local/openmpi-1.3.3_pgi72_ib and the executables (mpicc, mpiCC, mpif90, mpirun, etc) are installed in /usr/local/openmpi-1.3.3_pgi72_ib/bin and this directory is not in users' default path. To compile a code and link to this version of OpenMPI, please use the full path. For example, to compile a Fortran90 code, use

/usr/local/openmpi-1.3.3_pgi72_ib/bin/mpif90 prog.f90

You can also add other compilation options, as appropriate.

For information on OpenMPI, please see http://www.open-mpi.org/

6. OpenMPI with GNU compilers:

OpenMPI version 1.3.3, compiled with GNU compilers (gcc/g++ v. 3.4.6 and gfortran v. 4.1.2), is available in /usr/local/openmpi-1.3.3_gcc346_ib and the executables (mpicc, mpiCC, mpif90, mpirun, etc) are installed in /usr/local/openmpi-1.3.3_gcc346_ib/bin and this directory is not in users' default path. To compile a code and link to this version of OpenMPI, please use the full path. For example, to compile a Fortran90 code, use

/usr/local/openmpi-1.3.3_gcc346_ib/bin/mpif90 prog.f90

You can also add other compilation options, as appropriate.

For information on OpenMPI, please see http://www.open-mpi.org/

Running a batch job on nodes interconnected with Infiniband

In order to run jobs on nodes interconnected with Infiniband (IB) and make use of the IB switch, your code must be linked to an MPI library built for Infiniband and you should submit the job to a batch queue that targets IB nodes. For a description of batch queues that target IB nodes, please see http://rcc.uga.edu/equipment/subpages/rcluster_runningjobs/#Queues (the sample commands below use a queue called mri-ib-q64-30d).

Example of job submission shell scripts (sub.sh)

In the examples below, the executable name will be called myprog

To run a parallel MVAPICH/PGI job (bash shell):

#!/bin/bash
cd working_directory
echo $LSB_HOSTS
cat /dev/null > mlist.$$
for variable in $LSB_HOSTS; do
  echo $variable >> mlist.$$
done
export NC=`wc mlist.$$ | awk '{print($1)}'`
/usr/local/mvapich-1.1_pgi72/bin/mpirun -np ${NC} -hostfile mlist.$$ ./myprog
rm -f mlist.$$

Note: the line that starts with /usr/local wraps around and ends in ./myprog

To run a parallel MVAPICH2/PGI job (bash shell):

#!/bin/bash
cd working_directory
echo $LSB_HOSTS
cat /dev/null > mlist.$$
for variable in $LSB_HOSTS; do
  echo $variable >> mlist.$$
done
export NC=`wc mlist.$$ | awk '{print($1)}'`
/usr/local/mvapich2-1.2_pgi72/bin/mpirun_rsh -np ${NC} -hostfile mlist.$$ LD_LIBRARY_PATH=/usr/local/mvapich2-1.2_pgi72/lib:${LD_LIBRARY_PATH} ./myprog
rm -f mlist.$$

Note that in the sample script above (MVAPICH2) we use mpirun_rsh instead of mpirun or mpiexec. Also, the line that starts with /usr/local wraps around twice and ends in ./myprog

To run a parallel OpenMPI/PGI job (bash shell):

#!/bin/bash
cd working_directory
echo $LSB_HOSTS
cat /dev/null > mlist.$$
for variable in $LSB_HOSTS; do
  echo $variable >> mlist.$$
done
export NC=`wc mlist.$$ | awk '{print($1)}'`
/usr/local/openmpi-1.3.3_pgi72_ib/bin/mpirun -np ${NC} -hostfile mlist.$$ ./myprog
rm -f mlist.$$

Note: the line that starts with /usr/local wraps around and ends in ./myprog

To submit your job to the IB queue:

For example, submit the job to the queue mri-ib-q64-30d, using 32 cores

chmod +x sub.sh

bsub -n 32 -q mri-ib-q64-30d -o out.%J -e err.%J ./sub.sh

For more information on how to submit batch jobs to the queue, please see http://rcc.uga.edu/equipment/subpages/rcluster_runningjobs/.

Request Consultation

Please initiate requests for help or consultation via our:

request support

Feel free to email or call us (we're in the UGA directory). The quickest way to get support is to use the RCC Support form.

For a complete list of all applications on RCC service, please check the complete list all applications page.