Equipment

Linux Cluster (rcluster)

Running Jobs

• Using the Batch Queues
• Batch Queues on the rcluster
• LSF Usage Information
• Submitting a Batch Job to the Queue
• Checking the Status of Jobs
• Files Created at Job Start
• Canceling/Removing a Job
• Receiving an Email when Job Terminates
• Runchaining Jobs
• Running an Interactive Job

Using the Batch Queues

Jobs of over ten (10) minutes duration must be submitted to the queues, not run on the login node (rcluster.rcc.uga.edu). Processes that use too much CPU or RAM on the headnode may be terminated by RCC staff, or automatically, in order to keep the cluster running properly. Graphical front ends to programs, programming tools, etc. will not be terminated.

The queueing system being used on the rcluster is Platform LSF.

Batch Queues on the rcluster

The rcluster compute nodes have 2 processors each. Some of the nodes have 2 single-core, some have 2 dual-core, and some have 2 quad-core processors, which means these nodes behave as though they had 2, 4, and 8 CPUs each, respectively. Because the rcluster is comprised of both AMD Opterons and Intel quad-core Xeons, code compiled on one type of node might not be optimized to run on the other type. For more information, please refer to Code Compilation on the rcluster.

The queue names specify the type of node to which the job will be submitted to and the maximum runtime of the job. In general, the queue names fall into the following categories:

Queue Names Beginning With	Submit Jobs To	Allowed Users
s	single-core Opterons	all
d	dual-core Opterons	all
r	either single- or dual-core Opterons	all
iob-s	single-core Opterons	IOB members
mri-q32	quad-core Xeons	IOB members not in NSF/MRI grant
mri-q64	quad-core Xeons	participants in NSF/MRI grant
stat-d	dual-core Opterons	Statistics dept

The batch queue can be used for serial jobs (that is, jobs that require only one CPU core) and for parallel jobs. The form of a queue name indicates how many processing cores (CPU cores) it is limited to and the run time limit. For example, the queue r8-24h has a limit of 8 CPU cores and 24 hours of run time per core. However, by default, a job submitted to this queue will only have 1 CPU core assigned to it. To request more cores, please refer to Submitting a Batch Job to the Queue below. To submit a job to the resource, first determine your CPU core number and time requirements. This will determine which queue you need.

Here are more examples of queue names:

r8-24h	Up to 8 CPU cores, maximum run time of 24h, sends job to either single-core or dual-core Opterons.
r8-10d	Up to 8 CPU cores, maximum run time of 10 days, sends job to either single-core or dual-core Opterons.
s8-24h	Up to 8 CPU cores, maximum run time of 24h per core, sends job to single-core Opterons.
d8-24h	Up to 8 CPU cores, maximum run time of 24h per core, sends job to dual-core Opterons.
iob-s16-10d	Up to 16 CPU cores, maximum run time of 10 days, sends job to single-core Opterons. For IOB's associate members only.
iob-s32-10d	Up to 32 CPU cores, maximum run time of 10 days, sends job to single-core Opterons. For IOB's full members only.
mri-q32-30d	Up to 32 CPU cores, maximum run time of 30 days, sends job to quad-core Xeons. For IOB's members that did not participate in the NSF/MRI grant only.
mri-q64-30d	Up to 64 CPU cores, maximum run time of 30 days, sends job to quad-core Xeons. For participants of the NSF/MRI grant only.
stat-d16-10d	Up to 16 CPU cores, maximum run time of 10 days, sends job to dual-core Opterons. For Statistics Dept. members only.

For a list of all valid queue names, please use the command queuenames from a rcluster shell prompt.

We recommend that users checkpoint their codes whenever possible to avoid losing valuable compute time if the system goes down before a job is completed. A long job that can be checkpointed can be run as a sequence of shorter jobs, which can be automatically submitted to the queue as described below in the Runchaining Jobs section. If you cannot fit your job within the established processor and runtime limits, please let us know.

LSF Usage Information

These are the common LSF commands:

bsub	Submit a job to the queue
bkill	Cancel a queued or running job
bhold	Place a queued job on hold
bjobs	Check the status of queued and running jobs
bqueues	List all valid queue names

Submitting a Batch Job to the Queue

The preferred way to submit a batch job to the queue is to use the bsub command to submit a job submission shell script. The syntax of the bsub command is:

bsub -n nprocs -q queuename -o stdout -e stderr ./shellscriptname

Where

Examples

1.To submit a serial job with script sub.sh to the r1-24h batch queue and have the standard output and error go to test.jobid.out and test.jobid.err, respectively, use

chmod u+x sub.sh

bsub -q r1-24h -o test.%J.out -e test.%J.err ./sub.sh

2.To submit a parallel MPI job (which uses 4 CPU cores) with script sub.sh to the r4-24h batch queue and have the standard output and error go to test.jobid.out and test.jobid.err, respectively, use

chmod u+x sub.sh

bsub -n 4 -q r4-24h -o test.%J.out -e test.%J.err ./sub.sh

3.To submit a parallel job using 2 threads with script sub.sh to the r4-24h batch queue

chmod u+x sub.sh

bsub -n 2 -q r4-24h -R "span[hosts=1]" ./sub.sh

The options -o test.%J.out and -e test.%J.err can also be included in this last example (they were left out just for simplicity).

chmod u+x sub.sh

Example of job submission shell scripts (sub.sh)

In the examples below, the executable name will be called myprog and it requires input parameters to be piped in. The input parameters are in a file called myin and the output data will be stored in a file called myout. The working_directory is the path to your working directory (e.g., it could be /home/labname/username/subdir or /scratch/username/subdir )

The file myin in the examples above is only necessary if your program requires standard input data and the file myout is only necessary if you want the standard output data (if any) to be stored in a separate file instead of the standard output file of the batch job (test.jobid.out in the example above). If your program does not require one or both of these files, you have to remove the corresponding piping symbols ( < and/or > ) in the last line of the scripts above.

MORE IMPORTANT NOTES:

1. MPI jobs executed with mpirun have to use the -machinefile option as shown in the examples above, otherwise your mpi job will not use the processors assigned to it by the queueing system. Using the script above for MPI jobs, a file called mlist.xxxxx containing a list of processors assigned to your job will be generated when your job starts running and it will be deleted when your job is done. The processors used for your job will be listed in the stdout.

2.When running threaded applications, please add the bsub option -R "span[hosts=1]" to ensure that all processors assigned to your job (up to 4 when running on dual core machines and up to 2 when running on single core ones) are on the same machine. Without this bsub option, LSF might assign processors on different machines to your job.

Checking the Status of Jobs

Use the bjobs command to check the status of jobs

code>bjobs [-u username] [-l] [jobid]

where username is the user whose jobs you want to check and jobid is the JOBID of a specific job. The -l option gives long output, with detailed information about the job(s).

Example

bjobs -u all	shows all the jobs in the pool
bjobs -u johndoe	shows all jobs for user johndoe
bjobs -l 10407	gives detailed information about the job with JOBID 10407

Files Created at Job Start

If you submit your job with the -o mystdout -e mystderr options, then the files mystdout and mystderr will be created when your job starts running, unless they already exist. In the latter case, the stdout and stderr of the job will be appended to the corresponding files. If you would like to have the jobid number incorporated into the stdout and stderr file names, use the special character %J in these file names.

If the -o and -e options are not specified at job submission, the stdout and stderr of the job will be sent to you by email to your rcluster account and rcluster will automatically forward it to the email address that you listed when you requested your rcluster account (for example, your ugamail or departmental account). The sender of the email is LSF. You might want to check whether your email server flags such messages as spam and filter them out. To ensure that this does not happen, you might want to whitelist messages sent by LSF.

Canceling/Removing a Job

Use the bkill command to cancel/remove a job from the job pool:

bkill [-u username] jobid [jobid]

Example

bkill 10408	cancels your job with JOBID 10408
bkill 10408 10409	cancels your jobs with JOBIDs 10408 and 10409
bkill -u your_user_id 0	cancels all jobs you have in the queue

Receiving an Email when Job Terminates

When you submit a batch job with bsub without the -o and -e options, you will receive the standard output and standard error of the job by email when the job terminates (whether it completes successfully or not). You can add the bsub option -N to have the standard output of the LSF job (not of the application) sent to you when the job terminates. The standard output of the application and the standard error of the job can still be written to files specified by the -o and -e options, respectively.

Example

bsub -n 4 -q r4-24h -o out.%J -e err.%J -N ./sub.sh

The 4 processor job running on the r4-24h queue will write the standard output of the application in the file out.jobid, write the standard error of the job in the file err.jobid, and it will send the standard output of the batch job (exit code, CPU time used, node used, etc) to the user's preferred email address.

Runchaining Jobs

We have found that a common need is to be able to run the same job over and over. For instance when you need to do a large number of iterations, you run so many and write in a data set the information needed to restart the job where it left off. When the job is restarted it reads the restart information and continues where the previous execution left off.

To have one job automatically submit the next one once it finishes, you can add the following lines at the end of your job submission script:

bsub -n nprocs -q queuename -o stdout -e stderr ./next_script_name
exit

Example

sub1.sh

In the examples below we assume that the executable myprog does not require any standard input. The working directory is assumed to be /home/labname/username/subdirectory.

First the script sub1.sh is submitted to the queue. Once it finishes running, it automatically submits script sub2.sh to the queue. This script can in turn submit sub3.sh to the queue when it completes, and so on. For this procedure, the user can prepare a sequence of scripts, which will then be submitted one at a time to the queue and run in sequence. Alternatively, the script sub1.sh can resubmit itself back to the queue once it finishes running. This would create an "infinite loop", a situation that is not recommended. To break the infinite loop, the user can set some termination rules for the job resubmission process.

Example of a termination rule

One way to break out of an infinite job resubmission loop is to have the code generate a file when the program finally "converges" (or when it completes a predetermined number of steps, for example). Let us call this file finalresults.txt. The job submission script sub.sh checks whether the file finalresults.txt exists. If it does not, then the script sub.sh is submitted to the queue again, otherwise the script simply exits and the resubmission chain is terminated.

A simple script sub.sh that accomplishes this is the following:

Running an Interactive Job

We have set aside the following nodes for interactive jobs:

• One dual-processor dual-core Opteron node(4 CPU cores) called inter1.
• One dual-processor quad-core Xeon node (8 CPU cores) called inter2.

These nodes are not part of the queueing system. To access these nodes, first login to rcluster.rcc.uga.edu and from there use ssh to connect to inter1 or to inter2.

For example, to connect to inter1:

rcluster>  ssh inter1

Your prompt on inter1 or inter2 will not read inter1 or inter2, it will read for example compute-2-12, or a similar name.

A single CPU executable (a.out) can be run on inter1 and inter2 as follows:

compute-2-12>  ./a.out

Or run the code using 'nohup' in order to be able to logout without interrupting the running job:

compute-2-12> nohup ./a.out &

To run a parallel MPI (MPICH) job interactively, first you need to create a file (for example, call it host.list) with the word 'inter1' (without the quotes) in it (or 'inter2' if you are running it on inter2 instead), repeated multiple times in a column (the number of times that it is repeated should be equal to the number of CPU cores you want to use for the MPI program). That is, the contents of host.list will be for example

inter1
inter1
inter1
inter1

Put this file (host.list) in your working directory and then run the MPI program a.out as follows (e.g. using 4 CPU cores):

compute-2-12>  mpirun -np 4 -machinefile host.list ./a.out
compute-2-12>  nohup mpirun -np 4 -machinefile host.list ./a.out &

If your code links to MPICH/GNU or OpenMPI, use the corresponding full path to the mpirun command in the lines above.

To run an interactive job that uses MPICH2, you will need to do the following steps:

1. Create a file called .mpd.conf (note the presence of the initial dot in the file name) in the upper level of your home directory (e.g. in /home/groupname/username or /iob_home/groupname/username) with the following content:

MPD_SECRETWORD=somepassword

where 'somepassword' is a password that mpich2 will check. Please do not use your rcluster login password or any other confidential information, because this password is not encripted in the system.

2. Change the permission of the .mpd.conf file by typing the following command at a shell prompt:

chmod 600 ${HOME}/.mpd.conf

These two steps only need to be done once, not for every job that you run.

3. Create a file called e.g host.list in your working directory with the word 'inter1' ( without the quotes) in it (or 'inter2' if you are running it on inter2 instead), repeated multiple times in a column (the number of times that it is repeated should be equal to the number of CPU cores you want to use for the MPI program). That is, the contents of host.list will be for example

inter1
inter1
inter1
inter1

4. Start the MPD daemons by issuing the command

compute-2-12> /usr/local/mpich2-1.1_gcc346/bin/mpdboot -n 1 --file=host.list$$

If your host.list contains more than one host (for example, if you want to run a job using both inter1 and inter2), the number of unique hosts in host.list should be used as the argument for the -n option in this line.

5. Run your program (e.g. a.out) with

compute-2-12> /usr/local/mpich2-1.1_gcc346/bin/mpiexec -machinefile host.list -np 4 ./a.out

6.After your program is done, remove the MPD daemons with

compute-2-12> /usr/local/mpich2-1.1_gcc346/bin/mpdallexit

Note that if you are running an MPICH2/PGI job, replace the MPICH2 installation path in items 4, 5, and 6 above with the MPICH2/PGI path (i.e. /usr/local/mpich2-1.0.8p1/bin/).

Because inter1 has a total of 4 CPU cores, users should not run parallel jobs that use more than 4 CPU cores or threads on it. Similarly, inter2 has a total of 8 CPU cores and parallel jobs on it should not use more than 8 CPU cores or threads.

These nodes should only be used for short jobs (for example, for code compilation and debugging purposes) and for those that cannot be run on the batch queueing system (for example, if the job requires an X windows front-end). The load of the node can be monitored using top or w.