SGE user commands
Understand SGE and queueing system. Submit batch jobs to SGE cluster.
September 17th, 2017. Pierre-Edouard GUERIN
Using a cluster environment is similar to using linux environments for your job submission. The difference is that you need to specify needed resources beforehand. The cluster is controlled by a SGE (Sun Grid Engine Software) that organizes the queues and resources. This sort of scheduling system is necessary when limited computational resources are shared by many. Here I show how to use Sun Grid Engine for job submission, monitoring and troubleshooting.
Sun Grid Engine
Cluster environment such as MBB is similar as unix/linux environments for job submission (e.g. running your scripts or other software). The difference is that you need to specify needed resources beforehand. The cluster is controlled by a SGE (Sun Grid Engine software) that orders the queues and resources. This sort of scheduling system is necessary when limited computational resources are shared by many.
SGE will do the job scheduling. That means you can submit all your jobs and SGE will queue them and run them when resources you requested becomes available. SGE will also achieve load balancing where the jobs will be distributed so that specific nodes do not get overloaded. In addition, SGE will allow you to do job monitoring which will be useful when you want to check if your job is running ,and if it failed it will help you understand what went wrong.
Access and use the cluster
The SSH protocol (also referred to as Secure Shell) is a method for secure remote login from one computer to another. It provides several alternative options for strong authentication, and it protects the communications security and integrity with strong encryption.
To access to a cluster, you need your administrator creates your account first. Then you can acces via SSH.
Job scheduling: command to submit a job
Here an example of bash script
submitBatchJob.sh to submit a batch job via SGE:
# Job name #$ -N myjob_1 # Using current working directory (otherwise, you will have to use '#$ wd /path/to/run') #$ -cwd # job time limits (h_rt is required [s_rt == software time limit / h_rt == hardware time limit]) #$ -l s_rt=999:55:00 #$ -l h_rt=920:00:00 # choose to run on a specific queue #$ -q cemeb20.q # Get a mail when the job begins, ends or is suspended #$ -m ebs #$ -M email@example.com # Redirects the standard output to the named file. #$ -o submitjob_sge_cluster/qsub_outputs/myjob_20190903-110055.out ##$ -e submitjob_sge_cluster/qsub_outputs/myjob_20190903-110055.err # merge standard and error outputs #$ -j yes # choose a parallel environment and run on 60 slots (use $PE_HOSTFILE) # -pe mpi 16 # Export all my environment variables into job runtime context #$ -V ################### job to submit ############################################# bash myjob.sh
The job to run into a Grid Engine queue is the the non-commentary line.
Lines starting with
#$are dedicated to
-Ndefines the name of the job
-qdefines cluster queues which may be used to execute this job.
-llaunches the job in a Grid Engine queue meeting the given resource request list
-mdefines or redefines under which circumstances mail is to be sent. The option arguments have the following meaning:
bMail is sent at the beginning of the job.
eMail is sent at the end of the job.
aMail is sent when the job is aborted.
sMail is sent when the job is suspended.
-Mdefines the list of users to which the server that executes the job has to send mail
-odefines the path used for the standard output stream of the job
-edefines the path used for the standard error stream of the job
-jspecifies yes or no whether or not the standard error stream of the job is merged into the standard output stream.
-peparallel programming environment (PE) to instantiate.
-Vexports all environment variables into job runtime context
To submit the job with these parameters, simply use the
Job monitoring: check the current status of my submitted job
You can see the current status of the available Grid Engine queues and the jobs associated with the queues with
job-ID prior name user state submit/start at queue slots ja-task-ID ----------------------------------------------------------------------------------------------------------------- 2041601 0.53741 myjob_1 peguerin r 09/17/2020 12:08:30 firstname.lastname@example.org 1 2041602 0.54128 myjob_2 peguerin qw 09/17/2017 12:07:12 1
- job-ID: the ID number of the job, useful to kill the job
- prior: priority of the job. The higher a job's priority value, the earlier it gets dispatched.
- name: the name of the job you submitted with
- user: the user name of the owner of the job
- submit/start at: the date the job was submitted
- queue: the name of the queue the job is using (defined with
- slots: the number of slot the job is taking.
- ja-task-ID -- The job array task ID
Specifies a full format display of information:
See jobs owned by the user
qstat | grep $USER
See a specific job status with a given job-ID
qstat -j <job-ID>
To delete Grid Engine job with a given job-ID
<job-ID> from queues:
qdel -j <job-ID>
This command displays a list of available queues in a cluster:
This command shows the status of Grid Engine hosts, queues, jobs
To manage the queues of the system, we use
qconf. Let's print information about the queue named
qconf -sq <queue>
qstat command displays a cluster queue summary:
qstat -g c
CLUSTER QUEUE CQLOAD USED RES AVAIL TOTAL aoACDS cdsuE -------------------------------------------------------------------------------- cemeb.q 0.70 64 0 39 104 8 0 cemeb20.q 1.17 121 0 20 176 40 48 mbb.q 0.00 0 0 40 40 0 0 mem.q 0.08 3 0 37 40 0 0
- CLUSTER QUEUE: the cluster queue name.
- CQLOAD: an average of the normalized load average of all queue hosts.
- USED: the number of currently used slots.
- RES: the number of slots reserved in advance.
- AVAIL: the number of currently available slots.
- TOTAL: the total number of slots.
- aoACDS: the number of slots which is in at least one of the states
a: Load threshold alarm
A: Suspend threshold alarm
C: Suspended by calendar
D: Disabled by calendar
- cdsuE: the number of slots which are in one of these states or in any combination of them:
c: Configuration ambiguous