HOWTO check up on jobs

Examining the queue

You can look at the queue by using the qstat command. qstat will display the queue ordered by JobID.

[username@hpc1 ~]$ qstat
Job id            Name             User              Time Use S Queue
----------------  ---------------- ----------------  -------- - -----
32.hpc1           JobName          username          351:04:3 R long
33.hpc1           JobName          username          351:06:1 R day
34.hpc1           JobName          username          390:30:2 R week
40.hpc1           JobName          username          496:38:2 R month
46.hpc1           JobName          username          506:13:5 R long

Checking a specific job

If you want to see the details of a specific job, use qstat -fx <JobID> on it:

[username@hpc1 ~]$ qstat -fx 40

If you want to look at the output of your job while it's still running, use the qpeek command.

[username@hpc1 ~]$ qpeek 40

Deleting a job you no longer want

If you want to delete a job (whether it's already running or not), use the qdel command:

[username@hpc1 ~]$ qdel 41

There's no output on a successful job deletion. Keep in mind that when running jobs are killed, files in scratch space will not sync back to your home directory. Orphaned scratch space will be moved to /orphans.

Overview of cluster usage

pestat gives a nice overview of which nodes are busy with which jobs for which users.

[username@hpc1 ~]$ pestat
Queues:  short day week month long
Node            state    cpu        memory   jobids/users
----                   tot used    tot used
comp001.hpc     busy     8    8    15G  51%  34
comp002.hpc     free    64   60   126G  12%  35 36 37 38

[username@hpc1 ~]$ pestat -u username
Queues:  short day week month long
Node            state    cpu        memory   jobids/users
----                   tot used    tot used
comp001.hpc     busy     8    8    15G  51%  34
comp002.hpc     free    64   60   126G  12%  38

[username@hpc1 ~]$ pestat -a -u username
Queues:  short day week month long
Node            state    cpu        memory   jobids/users
----                   tot used    tot used
comp001.hpc     busy     8    8    15G  51%  34 username
comp002.hpc     free    64   60   126G  12%  38 username