What Is Preemption
CAC resources are available to the members of the University of Michigan, however certain groups have purchased or rented resources through the CAC for their research projects. However, when those resources aren't being actively used, we make some of those resources available to the general user community. While all of our users have a right to use this, the groups who will be best served by this are users with :
- programs with shorter runtimes
- programs that checkpoint themselves - that is, they save their state during their run, so they can be restarted after they have been stopped)
- programs that are serial or use a small number of nodes
The reason for this may be obvious - you want a higher probability your job won't be interrupted (or recovers nicely in the case of checkpointing).
How Preemption Works At The CAC
When you submit your job, there are several policies at work, but in a nutshell, the scheduler will attempt to find a place on the public nodes for the job to run. If none can be found and there are no private jobs to be scheduled on private resources, then public jobs will be scheduled to run on private nodes, using the same criteria of fairshare, queue time, etc among the jobs which are willing to be preempted. If an owner of the private resources submits a job, the public job may be requeued or deleted (depending on the settings when submitted).
How To Use Preemption
Currently, the preemption system is publicly available. You just need to ask for the preempt qos, which is requested like a resources (nodes, walltime, etc.)
#PBS -l nodes=1,walltime=12:00:00,qos=preemptIf you choose not to have your job requeued and would rather have your job terminated, you only need to specify the job is not rerunnable.
#PBS -r n
Preemption Example
Here is an example cluster and how preemption works. This example cluster contains 2 public nodes and 4 private nodes.
Timestep 1: 2 Public jobs are submitted. There are adequate public node available, so the job run there.
| Private | Public | Queued |
| Idle | PublicJob1 | |
| Idle | PublicJob2 | |
| Idle | ||
| Idle |
Timestep 2: 6 Additional public jobs are submitted. Because there are no public nodes available, but there are private nodes available (although not enough), 4 jobs are scheduled on the private nodes.
| Private | Public | Queued |
| PublicJob3 | PublicJob1 | PublicJob7 |
| PublicJob4 | PublicJob2 | PublicJob8 |
| PublicJob5 | ||
| PublicJob6 |
Timestep 3: 2 private jobs are submitted. Because there are no private or public nodes available, 2 public jobs are preempted, put into the queue, and the 2 private jobs are allowed to start.
| Private | Public | Queued |
| PrivateJob1 | PublicJob1 | PublicJob3 |
| PrivateJob2 | PublicJob2 | PublicJob4 |
| PublicJob5 | PublicJob7 | |
| PublicJob6 | PublicJob8 |
Timestep 4: The 2 private jobs complete and the 2 public jobs also complete. PublicJob3 and PublicJob4 move to the public nodes (which is where the scheduler looks first for public jobs). Once filled, the scheduler then looks at the two free private nodes, and schedules PublicJob7 and PublicJob8.
| Private | Public | Queued |
| PublicJob7 | PublicJob3 | |
| PublicJob8 | PublicJob4 | |
| PublicJob5 | ||
| PublicJob6 |