Using the "large memory" nyx nodes

Compiling

If you are compiling your own code for use on this system using the PGI compilers on nyx (these are the default) you should include the option:

-mcmodel=medium

on your compile line. This will allow your code to use more than 2GB per thread.

Also, since this system has 8 CPUs, you may also want to try using the PGI Compiler's automatic parallelization flags. To compile your code this way, use the option:

-Mconcur

If you do this, before you run your code you need to set the environment variable OMP_NUM_THREADS to the number of CPUs you want to use. Note that more isn't always better, you should test your code with different numbers to find the optimal number of CPUs for you.

For example, to compile and run a program on this system, you might do something like this:

pgf77 -o myProg -Mconcur -mcmodel=medium -fastsse -O3 myProg.f
export OMP_NUM_THREADS=4
./myProg

Note that, unless you are using an interactive PBS session, you shouldn't do these last two steps on the command line, but in your PBS script.

Submitting Batch Jobs

To use this system via PBS, you should submit your jobs to the bigmem queue (if you have a reservation and need more time that bigmem allows, you should submit your jobs to bigmem_r).

Contining the example from above, after you compile your program, you should create a PBS script for it, we will call ours myPbs. It should look something like:

#!/bin/sh
#PBS -N bigmemjob
#PBS -l nodes=1:ppn=8,mem=16gb,walltime=48:00:00
#PBS -M your_uniqname@umich.edu
#PBS -m abe
# the queue should be bigmem_r if you have a reservation
#PBS -q bigmem
#PBS -V

echo "I ran on $HOSTNAME"

# the next line is only required if you using multiple CPUs
export OMP_NUM_THREADS=4
cd /path/to/my/executable
./myProg

If you don't need all of the CPUs or memory on the system, you should alter the nodes=1:ppn=8 line. You can set it to just nodes=1 and you'll be able to share the system with other people. However, if you do need all of the memory, even for a single-threaded job, you should request all 8 CPUs (as shown in the example script). You can also alter the memory requirement so that if you are sharing the node with someone else, the resource manager will insure that a job requesting more memory doesn't start.

Some programs (Abaqus, for example) always write scratch files to disk, regarless of the amount of RAM available. If you can control where these files are written and you don't need all of the RAM in the system for the rest of your problem, you can use the directory /dev/shm to write your scratch files. This directory is actually in memory, not disk, so it is very fast at the expense of using up RAM. Also, if the system is restarted (crashes, is rebooted, etc.), anything written to /dev/shm is lost.