Getting Started on nyx

Overview

Like all of the CAC clusters, nyx is a batch computing environment, wherein you create a job that you wish to run (using your code or a prebuilt application), submit that job to the PBS system to be scheduled to run at a later time, and get your results upon completion. We have a login node nyx-login for development and testing, nyx which handles scheduling of jobs, and the nyx compute nodes, which actually do the work.

Running on the nyx cluster happens in one of two ways. To develop & debug your code, you can compile using the ethernet based compilers (either PGI or gnu for serial codes and OpenMPI-based compilers for parallel code). Only run short small jobs on the login node, as it is shared by many users.

To run your job, you'll want to write a PBS script and submit it to the system to be run.

New and Noteworthy

The new system software for nyx has several differences from the previous system you need to be aware of to run effectively.

New login node: nyx-login is now the machine everyone should log into for compiling, testing and submitting. This machine will look like every other compute node on the system, in terms of system setup. This will help minimize bad code from impacting the overall system. Please use common sense and courtesy when testing on nyx-login - it is a shared resource.
Dedicated testing nodes: we have a few nodes set aside for testing PBS scripts and basic code testing. Use the route queue and set your walltime of your job to 15 minutes or less.
System Software: nyx is now running Red Hat Enterprise Linux 4, which has updated GNU compilers and glibc versions. We've also upgraded the PGI compilers to the latest version (6.1).
PBS: Jobs are now allocated by processor, not by node. Because of this, you will also need to specify how much memory you require (250M is the default). If you have a serial job, just list the walltime and memory. If you are running parallel jobs, make your request like: -l nodes=4:ppn=2,pmem=1000mb (for 4 nodes w/ 2 processors, each process using 1GB of memory. All of our nodes have 2 processors).
Modules: We're now using the Modules package to control which version of applications you wish to run and get info on (PATH, MANPATH, etc). The defaults are to use PGI compilers and OpenMPI. Important: to pass your environment (library and path variables are 2 examples) to PBS, you must submit your jobs like: qsub -V your_job_name
AFS: You can now move files to/from your AFS filespace. The location of home directories on nyx is still to be determined, but local filesystems are only to be used for temporary storage. ITD offers 1G of space for users by default and you can currently purchase up to 10G. See CAEN's AFS and ITD's IFS pages for more information.
We have one node that has 8 CPUs and 64GB of RAM. For more information see Using the "large memory" nyx nodes

Compiling

To compile using the default setup -- the Portland Group compilers (PGI), running in batch (PBS), simply use the default MPI compile scripts: mpicc, mpif77, and mpif90.

However, if you want to do something unusual, such as any of the following, you will need to consider the following four topics:

Use the GNU compilers (instead of the PGI compilers) to compile your MPI code.
Compile your MPI code without using the OpenMPI compile scripts
Compile non-MPI code,

Consideration one (which compiler suite), we strongly recommend that you use the PGI compilers for both batch and interactive use. We prefer the PGI compilers for three reasons: 1) the PGI compilers are generally stronger than gcc/g77, 2) they include a f90 compiler; they all support OpenMP, and they have an autoparallelizer (for shared memory), and most importantly, 3) the CAC doesn't have the resources to support both PGI and GNU versions of all the MPI and parallel libraries. Therefore, if you want to use the GNU compilers, you're basically on your own.

NOTE: If at any time you are uncertain as to which version of MPI your executable uses, you can use the following utility to probe your executable:

% which mpicc

whieh will give you the path - you can look for PGI or gcc.

Consideration two (using the OpenMPI compilation scripts), we strongly recommend using the OpenMPI compilation scripts (mpicc/mpiCC/mpif77/mpif90) since they're easier to use (and for us to support) than compiling and linking by explicitly specifying all the include file paths and library locations. For those troublemakers who insist on compiling explicitly, see our explicit compilation web page.

Consideration three (compiling non-MPI code), you will need to specify your compiler and libraries explicitly, as in:

% pgcc file.c -o program
% pgCC file.C -o program
% pgf77 file.f -o program
% pgf90 file.f90 -o program

FYI, the installation directory for the PGI compilers is /home/software/rhel4/pgi. Look there for compiler-specific include files and libraries.

NOTE: Your default PATH is configured to allow you to use the PGI compilers and run in batch using PBS. To use these tools, you DO NOT need to change your PATH or set any environment variables at all. Just proceed to compile and run your code via PBS.

NOTE: You must recompile your program if you move your code between clusters.

For maximum optimization on opterons, we recommend using some or all of these compiler optimization flags:

-fastsse -O3

Remember, however, that if your code executes incorrectly, the first thing to do in debugging the problem is to recompile without any of these optimization flags and instead use only "-O0 -g -Mbounds".

Running the PGI Debugger (pgdbg) with OpenMPI

At this time (7/2006), PGI doesn't support OpenMPI with their debugger, although they plan to later this year, and the OpenMPI people are now aware of this. That said, it still may work for you. Here are the steps to try it:

You must use version 1.1 of OpenMPI, so type:
module swap openmpi openmpi/1.1.0-pgi
You now need to (re)compile your code with this version of OpenMPI and the debugging flags (-g).
If you can do your debugging on two CPUs, you can use the head node, if you can't, you should start an interactive PBS session, like:
qsub -I -V -q queue -l nodes=2:ppn=2,walltime=1:00:00
where queue is the name of the queue you normally use.
Now you can launch the debugger for a 4-CPU job, like:
mpirun --debugger "pgdbg @mpirun@ @mpirun_args@" --debug -np 4 ./cpi
where, in my case, the executable is called "cpi". The first thing you'll see is:
Loaded: /home/software/rhel4/openmpi-1.1a9/pgi/bin/orterun pgdbg>
At this point type run, even though "orterun" isn't your application. You'll then see all of the OpenMPI mca's load and unload. If you are prompted to add a host to your list of known hosts during this, type "yes" (pgdbg uses ssh to launch its debugger processes). You'll then see a whole slew of messages that say "db_set_code_brk : DiBreakpointSet fails" - I ignore these, but I'm not sure what they mean - this is what the PGI guys are going to fix later, I guess. After a while, this will stop and you'll be at the first command after MPI_Init in your code. In my example, this looks like:
[0] Stopped at 0x402929, function main, file cpi.c, line 21 #21: MPI_Comm_size(MPI_COMM_WORLD,&numprocs); pgdbg>
You can then type, for example:
pgdbg [all] 0> procs ID IPID STATE THREADS HOST => 0 11634 Stopped 1 nyx307.engin.umich.edu 1 11635 Stopped 1 nyx307.engin.umich.edu 2 24176 Stopped 1 nyx296.engin.umich.edu 3 24177 Stopped 1 nyx296.engin.umich.edu pgdbg [all] 0> proc 2 [2] Stopped at 0x402929, function main, file cpi.c, line 21 #21: MPI_Comm_size(MPI_COMM_WORLD,&numprocs); ([2] Process Stopped) pgdbg [all] 2> procs ID IPID STATE THREADS HOST 0 11634 Stopped 1 nyx307.engin.umich.edu 1 11635 Stopped 1 nyx307.engin.umich.edu => 2 24176 Stopped 1 nyx296.engin.umich.edu 3 24177 Stopped 1 nyx296.engin.umich.edu
and set break points and use "step" and "next" and "cont" just like
normal, but you'll still continue to get a whole slew of "db_set_code_brk : DiBreakpointSet fails" messages that you'll just have to ignore for now. I hope that helps a little, and we'll keep you up to date as we hear more from PGI.

Submitting Batch Jobs
When submitting batch jobs, you do not need to use the "interactive" or "batch" scripts. Simply create a PBS batch script, and then submit it to the queues using the PBS command qsub.
The nyx PBS page shows how to submit batch jobs and provides some example scripts. Submitting a batch script is easy, just type:
% qsub -V YourScriptName



About the CAC Academics Getting Started Getting Help Resources Hardware Getting Started Using PBS Software Current Outages No outages have been reported Planned Outages Aon – 07/26 Search The CAC	Getting Started on nyx Overview Like all of the CAC clusters, nyx is a batch computing environment, wherein you create a job that you wish to run (using your code or a prebuilt application), submit that job to the PBS system to be scheduled to run at a later time, and get your results upon completion. We have a login node nyx-login for development and testing, nyx which handles scheduling of jobs, and the nyx compute nodes, which actually do the work. Running on the nyx cluster happens in one of two ways. To develop & debug your code, you can compile using the ethernet based compilers (either PGI or gnu for serial codes and OpenMPI-based compilers for parallel code). Only run short small jobs on the login node, as it is shared by many users. To run your job, you'll want to write a PBS script and submit it to the system to be run. New and Noteworthy The new system software for nyx has several differences from the previous system you need to be aware of to run effectively. New login node: nyx-login is now the machine everyone should log into for compiling, testing and submitting. This machine will look like every other compute node on the system, in terms of system setup. This will help minimize bad code from impacting the overall system. Please use common sense and courtesy when testing on nyx-login - it is a shared resource. Dedicated testing nodes: we have a few nodes set aside for testing PBS scripts and basic code testing. Use the route queue and set your walltime of your job to 15 minutes or less. System Software: nyx is now running Red Hat Enterprise Linux 4, which has updated GNU compilers and glibc versions. We've also upgraded the PGI compilers to the latest version (6.1). PBS: Jobs are now allocated by processor, not by node. Because of this, you will also need to specify how much memory you require (250M is the default). If you have a serial job, just list the walltime and memory. If you are running parallel jobs, make your request like: -l nodes=4:ppn=2,pmem=1000mb (for 4 nodes w/ 2 processors, each process using 1GB of memory. All of our nodes have 2 processors). Modules: We're now using the Modules package to control which version of applications you wish to run and get info on (PATH, MANPATH, etc). The defaults are to use PGI compilers and OpenMPI. Important: to pass your environment (library and path variables are 2 examples) to PBS, you must submit your jobs like: qsub -V your_job_name AFS: You can now move files to/from your AFS filespace. The location of home directories on nyx is still to be determined, but local filesystems are only to be used for temporary storage. ITD offers 1G of space for users by default and you can currently purchase up to 10G. See CAEN's AFS and ITD's IFS pages for more information. We have one node that has 8 CPUs and 64GB of RAM. For more information see Using the "large memory" nyx nodes Compiling To compile using the default setup -- the Portland Group compilers (PGI), running in batch (PBS), simply use the default MPI compile scripts: mpicc, mpif77, and mpif90. However, if you want to do something unusual, such as any of the following, you will need to consider the following four topics: Use the GNU compilers (instead of the PGI compilers) to compile your MPI code. Compile your MPI code without using the OpenMPI compile scripts Compile non-MPI code, Consideration one (which compiler suite), we strongly recommend that you use the PGI compilers for both batch and interactive use. We prefer the PGI compilers for three reasons: 1) the PGI compilers are generally stronger than gcc/g77, 2) they include a f90 compiler; they all support OpenMP, and they have an autoparallelizer (for shared memory), and most importantly, 3) the CAC doesn't have the resources to support both PGI and GNU versions of all the MPI and parallel libraries. Therefore, if you want to use the GNU compilers, you're basically on your own. NOTE: If at any time you are uncertain as to which version of MPI your executable uses, you can use the following utility to probe your executable: % which mpicc whieh will give you the path - you can look for PGI or gcc. Consideration two (using the OpenMPI compilation scripts), we strongly recommend using the OpenMPI compilation scripts (mpicc/mpiCC/mpif77/mpif90) since they're easier to use (and for us to support) than compiling and linking by explicitly specifying all the include file paths and library locations. For those troublemakers who insist on compiling explicitly, see our explicit compilation web page. Consideration three (compiling non-MPI code), you will need to specify your compiler and libraries explicitly, as in: % pgcc file.c -o program % pgCC file.C -o program % pgf77 file.f -o program % pgf90 file.f90 -o program FYI, the installation directory for the PGI compilers is /home/software/rhel4/pgi. Look there for compiler-specific include files and libraries. NOTE: Your default PATH is configured to allow you to use the PGI compilers and run in batch using PBS. To use these tools, you DO NOT need to change your PATH or set any environment variables at all. Just proceed to compile and run your code via PBS. NOTE: You must recompile your program if you move your code between clusters. For maximum optimization on opterons, we recommend using some or all of these compiler optimization flags: -fastsse -O3 Remember, however, that if your code executes incorrectly, the first thing to do in debugging the problem is to recompile without any of these optimization flags and instead use only "-O0 -g -Mbounds". Running the PGI Debugger (pgdbg) with OpenMPI At this time (7/2006), PGI doesn't support OpenMPI with their debugger, although they plan to later this year, and the OpenMPI people are now aware of this. That said, it still may work for you. Here are the steps to try it: You must use version 1.1 of OpenMPI, so type: `module swap openmpi openmpi/1.1.0-pgi` You now need to (re)compile your code with this version of OpenMPI and the debugging flags (`-g`). If you can do your debugging on two CPUs, you can use the head node, if you can't, you should start an interactive PBS session, like: `qsub -I -V -q queue -l nodes=2:ppn=2,walltime=1:00:00` where queue is the name of the queue you normally use. Now you can launch the debugger for a 4-CPU job, like: `mpirun --debugger "pgdbg @mpirun@ @mpirun_args@" --debug -np 4 ./cpi` where, in my case, the executable is called "cpi". The first thing you'll see is: `Loaded: /home/software/rhel4/openmpi-1.1a9/pgi/bin/orterun pgdbg>` At this point type `run`, even though "orterun" isn't your application. You'll then see all of the OpenMPI mca's load and unload. If you are prompted to add a host to your list of known hosts during this, type "yes" (pgdbg uses ssh to launch its debugger processes). You'll then see a whole slew of messages that say "`db_set_code_brk : DiBreakpointSet fails`" - I ignore these, but I'm not sure what they mean - this is what the PGI guys are going to fix later, I guess. After a while, this will stop and you'll be at the first command after MPI_Init in your code. In my example, this looks like: `[0] Stopped at 0x402929, function main, file cpi.c, line 21 #21: MPI_Comm_size(MPI_COMM_WORLD,&numprocs); pgdbg>` You can then type, for example: pgdbg [all] 0> procs ID IPID STATE THREADS HOST => 0 11634 Stopped 1 nyx307.engin.umich.edu 1 11635 Stopped 1 nyx307.engin.umich.edu 2 24176 Stopped 1 nyx296.engin.umich.edu 3 24177 Stopped 1 nyx296.engin.umich.edu pgdbg [all] 0> proc 2 [2] Stopped at 0x402929, function main, file cpi.c, line 21 #21: MPI_Comm_size(MPI_COMM_WORLD,&numprocs); ([2] Process Stopped) pgdbg [all] 2> procs ID IPID STATE THREADS HOST 0 11634 Stopped 1 nyx307.engin.umich.edu 1 11635 Stopped 1 nyx307.engin.umich.edu => 2 24176 Stopped 1 nyx296.engin.umich.edu 3 24177 Stopped 1 nyx296.engin.umich.edu and set break points and use "step" and "next" and "cont" just like normal, but you'll still continue to get a whole slew of "db_set_code_brk : DiBreakpointSet fails" messages that you'll just have to ignore for now. I hope that helps a little, and we'll keep you up to date as we hear more from PGI. Submitting Batch Jobs When submitting batch jobs, you do not need to use the "interactive" or "batch" scripts. Simply create a PBS batch script, and then submit it to the queues using the PBS command qsub. The nyx PBS page shows how to submit batch jobs and provides some example scripts. Submitting a batch script is easy, just type: % qsub -V YourScriptName