Getting Started on eliza

On the eliza cluster, all users should log in to the head node for interactive use (editing, compiling, and serial or parallel MPI debugging). This is also the best place to submit to the PBS queue:

All users:
% ssh -l your-account-name eliza.engin.umich.edu

If you don't have ssh (Secure Shell) on your workstation, visit our SSH page to learn how to download and install it.

Interactive mode is to be used for debugging only. All serial and parallel jobs need to be submitted via batch. Users who attempt to circumvent the batch system will lose access to the clusters. By default, there is a CPU time limit of 5 minutes on eliza for all processes run in interactive mode.

A quick note about storage. There are no user quotas on disk use, but please archive off any excess amounts of data to TSM or ADSM. A good rule of thumb -- if you're storing more than 1 GB, please move the rest to TSM. You can check how much space you're using by typing the command du -sh ~

Writing Code

You can write your MPI code the same way as on the other parallel machines.

The same holds for the Software Optimization Guide for the Athlon64 and AMD Opteron Processor.

Compiling, Running, and Debugging

Depending on whether your code is serial or parallel will determine how you compile and run your code. Also, we have a test myrinet switch on eliza, so you can also build your code to run over myrinet, which has lower latency and larger bandwidth than the Force10.

Serial

Compiling

To compile a serial code, you can use the GNU C and C++ compilers, or the IBM xlf compilers. Examples of which are:

% gcc file.c -o program
% xlf file.f -o program

Parallel on Force10

Compiling

To compile and run MPI programs using the GNU and xlf compilers, you should compile using the LAM-MPI scripts (mpicc, mpiCC, mpif77, mpif90). Examples of which are:

% mpicc file.c -o program
% mpiCC file.C -o program
% mpif77 file.f -o program
% mpif90 file.f90 -o program

For best results on eliza, we recommend using these compiler flags:

-fastsse -pc 64 -Mvect=sse -Mcache_align -Kieee -Mvect=prefetch

Running Interactively with MPI

To run MPI executables interactively, simply invoke:

% lamboot
% mpidebug -np 4 ./your-program-name
% lamhalt

Remember that you should use no more than 4 processes, and run for no more than 5 minutes. Also, remember to invoke "lamhalt", or your MPI processes will continue to run.

Parallel over Myrinet

Compiling

To compile and run MPI programs using the GNU and xlf compilers, you should compile using the mpich-gm scripts (mpicc, mpiCC, mpif77, mpif90), found in /opt/mpi-gm/bin. These are not in your path, so you need to specify the full path. Examples of which are:

% /opt/mpi-gm/bin/mpicc file.c -o program
% /opt/mpi-gm/bin/mpiCC file.C -o program
% /opt/mpi-gm/bin/mpif77 file.f -o program
% /opt/mpi-gm/bin/mpif90 file.f90 -o program

For best results on eliza, we recommend using these compiler flags:

-fastsse -pc 64 -Mvect=sse -Mcache_align -Kieee -Mvect=prefetch

Running Interactively with MPI

To run MPI executables, simply invoke:

% /opt/gm-mpi/bin/mpirun -np 2 ./program

Remember that you should use no more than 2 processes when testing on eliza, and run for no more than 5 minutes. Myrinet does not allow as many connections per machine as is allowed with ethernet.

Submitting Batch Jobs

Our Eliza PBS page has more on how to submit batch jobs and provides some example scripts. Submitting Batch Jobs

% qsub your-scriptname

When running a MPI or PVM task in PBS, the number of nodes will be set by your batch script's line '#PBS -l nodes=##'. Note that a 'node' is a physical dual processor machine. Thus if you ask for two nodes, you will get 4 processors. There is no way to request only 3 processors, unless you can chainsaw a machine in half. :-} See Our Eliza PBS Page for example batch scripts.

More Information

Here are three (Powerpoint) CAC presentations that will help you make the most of our Apple Cluster:

Introduction to the MPI Programming
Performance Tuning on the Apple G5



About the CAC Academics Getting Started Getting Help Resources Hardware Getting Started Using PBS Software Current Outages No outages have been reported Planned Outages No outages have been scheduled Search The CAC	Getting Started on eliza On the eliza cluster, all users should log in to the head node for interactive use (editing, compiling, and serial or parallel MPI debugging). This is also the best place to submit to the PBS queue: All users: % ssh -l your-account-name eliza.engin.umich.edu If you don't have ssh (Secure Shell) on your workstation, visit our SSH page to learn how to download and install it. Interactive mode is to be used for debugging only. All serial and parallel jobs need to be submitted via batch. Users who attempt to circumvent the batch system will lose access to the clusters. By default, there is a CPU time limit of 5 minutes on eliza for all processes run in interactive mode. A quick note about storage. There are no user quotas on disk use, but please archive off any excess amounts of data to TSM or ADSM. A good rule of thumb -- if you're storing more than 1 GB, please move the rest to TSM. You can check how much space you're using by typing the command `du -sh ~` Writing Code You can write your MPI code the same way as on the other parallel machines. The same holds for the Software Optimization Guide for the Athlon64 and AMD Opteron Processor. Compiling, Running, and Debugging Depending on whether your code is serial or parallel will determine how you compile and run your code. Also, we have a test myrinet switch on eliza, so you can also build your code to run over myrinet, which has lower latency and larger bandwidth than the Force10. Serial Compiling To compile a serial code, you can use the GNU C and C++ compilers, or the IBM xlf compilers. Examples of which are: % gcc file.c -o program % xlf file.f -o program Parallel on Force10 Compiling To compile and run MPI programs using the GNU and xlf compilers, you should compile using the LAM-MPI scripts (mpicc, mpiCC, mpif77, mpif90). Examples of which are: % mpicc file.c -o program % mpiCC file.C -o program % mpif77 file.f -o program % mpif90 file.f90 -o program For best results on eliza, we recommend using these compiler flags: -fastsse -pc 64 -Mvect=sse -Mcache_align -Kieee -Mvect=prefetch Running Interactively with MPI To run MPI executables interactively, simply invoke: % lamboot % mpidebug -np 4 ./your-program-name % lamhalt Remember that you should use no more than 4 processes, and run for no more than 5 minutes. Also, remember to invoke "lamhalt", or your MPI processes will continue to run. Parallel over Myrinet Compiling To compile and run MPI programs using the GNU and xlf compilers, you should compile using the mpich-gm scripts (mpicc, mpiCC, mpif77, mpif90), found in /opt/mpi-gm/bin. These are not in your path, so you need to specify the full path. Examples of which are: % /opt/mpi-gm/bin/mpicc file.c -o program % /opt/mpi-gm/bin/mpiCC file.C -o program % /opt/mpi-gm/bin/mpif77 file.f -o program % /opt/mpi-gm/bin/mpif90 file.f90 -o program For best results on eliza, we recommend using these compiler flags: -fastsse -pc 64 -Mvect=sse -Mcache_align -Kieee -Mvect=prefetch Running Interactively with MPI To run MPI executables, simply invoke: % /opt/gm-mpi/bin/mpirun -np 2 ./program Remember that you should use no more than 2 processes when testing on eliza, and run for no more than 5 minutes. Myrinet does not allow as many connections per machine as is allowed with ethernet. Submitting Batch Jobs Our Eliza PBS page has more on how to submit batch jobs and provides some example scripts. Submitting Batch Jobs % qsub your-scriptname When running a MPI or PVM task in PBS, the number of nodes will be set by your batch script's line '#PBS -l nodes=##'. Note that a 'node' is a physical dual processor machine. Thus if you ask for two nodes, you will get 4 processors. There is no way to request only 3 processors, unless you can chainsaw a machine in half. :-} See Our Eliza PBS Page for example batch scripts. More Information Here are three (Powerpoint) CAC presentations that will help you make the most of our Apple Cluster: Introduction to the MPI Programming Performance Tuning on the Apple G5