This document includes basic information about running MPI programs through the Java CoG Kit.
The best way to read this document is to use the PDF version and read it with Adobe Acrobat Reader. Please make sure you configure Adobe Acrobat Reader appropriately so you can follow hyperlinks. This is the case if you follow the default installation. Acrobat Reader is available at http://www.adobe.com/products/acrobat/readermain.html. Because the hyperlinks are not available in the printed form of this manual and we support saving our environment we strongly discourage printing this document.
We recommend that you save this manual locally on your machine and use Acrobat Reader. This has the advantage that you do not lose your anchor points while switching back and forth between different hyperlinks. An HTML version of this manual is planed, but not available yet.
We have augmented the document with some comments at places where we found issues. Our intend is to address these issues in a future release. The comments are marked by the icon to do and the name of the person that will work on the removal of the issue.
Please be a team player and support us indirectly by registering with us or reporting your use of the Java CoG Kit. Although this software is free, we still need to justify to our funders the usefulness of the projects. If you want to help us with our efforts please take a few seconds to complete this information. We do not use this information for other purposes. If you have special needs or concerns please contact gregor@mcs.anl.gov. The registration form can filled out in a variety of formats. The online form can be found at
This form is available also as ASCII text at
which you can FAX to
Gregor von Laszewski, Fax: 630 252 1997
The Message Passing Interface (MPI) provides a powerful programming paradigm for high performance computing based on the interchange of messages between processes. An optimized standardized interface is available on almost all high performance computing platforms.
In this guide we will focus on the submission of MPI jobs to a supercomputer or distributed cluster. As an example serves the TeraGrid [1]. However, the examples discussed in this guide are generic enough for you to modify it for your own server environment. We will demonstrate how to submit MPI programs through the Java CoG Kit with the help of the mpirun and the qsub commands. Although we will use the GT2 provider for our submission examples, other providers such as the SSH provider can also be used.
We assume that you have acquired a TeraGrid accounts and that you have a Grid certificate that allows you to submit jobs on the TeraGrid. If you do not know how to do this, please consult with the TeraGrid Web site at http://www.tergrid.org.
We have chosen an extremely simple test program that prints on each processor the rank and the number of requested processors.
For the rest of this guide, we will assume that you have this program on the TeraGrid under the following directory: $HOME/mpitest/helloworld.c
It is important to make sure that you can compile the program on a teragrid frontend node, just to verify that your TeraGrid account is properly configured. As an example configuration we will submit it on the TeraGrid frontend node and run it on four backend nodes.
To test and build an executable for the TeraGrid, please login first to the frontend node. In this document we will use the name “laszewski” as a place-holder for your username. Please change the example appropriately.
Place the MPI helloworld.c program in the directory mpitest. This program can now be compiled with the mpicc compiler.
We recommend that you check out if you can run this program on the TeraGrid by simply using the TeraGrid mpirun program. In our example we execute it on four processors.
You should get the following output:
If for some reason this simple test program does not work, your account may not properly configured and you should consult with a TeraGrid systems administrator. However, if your program works, you can proceed to the next examples. The next examples do not require that you are logged into a TeraGrid node. Instead, you can execute them from the clients where you have installed the Java CoG Kit.
We assume that you will submit the job to the host tg-grid1.uc.teragrid.org while using the Globus Toolkit version 2 submission (GRAM) service. We will be using the mpirun command that is available in a directory under /soft on the TeraGrid. We will place the output from this program in your home directory on the TeraGrid with the name cog-mpirun.out. We will reuse the helloworld program that you have previously compiled on the TeraGrid as discussed in Section 5.
Before you can submit a job you have to first authenticate your self. Do do so, you create a proxy certificate on the client machine that is used as part of the initialization step.
The command for submitting the job is
The meaning of the flags is as follows:
The output for this command looks similar to the following. However, the contents maybe slightly different as we have reformatted the output for this guide.
On the TeraGrid, the file cog-mpirun.out will be created with the following contents.
If this program does not work, make sure that you log once more in on the TeraGrid and verify with
if the location of mpirun is properly obtained.
Previously we showed how to submit an MPI program with the help of mpirun. In certain circumstances you may need more control over the queue parameters or may just submit a non MPI program. In these cases it is useful to use the job submission routine that is part of the batch processing system installed on the machine. At time of writing of this guide, the UC teragrid node used the qsub program from torque. To submit a job through qsub you need to make sure that you create a script that is submitted as part of the submission process. We assume that you have placed the script (qsub-script) in your home directory on the TeraGrid (/home/laszewski/qsub-script).
The meaning of the flags has been explained in the previous section. They just have different values.
The output for this command looks similar to the following. However, the contents maybe slightly different as we have reformatted the output for this guide.
This example assumes you have the script ”qsub-script” on the remote machine. The contents of the file is as follows.
However, in order to make this script work, you must adapt the account information appropriately (we use here TG-ABC).
Once this program is run, the output in form of the standard output and standard error are written into the files “helloworld-pbs.out” and “helloworld-pbs.err”. The contents of the file helloworld-pbs.out will look similar to the one listed below.
For more details on PBS scripts please take a look at the TeraGrid help pages for PBS scripts on: http://www.teragrid.org/userinfo/guide_jobs_pbs.html
In this section we discuss MPI job submissions on the TeraGrid from the Java CoG Kit using the Globus Toolkit’s MPICH-G2 libraries. We assume that the user is familiar with formulating MPICH-G2 tasks as well as the TeraGrid MPICH-G2 environment. A good introductory background on basic MPICH-G2 concepts is available at http://www3.niu.edu/mpi/. Information on establishing the MPICH-G2 environment on TeraGrid is available at http://www.teragrid.org/userinfo/guide_jobs_mpich_g2.html.
For this Section, we assume the TeraGrid UC/ANL site. Further, we assume the following “soft” environment on the ANL machines.
For our example, we will use the same helloworld.c program described in Section 4. Place the MPI helloworld.c program in the directory mpichg2. This program can now be compiled with the MPICH-G2 based mpicc compiler.
Once the compilation is successful, the server-side environment setup is completed. Next, we submit an MPICH-G2 task from the client machine which has the Java CoG Kit installed. For our example, we will formulate the following RSL:
The above RSL is formulated based on the information available at http://www.teragrid.org/userinfo/guide_jobs_mpich_g2.html and our TeraGrid environment. The various parameters should be appropriately adjusted to reflect your TeraGrid server environment. Additionally, replace the “(project=your project number)” with the appropriate TeraGrid project number.
Please Note that at the time of writing this guide, it was required to include ∕soft∕intel-c- 8.0.066 -f - 8.0.046∕lib∕ directory in the “LD_LIBRARY_PATH” for this example to work. This was due to an MPICH-G2 installation problem at the ANL/UC site, which might have been fixed now.
Next, we create a proxy certificate on the client machine.
The command for submitting the job using the Java CoG Kit is as follows:
The output for this command looks similar to the following. However, the contents maybe slightly different as we have reformatted the output for this guide.
On the TeraGrid, the file helloworld.out will be created with the following contents.
[1] “TeraGrid,” Web Page, 2001. [Online]. Available: http://www.teragrid.org/
[2] G. von Laszewski, I. Foster, J. Gawor, and P. Lane, “A Java Commodity Grid Kit,” Concurrency and Computation: Practice and Experience, vol. 13, no. 8-9, pp. 643–662, 2001. [Online]. Available: http://www.mcs.anl.gov/~gregor/papers/vonLaszewski--cog-cpe-final.pdf
[3] “Java CoG Kit Wiki,” 2004. [Online]. Available: http://www.cogkit.org/wiki
[4] “Java CoG Kit Registration,” 2004. [Online]. Available: http://www.cogkit.org/register
Additional publications about the Java CoG Kit can be found as part of the vita of Gregor von Laszewski http://www-unix.mcs.anl.gov/~laszewsk/vita.pdf. Most documents are avalable online if you follow the links. In future we intend to provide this information without Gregors vita data.
If you need to cite the Java CoG Kit, please use [2].
The Java CoG Kit documentation is distributed as a series of guides. These guides for version 4.1.3 include:
| Short Title | Audience | Description | Format |
| Install | All | A guide to the different ways of installing the Java CoG Kit | [PDF] [HTML] |
| Commands | User | A guide to the command line tools of the Java CoG Kit | [PDF] [HTML] |
| Workflow | User | A guide to the Karajan Workflow | [PDF] [HTML] |
| Abstractions | User | A guide to the Java CoG Kit abstractions API | [PDF] [HTML] |
| MPI | User | A guide to execute MPI programs on the TeraGrid | [PDF] [HTML] |
| Coding | Developer | A guide to the Coding rules for the Java CoG Kit | [PDF] [HTML] |
More guides are under development. The following guides are not yet completed, but are listed here to help us improving these guides. Please, explore them and send us e-mail about improvement suggestions. If you like to contribute a guide yourself, please contact gregor@mcs.anl.gov.
Before downloading the Java CoG Kit, users should read the “Guide to Installing the Java CoG Kit” [PDF] [HTML]. We hope that you will find this guide useful to decide which bundles you need. For the more experienced user, we provide the following index.
The newest version of this document can be downloaded by the developers from
It is not allowed to reproduce this document or the source. This documentation is copyrighted and is not distributed under the CoG Kit license.
We use Bugzilla for tracking bugs and for enhancement suggestions. It is located in the bugzilla.globus.org, but you may find it easier to use one of the following quick links:
The Java CoG Kit project has been initiated and is managed by Gregor von Laszewski. To contact him, please use the information below.
Gregor von Laszewski
Argonne National Laboratory
Mathematics and Computer Science Division
9700 South Cass Avenue
Argonne, IL 60439
Phone: (630) 252 0472
Fax: (630) 252 1997
gregor@mcs.anl.gov
Contact, 13
Registration, 2