Java CoG Kit Abstractions Guide

Kaizar Amin and Gregor von Laszewski

Software Version: 4_1_2
Manual version: 03/15/05
Url: \MANUALBASE/abstractions.pdf
Url: http://www.cogkit.org/release/4_1_2/manual/abstractions/abstractions.html
Last update: July 11, 2005

Contents

1 About this Document
 1.1 Reproduction
 1.2 Viewing
 1.3 Format
2 Registration
3 Introduction
4 Installation
 4.1 Examples
5 Design
 5.1 ExecutableObject
 5.2 Task
 5.3 Specification
 5.4 Service
 5.5 TaskGraph
 5.6 Status
 5.7 Handlers
 5.8 FileResource
 5.9 GridFile
 5.10 Permissions
6 Programmer’s Guide
 6.1 How to execute a remote job execution task
 6.2 How to execute a file transfer task
 6.3 How to execute a file operation task
 6.4 How to execute a simple task graph (DAG)
 6.5 How to execute a hierarchical task graph
 6.6 How to checkpoint a task
 6.7 How to execute a checkpointed task
 6.8 How to checkpoint a task graph
 6.9 How to resume a checkpointed task graph
 6.10 How to invoke remote file operations without adhering to the task model
 6.11 How to run a native Condor job using the CoG Kit abstractions
 6.12 How to integrate a new provider module in the Java CoG Kit
Appendix
A Java CoG Kit Guides
B Java CoG Kit Guides Under Construction
C Available Downloads
D Availability of the Document
E Bugs
F Administrative Contact

 

1 About this Document

This document includes basic information about the different Java CoG Kit abstractions and their intended usage.

1.1 Reproduction

The material presented in this document can not be published, mirrored, electronically or otherwise reproduced without prior written consent. As you can link to this document, this should not pose much of a restriction.

1.2 Viewing

The best way to read this document is with Adobe Acrobat Reader. Please make sure you configure Adobe Acrobat Reader appropriately so you can follow hyperlinks. This is the case if you follow the default installation. Acrobat Reader is available at http://www.adobe.com/products/acrobat/readermain.html. Because the hyperlinks are not available in the printed form of this manual and we support saving our environment we strongly discourage printing this document.

We recommend that you save this manual locally on your machine and use Acrobat Reader. This has the advantage that you do not lose your anchor points while switching back and forth between different hyperlinks. An HTML version of this manual is planed, but not available yet.

1.3 Format

We have augmented the document with some comments at places where we found issues. Our intend is to address these issues in a future release. The comments are marked by the icon  to do  and the name of the person that will work on the removal of the issue.

2 Registration

Please be a team player and support us indirectly by registering with us or reporting your use of the Java CoG Kit. Although this software is free, we still need to justify to our funders the usefulness of the projects. If you want to help us with our efforts please take a few seconds to complete this information. We do not use this information for other purposes. If you have special needs or concerns please contact gregor@mcs.anl.gov. The registration form can filled out in a variety of formats. The online form can be found at

http://www.cogkit.org/register

This form is available also as ASCII text at

http://www.cogkit.org/register/form.txt

which you can FAX to

Gregor von Laszewski, Fax: 630 252 1997

3 Introduction

The Java Cog Kit abstractions offers a programming model that supports elementary Grid patterns such as job execution, file transfer, and file operations. It also provides advanced patterns such as execution (control) flows in the form of directed acyclic graphs (DAG).

The abstractions module is based on a Grid abstraction model, de-coupling the Grid patterns from their implementation. The applications developed with the offered Grid abstractions are independent of the low-level Grid implementations. Hence, applications are shielded from changes in the low-level Grid protocols and interfaces.

The abstractions module offers the following benefits to Grid application programmers:

The job execution pattern in cog-abstractions currently supports GT v2.4.3, GT v3.0.2, GT v3.2.0, GT v3.2.1, GT v4.0.0, Condor, and SSH implementations. Other platforms (Condor and Unicore) will be supported in future releases upon request by the community. We like to invite the community to participate in these activities.

The file transfer pattern in cog-bastractions allows file transfers between GridFTP, FTP, WebDAV, and SSH resources.

The file operation pattern in cog-abstractions permits remote file access operations (such as ls, chmod, cd, get, put) on files hosted on GridFTP, FTP, and WebDAV resources.

4 Installation

Cog-abstractions source is available with the Java CoG Kit v4. Instructions for downloading the Java Cog Kit are available in the Java CoG Kit Installation Guide at Url: \MANUALBASE/install.pdf.

We note that cog-abstractions is explicitly a client-side library. The current version of cog-abstractions provides support for GT v2.4.3, GT v3.0.2, GT v3.2.0 GT v3.2.1, GT v4.0.0, Condor, SSH, FTP, and WebDAV. In future, we will provide also support for GT4.

Hence, in order to execute job submission tasks with our abstractions, the reader is directed to install GT v2.4.3, GT v3.0.2, GT v3.2.0, GT v3.2.1, GT v4.0.0, Condor, or an SSH job submission server. Likewise, in order to execute file transfer and file operation tasks, the reader is directed to install the GridFTP, FTP, and WebDAV file servers.

For details on installing the Globus Toolkit please visit the Globus Alliance Web page http://www.globus.org

4.1 Examples

Several examples that demonstrate the ease of use and functionality of the Java CoG Kit abstractions are provided with the Java CoG Kit distribution. These examples are well documented in the Java CoG Kit Example Guide at Url: \MANUALBASE/examples.pdf.

PIC

Figure 1: UML Class Diagram of the Abstractions.

The examples are further divided into the following packages:

execution
examples demonstrating the execution of a job on a remote job execution service (GT v2.4.3, GT v3.0.2, GT v3.2.0, GT v3.2.1, GT v4.0.0, Condor, and SSH).
transfer
examples demonstrating the transfer of files and directories between two file servers (GridFTP, FTP, and WebDAV).
file
examples demonstrating the operations on files hosted on remote file servers (GridFTP, FTP, and WebDAV).
taskgraph
examples demonstrating the execution of hierarchical task graphs and control dependencies between Grid tasks.
queue and set
examples demonstrating the execution of hierarchical queues and sets of tasks respectively.
xml
examples demonstrating the ability to checkpoint partially executed task graphs to an xml document and construct task graphs from checkpointed xml documents.

After successfully compiling the Java CoG Kit, these examples can be executed from the launcher scripts available in cog/dist/cog-<version>/bin/examples directory.

5 Design

One of the most elementary usage pattern in Grid computing is the execution of a Grid task (job submission, file transfer, file operation, or information query). An extension to this basic Grid pattern is a Grid workflow pattern that enables the user to submit a set of Grid tasks along with an execution dependency. Therefore, the initial design of cog-abstractions concentrates on providing the elements required to support these important usage patterns. Other Grid patterns can be supported by extending the flexible cog-abstractions design based on community feedback.

Figure 1 shows the class diagram of cog-abstractions. A detailed listing of the attributes and functions for each class has been omitted for simplicity. In the rest of this section we describe the important entities listed in Figure 1 and outline their semantics as a part of the offered Grid functionality.

5.1 ExecutableObject

An ExecutableObject provides a high-level abstraction for artifacts that can be executed on the Grid. It can be specialized as a Grid Task or a TaskGraph. An ExecutableObject in the Java CoG Kit has a unique identity, name, and an execution status.


  public interface ExecutableObject {
  
      public static final int TASK = 1;
      public static final int TASKGRAPH = 2;
  
      public int getObjectType();
  
      public void setName(String name);
      public String getName();
  
      public void setIdentity(Identity id);
      public Identity getIdentity();
  
      public void setStatus(Status status);
      public void setStatus(int status);
      public Status getStatus();
  
      public void addStatusListener(StatusListener listener);
      public void removeStatusListener(StatusListener listener);
  
  }

5.2 Task

A Task is an atomic unit of execution in cog-abstractions. It represents a generic Grid functionality including remote job execution, file transfer request, file operation, or information query. It has a unique identity, name, execution status, specification, and set of services (for remote execution).

The task identity helps in uniquely representing the task across the Grid. The task specification represents the actual attributes or parameters required for the execution of the Grid-centric task. The generalized specification can be extended for common Grid tasks such as remote job execution, file transfer, and information query. A Grid task also contains a set of Grid services that enable the remote execution of the task. For example, a job submission task contains a remote job execution service that will ultimately enable the execution of the task. Likewise, the file transfer task contains a source file server and a destination file server as its supporting services.

A Grid task contains the abstract elements required for any Grid functionality. However, the actual execution of a Grid task requires a specific abstract-to-protocol mapping that translates all the abstract components into backend protocol-specific entities. This translation is done by the Handlers (see 5.7). A task of a given type (execution, file transfer, or file operation) is submitted to an appropriate task handler (execution task handler, file transfer task handler, or a file operation task handler respectively). Based on the provider attribute of the task, the handler provides the appropriate abstract-to-protocol transformation.


  public interface Task extends ExecutableObject {
      public static final int JOB˘SUBMISSION = 1;
      public static final int FILE˘TRANSFER = 2;
      public static final int INFORMATION˘QUERY = 3;
      public static final int FILE˘OPERATION = 4;
  
      public void setType(int type);
      public int getType();
  
      public void setProvider(String provider);
      public String getProvider();
  
      public void setService(int index, Service service);
      public void addService(Service service);
      public Service removeService(int index);
      public Collection removeAllServices();
      public void removeService(Collection collection);
      public Service getService(int index);
      public Collection getAllServices();
  
      public void setRequiredService(int value);
      public int getRequiredServices();
  
      public void setSpecification(Specification specification);
      public Specification getSpecification();
  
      public void setStdOutput(String output);
      public String getStdOutput();
  
      public void setStdError(String error);
      public String getStdError();
  
      public void setAttribute(String name, Object value);
      public Object getAttribute(String name);
      public Enumeration getAllAttributes();
  
      public void addOutputListener(OutputListener listener);
      public void removeOutputListener(OutputListener listener);
  
      public void toXML(File file) throws MarshalException;
      public String toString();
  
      public boolean isUnsubmitted();
      public boolean isActive();
      public boolean isCompleted();
      public boolean isSuspended();
      public boolean isFailed();
      public boolean isCanceled();
  
      public Calendar getSubmittedTime();
      public Calendar getCompletedTime();
  
  }

5.3 Specification

Every Grid Task has an associated Specification that dictates the objective of the task and the environment required to achieve the objective. The TaskHandler manages the tasks based on the parameters specified in the task specification.


  public interface Specification {
  
      public static final int JOB˘SUBMISSION = 1;
      public static final int FILE˘TRANSFER = 2;
      public static final int INFORMATION˘QUERY = 3;
  
      public void setType(int type);
      public int getType();
  
      public void setSpecification(String specification);
      public String getSpecification();
  }

A task specification is a generalized concept and can be further extended as JobSpecification, FileSpecification, and InformationSpecification1 . We note that the specific parameters required in a task specification depend on the underlying Grid implementation used for the execution of the Task. For example, GT4 has several required parameters that are not supported by GT2 (and vice versa). However, the specification classes in cog-abstractions offer some commonly used attributes that can be extended or omitted based on the requirements of the task and specific Grid implementation.

The JobSpecification mentions the common parameters needed for the remote job execution independent of the low-level implementation. Implementation-specific parameters can be added to the specification as additional attributes.


  public interface JobSpecification extends Specification {
  
      public void setExecutable(String executable);
      public String getExecutable();
  
      public void setDirectory(String directory);
      public String getDirectory();
  
      public String setArgument(String argument);
      public String setArgument(Vector argument);
      public String getArgument(int index);
      public String getArgumentsAsString();
      public Vector getArgumentsAsVector();
      public void addArgument(String argument);
      public void addArgument(int index, String argument);
      public removeArgument(int index);
      public removeArgument(String argument);
  
      public void setStdOutput(String output);
      public String getStdOutput();
  
      public void setStdInput(String input);
      public String getStdInput();
  
      public void setStdError(String error);
      public String getStdError();
  
      public void setBatchJob(boolean bool);
      public boolean isBatchJob();
  
      public void setRedirected(boolean bool);
      public boolean isRedirected();
  
      public void setLocalExecutable(boolean bool);
      public boolean isLocalExecutable();
  
      public void setLocalInput(boolean bool);
      public boolean isLocalInput();
  
      public void setAttribute(String name, Object value);
      public Object getAttribute(String name);
      public Enumeration getAllAttributes();
  }

The FileTransferSpecification provides the commonly used attributes for file transfers between Grid resources. We note once again that not all attributes are supported by every Grid implementation.


  public interface FileTransferSpecification extends Specification {
  
      public void setSourceDirectory(String directory);
      public String getSourceDirectory();
  
      public void setDestinationDirectory(String directory);
      public String getDestinationDirectory();
  
      public void setSourceFile(String file);
      public String getSourceFile();
  
      public void setDestinationFile(String file);
      public String getDestinationFile();
  
      public void setSource(String source);
      public String getSource();
  
      public void setDestination(String destination);
      public String getDestination();
  
      public void setThirdParty(boolean bool);
      public boolean isThirdParty();
  
      public void setAttribute(String name, Object value);
      public Object getAttribute(String name);
      public Enumeration getAllAttributes();
  }

The FileOperationSpecification offers the functionality to invoke important operations on files hosted on remote Grid resources.

A list of all the supported operations and their corresponding arguments are presented in table 1.

Table 1: Valid command names for FileOperationSpecifications.

Operation

Argument 0

Argument 1

Usage

start

Initializes the connection to the file server

stop

Terminates the connection to the file server

pwd

Returns the current working directory of the file server as a String

cd

String: directoryName

Sets the working directory of the file server

ls

Returns all files in the current directory of the file server as a Collection of GridFiles.

ls

String: directoryName

Returns all files in the specified directory of the file server as a Collection of GridFiles

mkdir

String: directoryName

Creates a directory on the file server with the specified name

rmdir

String: directoryName

Deletes the given directory on the file server, only if it is empty.

rmdir

String: directoryName

boolean: force

Deletes the specified directory on the file server either if it is empty or if force==true

rmfile

String: fileName

Deletes the given file from the file server

exists

String: fileName

Returns a Boolean specifying if the file exists on the file server

isDirectory

String: directoryName

Returns a Boolean specifying if the given directoryName is a valid directory on the file server

getfile

String: remoteFilename

String: localFileName

Transfers the remote file on the file server to the local file on he client machine

putfile

String: localFilename

String: remoteFileName

Transfers the local file on the client machine to the remote file on the file server

getdir

String: remoteDirectoryName

String: localDirectoryName

Transfers the remote directory on the file server to the local directory on the client machine

putdir

String: localDirectoryName

String: remoteDirectoryName

Transfers the local directory on the client machine to the remote directory on the file server

rename

String: oldFileName

String newFileName

Change the name of the old name of the file to the given new name

chmod

String: fileName

Integer: mode

Changes permissions of file on the file server; the mode argument is as per the Unix chmod command.


  public interface FileOperationSpecification extends Specification {
  
      public static String START = __start'';
      public static String STOP = __stop'';
      public static String PWD = __pwd'';
      public static String CD = __cd'';
      public static String LS = __ls'';
      public static String MKDIR = __mkdir'';
      public static String RMDIR = __rmdir'';
      public static String RMFILE = __rmfile'';
      public static String GETFILE = __getfile'';
      public static String PUTFILE = __putfile'';
      public static String GETDIR = __getdir'';
      public static String PUTDIR = __putfile'';
      public static String MGET = __mget'';
      public static String MPUT = __mput'';
      public static String RENAME = __rename'';
      public static String CHMOD = __chmod'';
      public static String EXISTS = __exists'';
      public static String ISDIRECTORY = __isDirectory'';
  
      public void setOperation(String operation);
      public String getOperation();
  
      public void setArgument(String arguments, int index);
      public int addArgument(String argument);
      public Collection getArguments();
      public String getArgument(int n);
      public int getArgumentSize();
  
      public void setAttribute(String name, Object value);
      public Object getAttribute(String name);
      public Enumeration getAllAttributes();
  }

5.4 Service

Every Grid Task has a set of remote Services that support the actual execution of the task. The service interface is a local representation of the remote Grid service. Every service has a provider attribute that specifies the technology and provider supported by that service. It also has a service contact and a security context specific to that provider.


  public interface Service {
      public static final int JOB˘SUBMISSION = 1;
      public static final int FILE˘TRANSFER = 2;
      public static final int INFORMATION˘QUERY = 3;
      public static final int FILE˘OPERATION = 4;
  
      public static final int JOB˘SUBMISSION˘SERVICE = 0;
      public static final int DEFAULT˘SERVICE = 0;
      public static final int FILE˘TRANSFER˘SOURCE˘SERVICE = 0;
      public static final int FILE˘TRANSFER˘DESTINATION˘SERVICE = 1;
  
      public void setIdentity(Identity identity);
      public Identity getIdentity();
  
      public void setName(String name);
      public String getName();
  
      public void setProvider(String provider);
      public String getProvider();
  
      public void setType(int type);
      public int getType();
  
      public void setServiceContact(ServiceContact serviceContact);
      public ServiceContact getServiceContact();
  
      public void setSecurityContext(SecurityContext securityContext);
      public SecurityContext getSecurityContext();
  
      public void setAttribute(String name, Object value);
      public Object getAttribute(String name);
      public Enumeration getAllAttributes();
  }

PIC

Figure 2: A TaskGraph can represent multiple levels of hierarchical DAG

5.5 TaskGraph

A TaskGraph provides a building block for expressing complex dependencies between tasks. Advanced applications require mechanisms to execute client-side workflows that process the tasks based on user-defined control dependencies. Hence, the data structure representing the TaskGraph aggregates a set of ExecutableObjects (Tasks and TaskGraphs) and allows the user to define dependencies between these tasks. In graph theoretical terms, a TaskGraph provides the elements to express workflows as a hierarchical directed acyclic graph (see Figure 2). A TaskGraph can theoretically contain infinite levels of hierarchy. However, practically it is constrained with the availability of resources (memory) on a particular system.


  public interface TaskGraph extends ExecutableObject {
  
      public void add(ExecutableObject graphNode) throws Exception;
      public ExecutableObject remove(Identity id) throws Exception;
      public ExecutableObject get(Identity id);
  
      public ExecutableObject[] toArray();
      public Enumeration elements();
  
      public void setDependency(Dependency dependency);
      public Dependency getDependency();
      public void addDependency(Identity from, Identity to);
      public boolean removeDependency(Identity from, Identity to);
  
      public void setAttribute(String name, Object value);
      public Object getAttribute(String name);
      public Enumeration getAllAttributes();
  
      public int getSize();
      public boolean isEmpty();
      public boolean contains(Identity id);
  
      public int getUnsubmittedCount();
      public int getSubmittedCount();
      public int getActiveCount();
      public int getCompletedCount();
      public int getSuspendedCount();
      public int getResumedCount();
      public int getFailedCount();
      public int getCanceledCount();
  
      public void toXML(File file) throws MarshalException;
  
      public Calendar getSubmittedTime();
      public Calendar getCompletedTime();
  
      public void addChangeListener(ChangeListener listener);
      public void removeChangeListener(ChangeListener listener);
  }

Cog-abstractions provides two additional utility classes that specialize the functionality of the TaskGraph. The Set is a special type of TaskGraph with no dependencies. Intuitively, it represents a bag of tasks that can be executed in parallel. The Queue is another specialized TaskGraph that represents a first-in-first-out (FIFO) queue. The dependencies in a Queue are not set explicitly but are maintained implicitly based on the addition of a Task to the Queue.

5.6 Status

Every ExecutableObject (Task or TaskGraph) has an associated execution status. An ExecutableObject can be in one of the following status: unsubmitted, submitted, active, suspended, resumed, failed, canceled, and completed. We note that not every status is supported by every Grid implementation. In other words, for some Grid implementations it may not be possible to suspend and resume remote execution.

It is easy to associate a simple Task with one of the above mentioned status. For example, initially the task is unsubmitted; its status changes to submitted when it is handled by a handler; its status changes to active when it is being executed remotely; and so on. However, it is not apparent how a TaskGraph is mapped to one the supported status. Cog-abstractions uses the following logic to map a TaskGraph to its appropriate status.


  public interface Status {
  
      public static final int UNSUBMITTED = 0;
      public static final int SUBMITTED = 1;
      public static final int ACTIVE = 2;
      public static final int SUSPENDED = 3;
      public static final int RESUMED = 4;
      public static final int FAILED = 5;
      public static final int CANCELED = 6;
      public static final int COMPLETED = 7;
      public static final int UNKNOWN = 9999;
  
      public abstract void setStatusCode(int status);
      public abstract int getStatusCode();
      public abstract void setPrevStatusCode(int status);
      public abstract int getPrevStatusCode();
      public abstract void setException(Exception exception);
      public abstract Exception getException();
      public abstract void setMessage(String message);
      public abstract String getMessage();
      public void setTime(Calendar time);
      public Calendar getTime();
  }

5.7 Handlers

Cog-abstractions contains the TaskHandler and the TaskGraphHandler, to process a Task and a TaskGraph respectively. Once a Task or a TaskGraph is submitted to the appropriate handler, the handler interacts with the desired Grid implementation and accomplishes the necessary tasks. The handlers in cog-abstractions can be viewed as adaptors that translate the abstract definitions of a Task and TaskGraph into implementation-specific constructs understood by the backend Grid services. For example, a GT3 execution TaskHandler will extract the appropriate attributes from the execution Task and make the necessary calls to the remote Grid service factory, retrieve the Grid service handle, and interact with the newly created service instance. Symmetric translations would be done for other Grid implementations. Intuitively, a handler is specific to the backed implementation and is the only part of cog-abstractions that needs to be extended for supporting additional Grid implementations.


  public interface TaskHandler {
      public void setType(int type);
      public int getType();
  
      public void submit(Task task)
          throws
              IllegalSpecException,
              InvalidSecurityContextException,
              InvalidServiceContactException,
              TaskSubmissionException;
  
      public void suspend(Task task)
          throws InvalidSecurityContextException,
          TaskSubmissionException;
  
      public void resume(Task task)
          throws InvalidSecurityContextException,
          TaskSubmissionException;
  
      public void cancel(Task task)
          throws InvalidSecurityContextException,
          TaskSubmissionException;
  
      public void remove(Task task)
          throws ActiveTaskException;
      public Task[] getAllTasks();
      public Collection getActiveTasks();
      public Collection getFailedTasks();
      public Collection getCompletedTasks();
      public Collection getSuspendedTasks();
      public Collection getResumedTasks();
      public Collection getCanceledTasks();
  }

The TaskGraphHandler provides a similar functionality as the task handler interface. However, it has an additional responsibility of enforcing the dependency on the graph-like task sets submitted to it. It can be implemented as an advanced workflow engine coordinating the execution of tasks on corresponding Grid resources honoring the user-defined dependencies.


  public interface TaskGraphHandler {
  
      public void submit(TaskGraph taskgraph)
         throws
         IllegalSpecException,
         InvalidSecurityContextException,
         InvalidServiceContactException,
         TaskSubmissionException;
  
      public void suspend()
         throws InvalidSecurityContextException,
         TaskSubmissionException;
  
      public void resume()
         throws InvalidSecurityContextException,
         TaskSubmissionException;
  
      public void cancel()
         throws InvalidSecurityContextException,
         TaskSubmissionException;
  
      public Task[] getAllTask();
      public Collection getActiveTasks();
      public Collection getFailedTasks();
      public Collection getCompletedTasks();
      public Collection getSuspendedTasks();
      public Collection getResumedTasks();
      public Collection getCanceledTasks();
  }

5.8 FileResource

An alternate model of abstraction for file operations, bypassing the task model, is also available for applications that desire direct interactions with the file hosting Grid servers. A FileResource provides all the necessary functionality to directly invoke remote file operations. Methods of a FileResource bear close correspondence with the commands outlined in Table 1.


  public interface FileResource{
  
      public static final String FTP = __ftp'';
      public static final String GridFTP = __gridftp'';
      public static final String WebDAV = __webdav'';
      public static final String Local = __local'';
  
      public String getProtocol();
  
      public void setServiceContact(ServiceContact serviceContact);
      public ServiceContact getServiceContact();
  
      public void setSecurityContext(SecurityContext securityContext);
      public SecurityContext getSecurityContext();
  
      public void start() throws IllegalHostException,
              InvalidSecurityContextException, GeneralException;
      public void stop() throws GeneralException;
      public boolean isStarted();
  
      public void setCurrentDirectory(String directoryName)
              throws DirectoryNotFoundException, GeneralException;
      public String getCurrentDirectory() throws GeneralException;
  
      public Collection list() throws GeneralException;
      public Collection list(String directoryName)
              throws DirectoryNotFoundException, GeneralException;
  
      public void createDirectory(String directoryName) throws GeneralException;
      public void deleteDirectory(String directoryName, boolean force)
              throws DirectoryNotFoundException, GeneralException;
      public void deleteFile(String fileName)
              throws FileNotFoundException, GeneralException;
  
      public void getFile(String remoteFileName, String localFileName)
              throws FileNotFoundException, GeneralException;
      public void putFile(String localFileName, String remoteFileName)
              throws FileNotFoundException, GeneralException;
  
      public void getDirectory(String remoteDirectoryName, String localDirectoryName)
              throws DirectoryNotFoundException, GeneralException;
      public void putDirectory(String localDirectoryName, String remoteDirectoryName)
              throws DirectoryNotFoundException, GeneralException;
  
      public void getMultipleFiles(String[] remoteFileNames, String[] localFileNames)
              throws FileNotFoundException, GeneralException;
      public void getMultipleFiles(String[] remoteFileNames, String localDirectoryName)
              throws FileNotFoundException, DirectoryNotFoundException,
              GeneralException;
      public void putMultipleFiles(String[] localFileNames, String[] remoteFileNames)
              throws FileNotFoundException, GeneralException;
      public void putMultipleFiles(String[] localFileNames, String remoteDirectoryName)
              throws FileNotFoundException, DirectoryNotFoundException,
              GeneralException;
  
      public void rename(String oldFileName, String newFileName)
              throws FileNotFoundException, GeneralException;
  
      public void changeMode(String fileName, int mode)
              throws FileNotFoundException, GeneralException;
      public void changeMode(GridFile gridFile)
              throws FileNotFoundException, GeneralException;
  
      public GridFile getGridFile(String fileName)
              throws FileNotFoundException, GeneralException;
  
      public boolean exists(String fileName)
              throws FileNotFoundException, GeneralException;
  
      public boolean isDirectory(String directoryName)
              throws GeneralException;
  
      public void submit(ExecutableObject commandWorkflow)
              throws IllegalSpecException, TaskSubmissionException;
  
      public void setAttribute(String name, Object value);
      public Enumeration getAllAttributes();
      public Object getAttribute(String name);
  }

5.9 GridFile

A GridFile is an abstract representation of a remote file or directory. It represents the basic properties of a file such as size, name, modification date, access permissions etc. This abstraction is a passive information carrier and cannot be used to modify the properties or contents of a remote resource directly. It can be used as an input to the file resource abstraction to change access permissions of a remote resource.


  public interface GridFile {
  
      public static final byte UNKNOWN˘TYPE = 0;
      public static final byte FILE˘TYPE = 1;
      public static final byte DIRECTORY˘TYPE = 2;
      public static final byte SOFTLINK˘TYPE = 3;
      public static final byte DEVICE˘TYPE = 4;
  
      public void setSize(long size);
      public long getSize();
  
      public void setName(String name);
      public String getName();
  
      public void setAbsolutePathName(String name);
      public String getAbsolutePathName();
  
      public void setLastModified(String date);
      public String getLastModified();
  
      public void setFileType(byte type);
      public byte getFileType();
  
      public boolean isFile();
      public boolean isDirectory();
      public boolean isSoftLink();
      public boolean isDevice();
  
      public void setMode(String mode);
      public String getMode();
  
      public void setUserPermissions(Permissions userPermissions);
      public Permissions getUserPermissions();
  
      public void setGroupPermissions(Permissions groupPermissions);
      public Permissions getGroupPermissions();
  
      public void setAllPermissions(Permissions allPermissions);
      public Permissions getAllPermissions();
  
      public boolean userCanRead();
      public boolean userCanWrite();
      public boolean userCanExecute();
  
      public boolean groupCanRead();
      public boolean groupCanWrite();
      public boolean groupCanExecute();
  
      public boolean allCanRead();
      public boolean allCanWrite();
      public boolean allCanExecute();
  }

5.10 Permissions

The permissions interface provides a means to get and set permissions for a class of users along with GridFile and FileResource abstractions.


  public interface Permissions{
  
      public void setRead(boolean canRead);
      public boolean getRead();
  
      public void setWrite(boolean canWrite);
      public boolean getWrite();
  
      public void setExecute(boolean canExecute);
      public boolean getExecute();
  
      public String toString();
  }

6 Programmer’s Guide

  1. How to execute a remote job execution task (6.1)
  2. How to execute a file transfer task (6.2)
  3. How to execute a file operation task (6.3)
  4. How to execute a simple task graph (DAG) (6.4)
  5. How to execute a hierarchical task graph (6.5)
  6. How to checkpoint a task (6.6)
  7. How to execute a checkpointed task (6.7)
  8. How to checkpoint a task graph (6.8)
  9. How to resume a checkpointed task graph (6.9)
  10. How to invoke remote file operations without adhering to the task model (6.10)
  11. How to run a native Condor job with the CoG Kit abstractions (6.11)
  12. How to integrate a new provider module in the Java CoG Kit(6.12)

6.1 How to execute a remote job execution task

Executing a remote job becomes extremely simple with cog-abstractions. To begin with, create a Task with the appropriate attributes.


  Task task = new TaskImpl(__myTestTask'', Task.JOB˘SUBMISSION);
  
  /⋆ Set the desired provider. Default options are
     GT2, GT3.0.2, GT3.2.0, GT3.2.1, GT4.0.0, Condor, or SSH
  ⋆/
  task.setProvider(__GT4.0.0'');

Then, create a JobSpecification for the task and set the appropriate attributes as per the task requirements.


  JobSpecification spec = new JobSpecificationImpl();
  
  /⋆ Set the location and name of the executable.
     If the executable is a local executable, then
     spec.setLocalExecutable(true)
  ⋆/
  spec.setExecutable(__/bin/ls'');
  
  /⋆ Set the arguments (if any)
     for the executable
  ⋆/
  spec.addArguments(__-l'');
  spec.addArguments(__-a'');
  
  /⋆ Set the name of the file which serves
     as the input to the executable
  
     If the input file needs to be redirected
     from the local machine, then
     spec.setLocalInput(true)
  ⋆/
  spec.setStdInput(__abstractions-testInput'');
  
  /⋆ Set the name of the file to which the remote
     output must be stored in.
  
     If the remote output needs to be redirected
     to the local machine, then
     spec.setRedirected(true)
  
     If the remote output needs to be manipulated at
     the local machine rather than storing it in a
     file, then
     spec.setRedirected(true);
     spec.setStdOutput(null);
     The output is now available from
     task.getOutput(); and can be used
     or displayed as desired.
  ⋆/
  spec.setStdOutput(__abstractions-testOutput'');
  
  /⋆ Set the execution mode of the job ⋆/
  spec.setBatchJob(true);
  
  /⋆ Add additional attributes that are not
     provided by default. These add on
     attributes will be considered by the
     handler only if it supports it.
  ⋆/
  spec.setAttribute(__count'',__546'');
  
  /⋆ Assign this specification to the task ⋆/
  task.setSpecification(spec);

Next, assign an appropriate service to the task with a provider, service contact, and security context. This step assumes you have a valid user certificate successfully obtained from appropriate certificate authority.


  Service service = new ServiceImpl(Service.JOB˘SUBMISSION);
  
  /⋆ Assign a provider to this service.
     The provider assigned to the service
     must be the same as that assigned to the task.
  ⋆/
  service.setProvider(__GT4.0.0'');
  
  /⋆ Retrieve the appropriate security context
     for this provider from the AbstractionFactory class
  ⋆/
  SecurityContext securityContext =
    AbstractionFactory.newSecurityContext(__GT4.0.0'');
  
  /⋆ Assign the default credentials
     available as a valid proxy certificate
     whose location is specified in the
     cog.properties file present in the
     HOME/.globus directory
  
     To assign non-default credentials
     create a GSSCredential and pass
     this GSSCredential as the argument
     instead of null
  ⋆/
  securityContext.setCredentials(null);
  
  /⋆ Assign this security credential to the service ⋆/
  service.setSecurityContext(securityContext);
  
  /⋆ Create a new service contact with the
     appropriate service endpoint address
  ⋆/
  ServiceContact serviceContact =
      new ServiceContactImpl(
      __https://127.0.0.1:8443/wsrf/services/ManagedJobFactoryService'');
  
  /⋆ Assign this service contact with the service ⋆/
  service.setServiceContact(serviceContact);
  
  /⋆ Assign the service to the task ⋆/
  task.setService(Service.JOB˘SUBMISSION˘SERVICE, service);

Next, create a TaskHandler and submit the task for execution.


      try {
        handler.submit(task);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

If it is required to monitor the status of the task (desired in most interactive tasks), then the monitoring class must implement the StatusListener interface and before submitting the task to a handler subscribe to the task for its status changes.


  task.addStatusListener(this);

If registered to listen to the status notification of the task, implement the statusChanged() function.


  public void statusChanged(StatusEvent event)
  {
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
          + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED) {
        /⋆ Makes sense if
           spec.setRedirected(true);
    spec.setStdOutput(null);
        ⋆/
        logger.debug(__Output = ''
            + task.getStdOutput());
        System.exit(1);
      }
  }

6.2 How to execute a file transfer task

Executing a file transfer is extremely simple with cog-abstractions. To begin with, create a Task with the appropriate attributes.


  Task task = new TaskImpl(__myTestTask'', Task.FILE˘TRANSFER);
  
  /⋆ Since a file transfer task has two providers:
     the source provider and the destination provider,
     we do not set the provider attribute of the task.
  ⋆/

Then, create a FileTransferSpecification for the task and set the appropriate attributes as per the task requirements.


  FileTransferSpecification spec = new FileTransferSpecificationImpl();
  
  /⋆ Set the source and destination files ⋆/
  spec.setSourceDirectory(__/home/username'');
  spec.setSourceFile(__sourceFile'');
  
  spec.setDestinationDirectory(__/home/username'');
  spec.setDestinationFile(__destinationFile'');
  
  /⋆ Assign this specification to the task ⋆/
  task.setSpecification(spec);

Next, assign the appropriate source and destination services service to the task.


  Service sourceService = new ServiceImpl(Service.FILE˘TRANSFER);
  
  /⋆ Assign a provider to this service.
  ⋆/
  sourceService.setProvider(__GridFTP'');
  
  /⋆ Retrieve the appropriate security context
     for this provider from the AbstractionFactory class
  ⋆/
  SecurityContext sourceSecurityContext =
    AbstractionFactory.newSecurityContext(__GridFTP'');
  
  /⋆ Assign the default credentials
     available as a valid proxy certificate
     whose location is specified in the
     cog.properties file present in the
     HOME/.globus directory
  
     To assign non-default credentials
     create a GSSCredential and pass
     this GSSCredential as the argument
     instead of null
  ⋆/
  sourceSecurityContext.setCredentials(null);
  
  /⋆ Assign this security credential to the source service ⋆/
  sourceService.setSecurityContext(sourceSecurityContext);
  
  /⋆ Create a new service contact with the
     appropriate service endpoint address
  ⋆/
  ServiceContact sourceServiceContact =
      new ServiceContactImpl();
  
  sourceServiceContact.setHost(__hot.anl.gov'');
  sourceServiceContact.setPort(2811);
  
  /⋆ Assign this service contact with the service ⋆/
  sourceService.setServiceContact(sourceServiceContact);
  
  /⋆ Assign the service to the task ⋆/
  task.setService(Service.SOURCE˘FILE˘TRANSFER˘SERVICE, sourceService);
  
  /⋆ Create a destination file transfer service ⋆/
  Service destinationService = new ServiceImpl(Service.FILE˘TRANSFER);
  
  /⋆ Assign a provider to this service.
     The provider assigned to the service
     must be the same as that assigned to the task.
  ⋆/
  destinationService.setProvider(__GridFTP'');
  
  /⋆ Retrieve the appropriate security context
     for this provider from the AbstractionFactory class
  ⋆/
  SecurityContext destinationSecurityContext =
    AbstractionFactory.newSecurityContext(__GridFTP'');
  
  /⋆ Assign the default credentials
     available as a valid proxy certificate
     whose location is specified in the
     cog.properties file present in the
     HOME/.globus directory
  
     To assign non-default credentials
     create a GSSCredential and pass
     this GSSCredential as the argument
     instead of null
  ⋆/
  destinationSecurityContext.setCredentials(null);
  
  /⋆ Assign this security credential to the destination service ⋆/
  destinationService.setSecurityContext(destinationSecurityContext);
  
  /⋆ Create a new service contact with the
     appropriate service endpoint address
  ⋆/
  ServiceContact destinationServiceContact =
      new ServiceContactImpl();
  
  destinationServiceContact.setHost(__cold.anl.gov'');
  destinationServiceContact.setPort(2811);
  
  /⋆ Assign this service contact with the service ⋆/
  destinationService.setServiceContact(destinationServiceContact);
  
  /⋆ Assign the service to the task ⋆/
  task.setService(Service.DESTINATION˘FILE˘TRANSFER˘SERVICE, destinationService);

Next, create a TaskHandler and submit the task for execution.


      try {
        handler.submit(task);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

If it is required to monitor the status of the task (desired in most interactive tasks), then the monitoring class must implement the StatusListener interface and before submitting the task to a handler subscribe to the task for its status changes.


  task.addStatusListener(this);

If registered to listen to the status notification of the task, implement the statusChanged() function.


  public void statusChanged(StatusEvent event) {
  
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
         + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED }}
          status.getStatusCode() == Status.FAILED) {
        logger.info(__Task Done'');
        System.exit(1);
      }
  }

6.3 How to execute a file operation task

Execution of file operation tasks work slightly differently compared to other tasks. Execution and file transfer tasks are submitted independently of other tasks, thereby displaying a connectionless behavior between the client machine and the remote server. On the other hand, file operation tasks are dependent on each other showing a connection-oriented behavior between the client machine and the remote server. For example, lets assume the client executes a file operation task to “start” a remote connection. The client then executes a task to “list” all the files in the current directory. It can be seen that the two task are not independent, and that the execution of the second task must be done in the context of the connection established in the first task.

Therefore, to execute connection-oriented file operation tasks, we adopt a notion of “sessionID”. To begin with, we create a Task with appropriate attributes.


  Task task = new TaskImpl(__myFirstTask'', Task.FILE˘OPERATION);
  
  /⋆ Set the desired provider. Default options are
     GridFTP, FTP, or WebDAV
  ⋆/
  task.setProvider(__gridftp'');

Then, create a FileOperationSpecification for the task and set the appropriate attributes as per the task requirements.


  FileOperationSpecification spec = new FileOperationSpecificationImpl();
  
  /⋆ Set the operation and the corresponding arguments.
     In our example, the operation is __start'' since we want to
     establish a new connection with the file server. We note that
     __start'' is ALWAYS the first operation to be executed in any
     sequence of related file operations.
  ⋆/
  spec.setOperation(FileOperationSpecification.START);
  
  /⋆ The __start'' operation has no arguments ⋆/
  
  /⋆ Assign this specification to the task ⋆/
  task.setSpecification(spec);

Next, assign an appropriate service to the task with a provider, service contact, and security context.


  Service service = new ServiceImpl(Service.FILE˘OPERATION);
  
  /⋆ Assign a provider to this service.
     The provider assigned to the service
     must be the same as that assigned to the task.
  ⋆/
  service.setProvider(__gridftp'');
  
  /⋆ Retrieve the appropriate security context
     for this provider from the AbstractionFactory class
  ⋆/
  SecurityContext securityContext =
    AbstractionFactory.newSecurityContext(__gridftp'');
  
  /⋆ Assign the default credentials
     available as a valid proxy certificate
     whose location is specified in the
     cog.properties file present in the
     HOME/.globus directory
  
     To assign non-default credentials
     create a GSSCredential and pass
     this GSSCredential as the argument
     instead of null
  ⋆/
  securityContext.setCredentials(null);
  
  /⋆ Assign this security credential to the service ⋆/
  service.setSecurityContext(securityContext);
  
  /⋆ Create a new service contact with the
     appropriate service endpoint address
  ⋆/
  ServiceContact serviceContact =
      new ServiceContactImpl(__hot.anl.gov:2119'');
  
  /⋆ Assign this service contact with the service ⋆/
  service.setServiceContact(serviceContact);
  
  /⋆ Assign the service to the task ⋆/
  task.setService(Service.FILE˘OPERATION˘SERVICE, service);

Next, create a TaskHandler and submit the task for execution.


      try {
        handler.submit(task);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

Since file operation tasks are always interactive tasks, we monitor the status of the task. The monitoring class must implement the StatusListener interface and before submitting the task to a handler subscribe to the task for its status changes.


  task.addStatusListener(this);

The monitoring class listen to the status notification of the task, by implementing the statusChanged() method.


  public void statusChanged(StatusEvent event)
  {
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
          + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED) {
        /⋆ The output of the __start'' operation is
            a sessionID. This sessionID can be used
     for all successive file operation tasks
     to invoke operations with context to this
     newly established connection.
        ⋆/
  
        logger.debug(__SessionID = ''
            + task.getAttribute(__output''));
      }
  }

We note that rather than extracting the output from task.getStdOutput() (which returns a String), we retrieve the output as task.getAttribute(“output”) (which returns an Object). The output Object can then be type cast to the target class as specified in Table 1.

Once we have obtained the connection sessionID, we can submit any file operation specified in Table 1. For example, lets execute the “list directoryName” operation. We create a Task with appropriate attributes.


  Task task = new TaskImpl(__mySecondTask'', Task.FILE˘OPERATION);
  
  /⋆ Set the desired provider. Default options are
     GridFTP, FTP, or WebDAV
  ⋆/
  task.setProvider(__gridftp'');

Then, create a FileOperationSpecification for the task and set the appropriate attributes as per the task requirements.


  FileOperationSpecification spec = new FileOperationSpecificationImpl();
  
  /⋆ Set the operation and the corresponding arguments.
     In our example, the operation is __list directoryName''.
  ⋆/
  spec.setOperation(FileOperationSpecification.LS);
  
  /⋆ We then set the arguments for this operations ⋆/
  spec.addArgument(__/home/amin/test'');
  
  /⋆ Assign this specification to the task ⋆/
  task.setSpecification(spec);

Since, we have already retrieved a sessionID for this connection, we do not need to create a new Service for this task. Instead, we simply specify the sessionID.


     executing the __start'' operation. We note here the
     importance of the initial __start'' operation. Without
     that operation, we will not get any sessionID.
  ⋆/
  task.setAttribute(__sessionID'', sessionID);

Next, create a TaskHandler and submit the task for execution.


      try {
        handler.submit(task);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

Once again we monitor the status of the task. The monitoring class must implement the StatusListener interface and before submitting the task to a handler subscribe to the task for its status changes.


  task.addStatusListener(this);

The monitoring class listen to the status notification of the task, by implementing the statusChanged() method.


  public void statusChanged(StatusEvent event)
  {
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
          + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED) {
        /⋆ The output of the __list'' operation is
            a Collection of GridFile objects. This Collection is
     manipulated as desired.
        ⋆/
      }
  }

Table 1 lists all valid operations, their input arguments, and their corresponding outputs.

6.4 How to execute a simple task graph (DAG)

To create a TaskGraph, we assume that we have created three tasks: task1, task2, and task3. Instructions for creating job submission and file transfer tasks are available in Sections 6.1 and 6.2 respectively. We then create a TaskGraph and add a dependency between these tasks.


  /⋆ Give a convenient name to the TaskGraph ⋆/
  tg.setName(__testGraph'');
  
  /⋆ Add the tasks to the TaskGraph ⋆/
  tg.add(task1);
  tg.add(task2);
  tg.add(task3);
  
  /⋆ Add dependencies between these tasks.
  
     Dependency is added as
     task1 --> task2 --> task3.
  
     This implies task1 is executed before task2
     and task2 is executed before task3.
  ⋆/
  tg.addDependency(task1,task2);
  tg.addDependency(task2,task3);

Next, create a TaskGraphHandler and submit the task for execution.


      try {
        handler.submit(tg);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

If it is required to monitor the status of the task graph (desired in most interactive task graphs), then the monitoring class must implement the StatusListener interface and before submitting the task graph to a handler subscribe to the task graph for its status changes.


  tg.addStatusListener(this);

If registered to listen to the status notification of the task graph, implement the statusChanged() function.


  public void statusChanged(StatusEvent event) {
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
         + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED }}
          status.getStatusCode() == Status.FAILED) {
        logger.info(__Task Graph Done'');
        System.exit(1);
      }
  }

6.5 How to execute a hierarchical task graph

To create a hierarchical TaskGraph, we assume that we have created three tasks and one TaskGraph: task1, task2, task3, and tg. Instructions for creating job submission and file transfer tasks and simple TaskGraphs are available in Sections 6.1, 6.2, and 6.4 respectively. We then create a TaskGraph and add a dependency between these ExecutableObjects.


  /⋆ Give a convenient name to the TaskGraph ⋆/
  htg.setName(__testGraph'');
  
  /⋆ Add the ExecutableObjects to the TaskGraph ⋆/
  htg.add(task1);
  htg.add(task2);
  htg.add(task3);
  htg.add(tg);
  
  /⋆ Add dependencies between these ExecutableObjects.
  
     Dependency is added as
     task1 --> task2 --> task3 --> tg.
  
     This implies task1 is executed before task2
     and task2 is executed before task3, and
     task3 is executed before TaskGraph tg.
  ⋆/
  htg.addDependency(task1,task2);
  htg.addDependency(task2,task3);
  htg.addDependency(task3,tg);

Next, create a TaskGraphHandler and submit the task for execution.


      try {
        handler.submit(htg);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'',ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'',tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'',ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'',isce);
        System.exit(1);
      }

If it is required to monitor the status of the task graph (desired in most interactive task graphs), then the monitoring class must implement the StatusListener interface and before submitting the task graph to a handler subscribe to the task graph for its status changes.


  htg.addStatusListener(this);

If registered to listen to the status notification of the task graph, implement the statusChanged() function.


  public void statusChanged(StatusEvent event) {
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
         + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED }}
          status.getStatusCode() == Status.FAILED) {
        logger.info(__Task Graph Done'');
        System.exit(1);
      }
  }

6.6 How to checkpoint a task

We assume you have created a Grid task as mentioned in Sections 6.1 and 6.2. To checkpoint this task you need to invoke the “marshall” method of the TaskMarshaller class, supplying it with the task and details of the checkpointing file.


  try {
      File xmlFile = new File(__Task.xml'');
      xmlFile.createNewFile();
      TaskMarshaller.marshal(task, xmlFile);
  } catch (Exception e) {
     logger.error(__Cannot marshal the task'', e);
  }
  

6.7 How to execute a checkpointed task

We assume the Grid task is checkpointed in an XML file called “Task.xml”. First, unmrshal the checkpointed file into a task object.


      File xmlFile = new File(__Task.xml'');
      task = TaskUnmarshaller.unmarshal(xmlFile);
  } catch (Exception e) {
    logger.error(__Cannot unmarshal task'', e);
  }

Next, create a TaskHandler and submit the task for execution.


      try {
        handler.submit(task);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

If it is required to monitor the status of the task (desired in most interactive tasks), then the monitoring class must implement the StatusListener interface and before submitting the task to a handler subscribe to the task for its status changes.


  task.addStatusListener(this);

If registered to listen to the status notification of the task, implement the statusChanged() function.


  public void statusChanged(StatusEvent event) {
  
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
         + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED }}
          status.getStatusCode() == Status.FAILED) {
        logger.info(__Task Done'');
        System.exit(1);
      }
  }

6.8 How to checkpoint a task graph

We assume you have created a task graph as mentioned in Sections 6.4 and 6.5. To checkpoint this task graph, you need to invoke the “marshall” method of the TaskGraphMarshaller class, supplying it with the task graph and details of the checkpointing file.


      File xmlFile = new File(__TaskGraph.xml'');
      xmlFile.createNewFile();
      TaskGraphMarshaller.marshal(taskGraph, xmlFile);
  } catch (Exception e) {
     logger.error(__Cannot marshal the task graph'', e);
  }

6.9 How to resume a checkpointed task graph

We assume the task graph is checkpointed in an XML file called “TaskGraph.xml”. First, unmrshal the checkpointed file into a task graph object.


  try {
      File xmlFile = new File(__TaskGraph.xml'');
      taskGraph = TaskGraphUnmarshaller.unmarshal(xmlFile);
  } catch (Exception e) {
    logger.error(__Cannot unmarshal task graph'', e);
  }

Next, create a TaskGraphHandler and submit the task graph for execution.


      try {
        handler.submit(taskGraph);
      } catch (InvalidSecurityContextException ise) {
        logger.error(__Security Exception'', ise);
        System.exit(1);
      } catch (TaskSubmissionException tse) {
        logger.error(__TaskSubmission Exception'', tse);
        System.exit(1);
      } catch (IllegalSpecException ispe) {
        logger.error(__Specification Exception'', ispe);
        System.exit(1);
      } catch (InvalidServiceContactException isce) {
        logger.error(__Service Contact Exception'', isce);
        System.exit(1);
      }

This will initiate the re-execution of all previously uncompleted elements of the task graph. If it is required to monitor the status of the task graph, then the monitoring class must implement the StatusListener interface and before submitting the taskgraph to a handler subscribe to the task graph for its status changes.


  taskGraph.addStatusListener(this);

If registered to listen to the status notification of the task graph, implement the statusChanged() function.


  public void statusChanged(StatusEvent event) {
  
      Status status = event.getStatus();
  
      logger.debug(__Status changed to ''
         + status.getStatusString());
  
      if (status.getStatusCode() == Status.COMPLETED }}
          status.getStatusCode() == Status.FAILED) {
        logger.info(__Task graph Done'');
        System.exit(1);
      }
  }

6.10 How to invoke remote file operations without adhering to the task model

File operations can invoked on any file server without adhering to the traditional task model. Create a FileResource object


  FileResource resource = AbstractionFactory.newFileResource(provider);
  
  /⋆ Set an appropriate name for this resource ⋆/
  resource.setName(__MyFileResource'');
  
  /⋆ Create appropriate security context for this resource ⋆/
  securityContext = AbstractionFactory.newSecurityContext(__local'');
  
  /⋆ Set the security context for this resource ⋆/
  resource.setSecurityContext(securityContext);
  
  /⋆ Create a new service contact with the
     appropriate service endpoint address
  ⋆/
  ServiceContact serviceContact =
      new ServiceContactImpl(__hot.anl.gov:2119'');
  
  /⋆ Set the service contact for this resource ⋆/
  resource.setServiceContact(serviceContact);
  
  /⋆ Initialize the file resource connection  ⋆/
  resource.start();
  
  /⋆ Invoke any method available on the file
     resource object and process its input
     as required
  ⋆/
  
  /⋆ When done with this file resource,
     terminate the connection
  ⋆/
  resource.stop();

6.11 How to run a native Condor job using the CoG Kit abstractions

Native Condor jobs are submitted to the Condor pool using job description files. This is in contrast to the interface- and API-based approach adopted in the Java CoG Kit. Nevertheless, the abstractions offered by the CoG Kit can be used to submit native Condor jobs without explicitly creating description files. Since the Condor job submission paradigm requires all job submissions to be done locally from a Condor submit machine, throughout this example we assume that the job will be executed locally from the Condor submit machine.

To begin with, create a Task with the appropriate attributes.


  Task task = new TaskImpl(__myCondorTask'', Task.JOB˘SUBMISSION);
  
  /⋆ Set the provider as Condor.
  ⋆/
  task.setProvider(__Condor'');

Then, create a JobSpecification for the task and set the appropriate attributes as per the task requirements.


  JobSpecification spec = new JobSpecificationImpl();
  
  /⋆ Set the location and name of the executable.
  ⋆/
  spec.setExecutable(__/bin/ls'');
  
  /⋆ Set the arguments (if any)
     for the executable
  ⋆/
  spec.addArguments(__-l'');
  spec.addArguments(__-a'');
  
  /⋆ Set the path name of the initial
     directory. This method is equivalent
     to the Initialdir attribute of the
     Condor submit description file.
  ⋆/
  spec.setDirectory(__/home/amin'')
  
  /⋆ Set the name of the file which serves
     as the input to the executable
  ⋆/
  spec.setStdInput(__condor-input'');
  
  /⋆ Set the name of the file to which the remote
     output and remote error must be stored in.
     If left blank or null, the output and error
     will be redirected to the /dev/null of the
     remote machine.
  ⋆/
  spec.setStdOutput(__condor-output'');
  spec.setStdError(__condor-error'');
  
  /⋆ Add additional attributes that are not
     provided by in the JobSpecification.
     All the attributes that are acceptable in
     Condor job submit file can be specified
     as a (name,value) pair.
     For example, to specify the LOG file
     in a job, the user can set the value as:
  ⋆/
  spec.setAttribute(__log'', __condor-log'');
  
  /⋆ Assign this specification�