This document includes basic information about the different Java CoG Kit workflow options.
The material presented in this document can not be published, mirrored, electronically or otherwise reproduced without prior written consent. As you can link to this document, this should not pose much of a restriction.
The best way to read this document is with Adobe Acrobat Reader. Please make sure you configure Adobe Acrobat Reader appropriately so you can follow hyperlinks. This is the case if you follow the default installation. Acrobat Reader is available at http://www.adobe.com/products/acrobat/readermain.html. Because the hyperlinks are not available in the printed form of this manual and we support saving our environment we strongly discourage printing this document.
We recommend that you save this manual locally on your machine and use Acrobat Reader. This has the advantage that you do not lose your anchor points while switching back and forth between different hyperlinks. An HTML version of this manual is planed, but not available yet.
We have augmented the document with some comments at places where we found issues. Our intend is to address these issues in a future release. The comments are marked by the icon to do and the name of the person that will work on the removal of the issue.
Please be a team player and support us indirectly by registering with us or reporting your use of the Java CoG Kit. Although this software is free, we still need to justify to our funders the usefulness of the projects. If you want to help us with our efforts please take a few seconds to complete this information. We do not use this information for other purposes. If you have special needs or concerns please contact gregor@mcs.anl.gov. The registration form can filled out in a variety of formats. The online form can be found at
This form is available also as ASCII text at
which you can FAX to
Gregor von Laszewski, Fax: 630 252 1997
Gregor: explain the name
___________________________________________________________________________________________________________________________________
Karajan is a parallel scripting language. It uses a declarative concurrency approach to parallel programming. An interpreter is also provided, featuring the ability to checkpoint the state of the execution.
Karajan supports two syntax modes: a native syntax and XML. There is no difference on the semantic level between the two forms.
The following conventions are used:
(xy) are used to group x and y
[x] indicates that x is optional
x+ denotes at least one occurrence of x
x* denotes zero or more occurrences of x
x∣y means either x or y
′x′ is to be interpreted as the literal x
E represents the empty production
The Karajan semantics revolve around the notion of elements. An element is relatively similar to a function in that it has a name, can accept arguments, and may return values. The general syntax for an element is:
Example:
An identifier can consist of alpha-numeric characters and certain symbols, but no whitespace. Symbols that cannot be used in an identifier are symbols that have other syntactic functions, such as brackets (all of them), commas, double quotes, and the equals sign. Identifiers are case insensitive.
Example:
A number cannot be an identifier. However, certain characters that are used to represent a number can be identifiers (such as ’+’, ’-’, ’.’)
Valid identifiers are also ’==’, ’<=’, ’>=’ (even though the general rule for identifiers is broken in their case).
The arguments can either be other elements or values, such as numeric values or strings. Elements can be separated by commas or the new line character (or both):
Example:
Arguments come in two flavors. Named arguments and unnamed arguments:
Named arguments provide a way of explicitly binding arguments to formal parameters:
Example:
Unnamed arguments can be either immediate values or elements. Immediate values can be numeric literals, string literals, variables, or quoted lists:
Example:
A quoted list is a special element that produces a list of identifiers. What is specific about a quoted list is that if its arguments are variables, the variables will not be evaluated. Instead their identifiers will be added to the list. Quoted lists are convenience syntax for expressing a list of formal arguments:
However, quoted lists are not limited to expressing list of arguments. They can also be used to express lists of values. The only thing to remember is that variable evaluation will not take place for immediate arguments of a quoted list.
Example:
A Karajan program is a list of arguments:
There exists an implicit root element that sits at the top of the element tree, and implements certain system functions.
And finally, Karajan uses C-style comments. Single-line comments begin with two forward slashes and end at the following new line character, while multi-line comments are delimited by ’/*’ and ’*/’:
Karajan also supports XML as its syntax. In the XML syntax, each XML element corresponds to a Karajan element. Arguments can be expressed either through XML attributes or nested elements.
One of the particular aspects of using XML with Karajan is that when using XML attributes for arguments, it is impossible to make a syntactic distinction between a numeric value and its string representation. In general, Karajan will try to use the context to figure out which one is desired, but there are instances when it is impossible to do so. Therefore, when using the XML syntax, the following elements can be used for the purpose of differentiating between numeric and string values: number and string
The equivalent Karajan construct would be:
Karajan can load and interpret arbitrary XML files, provided that definitions exist for the XML elements present in the file, but XML mixed content is not handled properly. The unfortunate aspect is that it is impossible to handle XML mixed content in a generic way. For example, it cannot be known whether whitespace between two XML elements is to be interpreted as content or not, without knowledge of the implementation of an element. Since Karajan is a dynamic language, the implementation of an element is not known statically, at the time the parsing takes place. Therefore, the following rule was adopted: An element will consider textual content content if and only if no nested elements exist. If nested elements exist, textual content will be ignored.
If processed, textual content will be mapped as a string argument. A consequence of the above rule is that textual content and multiple arguments are mutually exclusive.
Lastly, a well-formed XML document must always have a root element. While in the native Karajan syntax, the root element is implicit, in XML the project or karajan elements can be used as root elements.
An element can accept any number of arguments and can generate any number of return values, and that includes an infinite number of arguments and/or return values (at least in theory).
Arguments are divided into two major types: single value arguments and channels. As their name implies, single value arguments can have only one value. By contrast, channels can be used for any number of values.
Single value arguments can be specified using the named argument form. For example, the print element has a message argument. Thus passing a string as the message argument to print element can be done in the following way:
Or in XML:
Single value arguments can be further divided into mandatory and optional arguments.
Channels can be used to pass multiple arguments to an element. Each channel has a name, except for the default channel. The default channel is similar to the notion of variable arguments in C. Passing arguments on the default channel is done implicitly when arguments are not passed as single value arguments:
In the above case, 1 and ”value” are both passed to the list element on the default channel.
Elements define whether they do receive arguments on a specific channel or not. One possibly interesting aspect is that an element that does not process arguments on a channel, will automatically return all values received on that channel. It is therefore possible to use named channels to return values to elements other than the immediate parent. Assuming that foo is an element that does not take any arguments on any channels, the following will produce the same result:
Since foo does not process any arguments, all the arguments it receives on the default channel will be returned to the parent element.
It is not always convenient to use the named argument form to pass arguments to an element. Elements in Karajan will automatically map arguments received on the default channel to single value arguments. The mapping is done dynamically, in the order arguments are received. Suppose there is an element foo that takes three arguments, namely ’one’, ’two’ and ’three’. The following would then be equivalent:
The unfortunate side-effect of using automatic mapping of default channel arguments to single value arguments is that elements that would accept both single value arguments and arguments on the default channel (variable arguments) cannot avoid mapping of variable arguments to certain single value arguments unless different semantics are introduced: optional arguments. Optional arguments do not need to be specified. However, if specified, the named form must always be used. An example is the print element, which has an optional argument named nl . It can be set to false to indicate that no new-line character should be appended at the end of the message argument:
or
The following however, is not valid:
Return values are a mirror image of the arguments concept. Whatever can be accepted as an argument by an element can also be returned by another. Thus, it is possible to define a single element that returns all arguments to any given element. The following example defines an element that returns both a message and the named form of the nl argument, suitable for the print element:
There is no imposed order for evaluating arguments. The order is controlled by each element. Most elements, by default, evaluate their arguments in sequential order. However, it is very easy to override the default order by using elements that use a different execution order. For example, the parallel element evaluates all of its arguments in parallel, returning all the resulting values. Evaluating the arguments to an element in parallel then becomes as easy as surrounding them with a parallel element:
Furthermore, the way in which an element processes the arguments is also left to each element. For example, an element can choose to start executing after the evaluation of all arguments has been completed, or process arguments as they arrive. In other words, and in the most general case, arguments are both generated and processed asynchronously.
A concrete example is the print element, which simply returns the message argument on the stdout channel. When Karajan starts execution, an implicit root element is created that receives arguments on the stdout channel, and prints them to the console. Since the processing is done asynchronously, the appearance of print doing the actual work when executed is achieved. The advantage of such a mechanism is that, provided that an element does not produce any side-effects, its execution becomes equivalent to the totality of values returned (both single values, and channels).
We will not insult the reader’s intelligence by explaining what variables are. There is no explicit declaration of variables in Karajan. A variable is defined when it is assigned the first time.
The scope of a variable extends to the element that it was defined in, and is pseudo-lexical. By pseudo-lexical it is meant that internally, the scoping is dynamic, but provisions are made to make it impossible to access variables outside the lexical scope. Therefore, Karajan does not support closures.
On a lexical level, it is possible to read the value of a variable defined in a parent element, but setting the value of the same variable will create a new scope. In other words, Karajan uses deep access and shallow binding. The following example should make things clearer:
In the above example, it would be impossible for the definition of list to access variable v, since the body of the definition of list does not fall within the lexical scope of the definition of v.
The reason for this kind of scoping is to reduce the ambiguity that could be introduced by not knowing the order in which child elements are executed. Please note that such ambiguity is not completely eliminated. The order in which the arguments to list are evaluated does matter, and can change the resulting list. However, the scope of the ambiguity is the same as the scope of the ambiguity in the order of evaluation of the elements. If set, list, and print are evaluated in sequence, no change in the way list evaluates its arguments can change the outcome of the execution of print(v). 1
As mentioned before, closures are not supported. The following example will not work, because at the time inner is evaluated, foo would not be defined in the scope of inner.
In fact, a very simple rule is that no variable can be accessed inside the definition of an element, unless it refers to an argument or a global variable.
Global variables are provided for conveniently defining settings that have a global scope. Please note that in the future, global variables will be single-assignment, this approaching more the notion of constants.
Karajan offers convenient variable expansion constructs. All pairs of curly brackets inside strings are replaced by the value of the variable with the name of the identifier inside the brackets. If no such variable exists, the element trying to access the string will fail. If the ’{’ literal is needed inside a string, it must be used twice. There is no need to escape the closing curly bracket, since it cannot be part of an identifier. If a closing bracket is part of a variable expansion expression, it will mark its end. If not, it will be interpreted as the closing curly bracket literal:
Futures are a mechanism of binding a variable to the results of a future computation. Until the value of the computation to which the future is bound to, the future exists in an unbound state. Any attempt to use the value of an unbound future will cause the execution of the thread that tried to access the future to block until the future becomes bound.
In Karajan there are two types of futures:
As mentioned in Section ??, Karajan understands two syntaxes: the native Karajan syntax and the XML syntax. The distinction between them is made using the file extension. A file with the “.k” extension will be parsed using the native parser, while a file with the “.xml” syntax will be parsed using an XML parser.2
Libraries are collections of elements grouped by the functionality they provide. A library is defined in a source file. Its functionality can be reused in other source files by using the include element:
It is possible to include XML libraries from native Karajan files. It is also possible to include native Karajan libraries from XML Karajan files. Consequently, the following are valid:
Namespaces provide a way of distinguishing between elements with conflicting names in different libraries. Suppose a library “a” defines an element named foo, and a library “b” also defines an element named foo. Also, suppose that both libraries are included in a certain file. Namespaces make it possible to access both instances of the foo definition, without ambiguity, by prefixing the name with the namespace prefix in which the element was defined: a:foo and b:foo. Any reference to foo without a prefix will result in an error. Nonetheless, if only one of the libraries is used, the use of foo without a prefix will be allowed. Namespaces are defined using namespace.
This section lists the current libraries available in Karajan.
The following conventions are used:
Normal arguments are listed in italics: argument
Optional arguments are listed in italics and with an asterisk in front: *optional
The default channel is symbolized by ...
For channels the bold-italics font variant is used: channel
Files: none
Namespace prefix: kernel
The Karajan kernel contains a minimum set of elements that are required in order to get the rest of the system running. All kernel elements are automatically available in any program.
_________________________________________________________________________________________
project(stdout)
Aliases: karajan
The root element of a Karajan program. Accepts arguments on the stdout channel and prints them immediately on the console.
_________________________________________________________________________________________
include(file)
Parses and includes and executes a file inside the current file at the location where include is present. An included file will not be included a second time in any sub-scope of the parent of the current include:
_________________________________________________________________________________________
namespace(prefix)
Allows the specification of a namespace prefix. Any elements defined in the scope of namespace will automatically have the prefix indicated by the prefix argument, unless another namespace is nested.
_________________________________________________________________________________________
elementdef(type , classname)
Used by the current implementation to map element names to Java implementation classes.
_________________________________________________________________________________________
named(value , *name)
Used internally by the named argument form, but can be used by the user equally well. The following are equivalent:
However, the fact that the *name argument is optional makes it impossible to completely avoid the named form.
If the *name argument is not present, it simply returns the value of the value argument on the default channel.
_________________________________________________________________________________________
number(value)
Can be used with the XML syntax to represent a number. There is no distinction between integral numbers and floating point numbers in Karajan.
_________________________________________________________________________________________
string(value)
Can be used with the XML syntax to represent a string value.
_________________________________________________________________________________________
variable(name)
Used to represent the value of a variable.
Example:
will return the list [10, “10”, 5].
_________________________________________________________________________________________
quotedlist(...)
Used internally to represent a quoted list. The following two lines of code will produce the same result:
Files: sys.k, sys.xml
Namespace prefix: sys
The system library contains general purpose elements that help implement the most common tasks in Karajan.
_________________________________________________________________________________________
sequential()
Executes all arguments in sequence.
_________________________________________________________________________________________
parallel()
Executes all arguments in parallel.
_________________________________________________________________________________________
unsynchronized()
Asynchronously executes all arguments in sequence. Does not return any value. If return values from asynchronous computations is required, use either future or futureIterator
_________________________________________________________________________________________
choice()
Executes arguments in succession. Terminates when one of the arguments completes successfully. If an argument fails, the next argument will have access to the following variables:
If no argument completes successfully choice will fail with the last failure encountered.
Choice exhibits a transactional behavior when it comes to return values. All single values and all channels are buffered until one argument completes successfully. If that happens then choice returns the buffered values. If an argument fails, all the buffered values produced by that argument are discarded.
_________________________________________________________________________________________
parallelChoice()
Can be used to race a number of arguments. ParallelChoice will execute all its arguments in parallel, buffer their return values, and wait for the first one that completes. It will then return all the values that the winner generated. If an any argument fails before any other argument completes, then parallelChoice will fail.
_________________________________________________________________________________________
for(name , in)
Can be used to iterate sequentially across a range of values. The name argument is an identifier that indicates the name of the variable that will be set to the successive values of the in argument, which Karajan will try to convert to an iterator before beginning the iteration process.
After evaluating the name and in , for will proceed and evaluate the rest of the arguments repeatedly, while setting the variable indicated by the name argument to each value produced by the iterator (in ).
Example:
will return true
_________________________________________________________________________________________
parallelFor(name , in)
Will behave in a similar way to for with the exception that iterations will occur in parallel. Each iteration will occur in a separate scope. Therefore variables set in one of the iterations will not be visible in the others. Each scope will have the variable indicated by the name argument set to one of the values obtained from the in argument.
_________________________________________________________________________________________
while(condition)
Will repeatedly execute its arguments in sequence until a value of false is received on the condition channel. A convenience element that returns a boolean argument on the condition channel is condition (or ?). While will check for a condition every time an argument completes. It is therefore possible to exit the loop after the termination of any of the arguments.
Example:
will return the following lists:
[1,2,3]
[1]
[]
[0]
Or, in XML:
_________________________________________________________________________________________
break()
Can be used to break out of a while loop. By contrast with using the condition channel, break will immediately exit the loop, not matter how deep the nesting level.
_________________________________________________________________________________________
continue()
Can be used to skip the evaluation of remaining arguments in a while loop and jump to the next iteration.
_________________________________________________________________________________________
if()
Executes its elements using the following scheme:
_________________________________________________________________________________________
then()
Aliases: else
Then and else are the same as sequential, but can be used to make constructs using if more intuitive:
_________________________________________________________________________________________
set(names , ...)
Sets a variable or more to a value (or more). Set tries to interpret the first argument as an identifier or a list of identifiers. If the first argument is an identifier, it is treated as being a list with one identifier. Set expects the number of arguments on the default channel to be the same as the number of identifiers. It is important that a quoted list be used to specify the list of identifiers in order to avoid evaluating the variables that the identifiers represent:
Differences in XML:
The XML variant of the set element uses a slightly different set of arguments. If a single variable is assigned, the name argument can be used. If multiple variables are assigned, the names argument must be used. As opposed to the native syntax, the names argument can be a string of comma separated identifiers which will be tokenized by set
_________________________________________________________________________________________
default(name , value)
Assigns a value to a variable if no binding of that variable can be accessed within the current scope. If an accessible variable with the indicated name is already defined, then the assignment does not take place:
_________________________________________________________________________________________
global(name , value)
Sets a global variable. A global variable is a variable that has a global scope, and thus can be accessed from anywhere within the program. While the use of global variables is discouraged, it can prove useful to create constant-like definitions.
Differences in XML:
The XML variant of global uses the same arguments as the XML variant of the set element.
_________________________________________________________________________________________
...()
Aliases: vargs
Can be used inside an element definition to return all arguments received on the default channel.
Differences in XML:
In the XML syntax the vargs alias must be used, because “...” is not a valid XML element name.
_________________________________________________________________________________________
channel:to(name , ...)
Returns all arguments received on the default channel on the specified channel.
_________________________________________________________________________________________
channel:from(name , ¡varies¿)
Returns all arguments received on the specified channel on the default channel.
_________________________________________________________________________________________
isDefined(name)
Determines whether a variable is accessible within the current scope. Returns true if it is; false otherwise.
_________________________________________________________________________________________
quoted(name)
Returns an identifier without evaluating the variable it might point to.
_________________________________________________________________________________________
discard(...)
Sometimes only the side-effect of an element is needed, while ignoring the return values of the element. Discard will evaluate its arguments, but avoid returning anything on the default channel.
_________________________________________________________________________________________
future(...)
Evaluates the arguments asynchronously. Returns a future representing the first return value generated by the arguments. All other arguments received on the default channel are ignored.
_________________________________________________________________________________________
futureIterator(...)
Evaluates its arguments asynchronously. Returns a future iterator representing all values received on the default channel. The particular aspect of a future iterator is that it can only be iterated over once. Every time a value is used from a future iterator, that value is removed. If access to the iterator values is needed more than once, a list can safely be created with the values. However, the process of creating the list will force synchronization with the thread that produced the values since the iterator is only closed when that thread completes.
_________________________________________________________________________________________
element(name , arguments)
Allows the definition of an element. Evaluates the name and arguments arguments. It expects the name argument to be an identifier, and arguments to be a list of identifiers. The scope of the definition is the same as the scope of a variable that could be defined instead of the element. The arguments are a list of mandatory arguments. Optional arguments can also be specified using the optional element. If the element accepts arguments on the default channel, the ... identifier can be used in the argument list. Other channels can be specified using the channel element. The rest of the arguments are not evaluated when the definition takes place, but will be evaluated whenever the element is invoked.
The following example defines an element foo, which takes no arguments, and prints ’foo’ on the console:
In the following example, foo takes two arguments and prints them both on the screen:
Arguments on the default channel can be accessed using the ... identifier:
Other channels can be used in a similar way:
Optional arguments can be assigned a default value using default:
Element can also be used to define anonymous elements. If the first argument evaluates to a list of identifiers instead of an identifier, element considers that an anonymous element was instead desired, defines the element, and returns the definition, which can later be used through executeElement:
Differences in XML:
The arguments list is a string of comma separated identifiers.
The XML version of element uses a different set of arguments. If an element accepts arguments on the default channel, the vargs=‘‘true’’ attribute must be used. The arguments received on the default channel will then be available in the body of the definition through the vargs identifier.
Optional arguments are indicated using the optargs attribute. The value must be a comma separated list of identifiers.
In a similar way, the channels are specified using a comma separated list of identifiers and the channels attribute.
_________________________________________________________________________________________
parallelElement(name , arguments)
Like element, parallelElement also defines an element. However, elements defined using parallelElement will evaluate their arguments in parallel with their bodies. All single value arguments will automatically be futures, and all channels will automatically be future iterators. ParallelElement can be used to define elements that process their arguments asynchronously.
Differences in XML:
The differences for the XML syntax from element also apply to parallelElement
_________________________________________________________________________________________
channel(...)
Used to specify channel arguments for an element. Quotes all arguments, such that identifiers are not evaluated. Returns values that can be interpreted by element and parallelElement as representing channels.
_________________________________________________________________________________________
optional(...)
Allows the specification of optional arguments for element definitions. Quotes all arguments and returns values that can be interpreted by element and parallelElement as representing optional arguments.
_________________________________________________________________________________________
list(*items , ...)
Constructs a list from values received on the default channel.
Alternatively the *items argument can be used to specify a a string with comma separated list of items. This may be particularly convenient with the XML syntax.
_________________________________________________________________________________________
list:append(list , *items , ...)
Appends all values received on the default channel to the list indicated by the list argument. Does not return anything.
Alternatively, the *items argument could be used with a string of comma separated items.
_________________________________________________________________________________________
list:prepend(list , ...)
Works like list:append with the exception that values are added to the beginning of the list. The order of the values in the list will be the reverse of the order in which they are received by list:prepend:
will print [3, 2, 1, 4, 5, 6]
_________________________________________________________________________________________
list:concat(...)
Concatenates all lists received on the default channel and returns the resulting list.
_________________________________________________________________________________________
list:size(list)
Returns the size of the list indicated by the list argument.
_________________________________________________________________________________________
list:first(list)
Returns the first element in a list.
_________________________________________________________________________________________
list:last(list)
Returns the last element in a list.
_________________________________________________________________________________________
list:butFirst(list)
Returns a list composed of all but the first element in the specified list.
_________________________________________________________________________________________
list:butLast(list)
Returns a list containing all elements but the last from the specified list.
_________________________________________________________________________________________
list:isEmpty(list)
Tests whether a list is empty. Returns true if the list is empty, and false otherwise.
_________________________________________________________________________________________
map(...)
Returns a map with the entries received on the default channel.
_________________________________________________________________________________________
map:entry(key , value)
Allows the specification of an entry that can be used with map to construct a map.
_________________________________________________________________________________________
map:put(map , ...)
Adds the entries received on the default channel to the specified map. Existing entries with the same key are replaced. Does not return any value.
_________________________________________________________________________________________
map:delete(map , key)
Deletes the entry with the specified key from a map. Does not return a value.
_________________________________________________________________________________________
map:get(map , key)
Returns the value corresponding to the specified key from a map.
_________________________________________________________________________________________
map:size(map)
Returns the size of the map (the number of entries in the map).
_________________________________________________________________________________________
map:contains(map , key)
Tests whether the map contains an entry with the specified key.
Logic elements do not at this time use shortcut evaluation. They always evaluate all their arguments.
_________________________________________________________________________________________
and(...)
Aliases: &
Returns the boolean and value of the arguments received on the default channel.
_________________________________________________________________________________________
or(...)
Aliases: |
Returns the boolean or value of the arguments received on the default channel.
_________________________________________________________________________________________
not(value)
Aliases: !
Returns the boolean negation of the value in the value argument.
_________________________________________________________________________________________
equals(value1 , value2)
Aliases: ==
Tests for the equality of two values. Makes a deep comparison of the arguments.
_________________________________________________________________________________________
true()
Returns the true boolean value. There is no syntactic way of specifying a boolean value of true in Karajan.
_________________________________________________________________________________________
false()
Returns the false boolean value.
_________________________________________________________________________________________
sum(...)
Aliases: +
Returns the sum of all the values received on the default channel.
_________________________________________________________________________________________
product(...)
Aliases: *
Returns the product of all the values received on the default channel.
_________________________________________________________________________________________
subtraction(from , value)
Aliases: -
Returns the difference between the value specified by the from argument and the value indicated by the value argument.
_________________________________________________________________________________________
quotient(divisor , dividend)
Aliases: /
Divides divisor by dividend and returns the resulting value.
_________________________________________________________________________________________
remainder(divisor , dividend)
Aliases: %
Returns the remainder of the division of divisor and dividend
_________________________________________________________________________________________
square(value)
Returns the square of a number.
_________________________________________________________________________________________
sqrt(value)
Returns the square root of a number.
_________________________________________________________________________________________
equalsNumeric(value1 , value2)
Makes a numeric comparison of the values specified by the two arguments. A numeric comparison will try to convert string values to numbers before the comparison is performed. Like equals, equalsNumeric performs a deep comparison.
_________________________________________________________________________________________
greaterThan(value1 , value2)
Aliases: >
Returns true if value1 is strictly larger than value2
_________________________________________________________________________________________
lessThan(value1 , value2)
Aliases: <
Returns true if value1 is strictly less than value2
_________________________________________________________________________________________
greaterOrEqual(value1 , value2)
Aliases: >=
Returns true if value1 is larger than or equal to value2
_________________________________________________________________________________________
lessOrEqual(value1 , value2)
Aliases: <=
Returns true if value2 is less than or equal to value2
_________________________________________________________________________________________
ignoreErrors(*match)
Executes its arguments returning any values as they are received. If any of the arguments fails, the failure will be matched against the regular expression in *match if present (otherwise it will be treated as .*). If the match is successful, then the error will be ignored and the next argument will be executed. If the match fails, the error will be propagated.
_________________________________________________________________________________________
restartOnError(times)
Evaluates its arguments in sequence. If a failure occurs, all arguments will be re-evaluated for a maximum of times times.
_________________________________________________________________________________________
generateError(error , *exception)
Allows the generation of an error. The message of the error is taken from error . *Exception is used in the current implementation to pass a Java exception to be attached to the error.
_________________________________________________________________________________________
onError(match)
OnError allows the definition of a custom error handler. Handlers defined with onError are scoped to anything executed within the scope of the parent of onError. Multiple handlers can be defined within the same scope.
Whenever an error occurs within the scope of an error handler, the error will be matched against error handlers starting with the inner-most handlers and ending with the outer-most handlers. If a handler matches, it is invoked. When a handler is invoked it will evaluate all its arguments except for match in sequence. The execution takes place in the context of the failing element. The following variables are defined automatically to be used by the body of the error handler:
Error handlers are not re-entrant. If an unhandled error occurs within the body of the handler, the handler will fail immediately.
_________________________________________________________________________________________
print(message , *nl)
Returns the value in the message argument on the stdout channel. If the *nl argument is not present or set to true, it appends a new line character to the message argument before returning it. The project (root element) automatically prints all argument received on the stdout channel on the console.
_________________________________________________________________________________________
echo(message , *nl , *stream)
Immediately prints the value in the message argument to the console. If the *nl argument is not present or set to true, it also prints a new line character. If the *stream argument is present, it instead tries to print the message on the specified output stream.
The difference between print and echo is that print does not rely on a side-effect to print values. Therefore the evaluation of print cannot be distinguished from returning the value generated by print.
It is recommended that print be used instead of echo if possible.
_________________________________________________________________________________________
checkpoint(*file , *automatic , *interval , *timestamped , *now)
Allows the configuration of checkpointing. If the *now arguments is present and set to true, creates a checkpoint of the state at the time of the evaluation of checkpoint, and writes it to the file indicated by the *file argument.
The *automatic argument, if set to true, indicates that automatic checkpoints should be created at the interval specified by the *interval argument (in seconds). If the *timestamped argument is also present and set to true, the file names in which the checkpoint is saved will have a date and time appended in the YYYYMMDDhhmm form.
_________________________________________________________________________________________
wait(*delay , *until)
When evaluated waits the number of milliseconds specified by the *delay argument or until the date in the *until argument before completing.
_________________________________________________________________________________________
file:execute(file)
Aliases: executeFile
Parses and executes a file. The difference between include and file:execute is that file:execute always parses and executes the file when evaluated, unlike include which only parses the file once. File:execute can therefore be used to execute files which change over time, or to execute different files based on a certain context.
_________________________________________________________________________________________
executeElement(element , args , ...)
Executes an element optionally passing the single value arguments indicated by the args argument, and the ... on the default channel. Args must be a map in which the keys are argument names and the values are argument values.
If args is not present, ... will be mapped to named arguments according to the rules in Section 4.1.3.
_________________________________________________________________________________________
elementList()
Returns a list containing the arguments to elementList but in non-evaluated form. The elements in the resulting list can then be evaluated using executeElement
_________________________________________________________________________________________
cache(on)
Caches all return values of the rest of its arguments based on the value of the on argument. Subsequent evaluations of this element in which the on argument will have the same value will not re-evaluate the rest of the arguments, but return the cached values instead. The cache is bound to this static instance of the cache element. In other words, if another cache element exists, it will not use the values cached by this element, irrespective of the value of the on argument.
Caching is not guaranteed. It is a mechanism that could help improve performance, but it should not be relied on to guarantee that certain elements are only evaluated once. Also, elements that rely on side-effects to perform their function will not be able to perform those functions if their cached valued is used. echo will, for example, not do anything if cached. However, print will, because it does not rely on a side-effect to print values to the console.
_________________________________________________________________________________________
numberFormat(pattern , value)
Allows the formatting of a decimal number. The pattern argument indicates the pattern to be used for formatting (as used by the java.text.DecimalFormat class). The value argument holds the decimal value that is to be formatted.
See also: http://java.sun.com/j2se/1.4.2/docs/api/java/util/Date.html
_________________________________________________________________________________________
file:contains(file , value)
Aliases: contains
File:contains determines whether a file contains a specific sequence of characters. The file argument points to the file to be checked, while the value argument specifies the value to be searched.
_________________________________________________________________________________________
uid(*prefix , *suffix)
Returns a string with a unique ID. The *prefix and *suffix arguments can be used to specify a prefix and a suffix respectively. In the current implementation, the uniqueness of the returned string is relative to the instance of the interpreter.
_________________________________________________________________________________________
file:read(name)
Aliases: readFile
File:read reads the contents of a file, pointed to by the name argument. This is intended for short text files that may possibly hold things like error messages or exit codes. The file is completely read into memory; therefore this element would not be suitable for manipulation of large files.
_________________________________________________________________________________________
outputStream(type , file)
Returns an output stream which can be used for writing values to. The type argument can be one of “stdout”, “stderr”, or “file”. If the type is “file”, the file argument must be present and indicate a valid file name.
_________________________________________________________________________________________
closeStream(stream)
Closes a stream opened with outputStream.
_________________________________________________________________________________________
concat(...)
Concatenates all arguments received on ... and returns the resulting string value.
_________________________________________________________________________________________
split(string , separator)
Returns a list obtained by splitting string in tokens separated by separator . The separator will not be part of the tokens.
_________________________________________________________________________________________
matches(string , regexp)
Returns true if string matches the regular expression specified by regexp , and false otherwise.
_________________________________________________________________________________________
filter(*regexp , *invert , ...)
Filters arguments based on a regular expression, specified by *regexp . If *invert is true, matches will be inverted (values that do not match are returned).
If only one argument is received on the default channel, and that argument is a list, a list is returned with the values of the argument filtered.
If more than one argument is received on the default channel, each argument will be matched against *regexp .
Files: java
Namespace prefix: java
The Java library allows limited interfacing with Java classes and objects from Karajan.
_________________________________________________________________________________________
new(classname , *types , ...)
Instantiates a new Java object and returns it. Classname represents the fully qualified name of the class. The *types argument is a list of fully qualified class names used to search for a constructor. The arguments on the default channel are passed to the constructor after performing a conversion based on the *types argument. The *types argument is not always necessary, but should be used if Karajan cannot determine the types of the arguments that need to be passed to the constructor.
Primitive Java types are represented by their corresponding keywords: “boolean”, “byte”, “char”, “int”, “long”, “float”, and “double”.
_________________________________________________________________________________________
invokeMethod(method , *static , *classname , *object , *types , ...)
Invokes a method on a Java object or a static method on a Java class. The invocation is static if *static is set to true or *classname is present. Otherwise the value of *object is taken to be a Java object and the invocation is done on a virtual method of the object. Method indicates the name of the method. Unless the *types argument is present, Karajan will try to determine the method signature from ... . A method is searched for in the inheritance hierarchy of the object. If one is found, the method is invoked and, if its return value is not void the returned value is returned on the default channel. The format of the type argument is identical to the one for new.
_________________________________________________________________________________________
executeMain(class , ...)
Invokes static void main(String[] args) on a class. The qualified class name should be passed in the class argument. The arguments on the default channel are converted to strings and passed as the args to the main method.
_________________________________________________________________________________________
getField(field , *object , *classname)
Returns the value of a field from an object or class. The filed name is passed in field . If *object is present, the value of the instance field of the object is retrieved. If *classname is present, the class field (static field) of the specified class is retrieved. Arguments *object and *classname are mutually exclusive.
_________________________________________________________________________________________
waitForEvent(...)
WaitForEvent is a rather obscure and incomplete element. It waits for a Java event such as a button being pressed, or a window being closed. Each argument is a list composed by three elements:
When the event occurs, waitForEvent completes returning the return value specified as the first item in the list corresponding to the event that the list represents.
_________________________________________________________________________________________
classOf(object)
Returns the Java class name of the specified object.
_________________________________________________________________________________________
null()
Karajan has no explicit null value. However, it may be necessary that a null value be used when invoking Java methods. Null provides that.
Files: task
Namespace prefix: task
The task library interfaces with the Java CoG Kit abstraction classes, allowing the use of services for job submission and file operations. The tasks in this library can function in two modes: scheduled or unscheduled. When scheduled, remote tasks are not executed directly. They are rather passed to a scheduler which can handle issues such as throttling, resource allocation, and task-to-resource mapping.
_________________________________________________________________________________________
scheduler(type , resources , handlers , *properties)
Defines the a scheduler to be used. The scope of the scheduler is the scope of the execution of the parent of scope. The type describes the particular type of scheduler that is desired. Currently only “default” is supported. The resources that can be used by the scheduler are passed in the resources argument and can be defined using resources. Each scheduler will also require a list of task handlers, specified using handlers with the help of the handler element. Each type a scheduler may support an optional set of properties. The *properties argument, if specified, must be a map containing string keys and values.
The “default” scheduler uses a round-robin scheduling policy. However it also performs lookahead matching. This means that if a certain host has reached its maximum allowable number of tasks, it will be skipped. Also, if a suitable host is not found for the next task in the queue, other tasks may be scheduled. The scheduler will use the handlers in the order they were specified in the handlers list, with the first handler having the highest priority.
The default scheduler supports the following properties:
_________________________________________________________________________________________
handler(type , provider)
A handler specifies a Java CoG Kit Abstraction handler. A handler is used to submit tasks. Type indicates the type of handler. They type is string and can have one of the following values: “execution”, “file”, and “file-transfer”.
Execution handlers are used for submitting jobs. File handlers are used for file operations (such as renaming, deleting, and listing of files). File transfer handlers are used only for transferring files. It is possible to transfer files using file handlers, but it is not possible to delete a file using a file transfer handler.
The provider argument indicates the provider to be used for the handler. For a list of currently supported providers please see the abstractions guide.
_________________________________________________________________________________________
resources(...)
Encapsulates a set of hosts which can be specified using host.
_________________________________________________________________________________________
host(name , *cpus , ...)
Returns a host definition. The name argument indicates the host name or IP address. The number of CPUs of the host can be specified using the *cpus argument. A set of services can also be specified on the default channel.
_________________________________________________________________________________________
service(type , provider , *uri , *project , )
Returns a service definition. The type of the service can be one of “execution”, “file”, or “file-transfer”. Provider indicates The Java CoG Kit abstraction provider for the service. For a list of currently supported providers please see the abstractions guide.
The *uri argument can be used to specify a URI for the service. If missing the host name of the host containing the service will be used.
The project argument can be used to automatically bind a queuing system project to the service in order to alleviate the need to do it with the task:execute element.
_________________________________________________________________________________________
securityContext(provider , ...)
Returns a Java CoG Kit abstraction security context. The returned context will be instantiated for the specified provider. The default channel can be used to specify additional properties for
_________________________________________________________________________________________
securityContextProperty(name , value)
Defines a property for a security context.
_________________________________________________________________________________________
allocateHost(name)
Allows tasks to be grouped on one host. By default, the scheduler assigns a different host to each task. AllocateHost can be used to reserve a host from the scheduler until it completes. The name indicates the name of the variable to be set with the allocated host, and is automatically quoted.
Or, in XML:
The default scheduler uses a late binding mechanism with allocateHost. It generates a virtual host that is only bound to an actual host when the first task using it is submitted to the scheduler. This removes the limitation on the number of parallel allocateHost that can be running, and allows contained jobs to be submitted to the scheduler, which will later handle the throttling issues.
Multiple allocateHost can be nested allowing the grouping of tasks on multiple dependent hosts.
_________________________________________________________________________________________
host:hasService(host , type , provider)
Checks if a host, specified with the host element contains a service of the specified type and with the specified provider . Returns true if such a service exists, and false otherwise.
_________________________________________________________________________________________
execute(executable , *arguments , *directory , *stdout *stderr , *stdin , *redirect , *provider , *host ,
*count , *jobtype , *maxtime , *maxwalltime , *maxcputime , *environment , *queue , *project , *minmemory ,
*maxmemory)
Executes a remote job. executable indicates the executable to be run. Arguments can be passed to the executable using *arguments . If present, the *directory argument specifies the remote directory in which the job will be executed. *Stdout and *stderr allow the redirection of the output and error streams to a remote file. *Stdin allows the redirection of the standard input from a remote file. If *redirect is set to true the standard output and standard error of the remote job is redirected to the local console. The *host argument allows the job to be executed on a specific host, and the *provider argument allows the job to be executed using a specific provider.
The rest of the arguments are passed to the underlying provider.
_________________________________________________________________________________________
transfer(*srcfile , *srcdir , *srchost , *destfile , *destdir , *desthost , *provider)
Transfers a file. The file can be transfered between the local machine and a remote machine, or between two remote machines. The name of the source file is specified by the *srcfile argument. If present, *destfile specifies the name of the target file, otherwise the source file name is used.
The *srcdir argument indicates the directory on the source machine where the source file can be found. If the *srcdir argument is missing, the default directory will be assumed (provider dependent).
The *destdir argument indicates the directory on the target machine where the file will be copied. If the *destdir argument is missing, the default directory will be assumed (provider dependent).
*Srchost and *desthost indicate the source and the target machines respectively, while the *provider argument can be used to force the scheduler to use a specific provider, or in the event a scheduler is not used.
_________________________________________________________________________________________
file:list(dir , *host , *provider)
Returns a list of files in a directory specified by dir , on the *host machine. The *provider argument can be used to select a specific provider for the operation. *Provider defaults to the local provider.
_________________________________________________________________________________________
file:remove(name , *host , *provider)
Removes a file specified by name , on the *host machine. The *provider argument can be used to select a specific provider for the operation. *Provider defaults to the local provider.
_________________________________________________________________________________________
file:exists(name , *host , *provider)
Returns true if the file specified by name exists on the *host machine. The *provider argument can be used to select a specific provider for the operation. *Provider defaults to the local provider.
_________________________________________________________________________________________
dir:make(name , *host , *provider)
Creates a directory specified by name , on the *host machine. The *provider argument can be used to select a specific provider for the operation. *Provider defaults to the local provider.
_________________________________________________________________________________________
dir:remove(name , *host , *provider)
Removes an empty directory.
_________________________________________________________________________________________
file:isDirectory(name , *host , *provider)
Returns true if the file specified by name exists on the *host machine and it is a directory. The *provider argument can be used to select a specific provider for the operation. *Provider defaults to the local provider.
_________________________________________________________________________________________
file:chmod(name , mode , *host , *provider)
Changes the permissions on the file specified with the name argument to the mode string indicated by the mode argument. If *host and *provider are present, the operation is done remotely using the respective provider.
_________________________________________________________________________________________
file:rename(from , to , *host , *provider)
Renames a file. The source and target name are specified using the from and to arguments. If *host and *provider are present, the operation is done remotely.
[1] G. von Laszewski, I. Foster, J. Gawor, and P. Lane, “A Java Commodity Grid Kit,” Concurrency and Computation: Practice and Experience, vol. 13, no. 8-9, pp. 643–662, 2001. [Online]. Available: http://www.mcs.anl.gov/~gregor/papers/vonLaszewski--cog-cpe-final.pdf
[2] “Java CoG Kit Wiki,” 2004. [Online]. Available: http://www.cogkit.org/wiki
[3] “Java CoG Kit Registration,” 2004. [Online]. Available: http://www.cogkit.org/register
Additional publications about the Java CoG Kit can be found as part of the vita of Gregor von Laszewski http://www-unix.mcs.anl.gov/~laszewsk/vita.pdf. Most documents are avalable online if you follow the links. In future we intend to provide this information without Gregors vita data.
If you need to cite the Java CoG Kit, please use [1].
The newest version of this document can be downloaded by the developers from
It is not allowed to reproduce this document or the source. This documentation is copyrighted and is not distributed under the CoG Kit license.
This guide is constantly improved and your input is highly appreciated. Please report suggestion, errors, changes, and new sections or chapters through our Bugzilla system at
http://www-unix.globus.org/cog/contact/bugs/
The Java CoG Kit project has been initiated and is managed by Gregor von Laszewski. To contact him, please use the information below.
Gregor von Laszewski
Argonne National Laboratory
Mathematics and Computer Science Division
9700 South Cass Avenue
Argonne, IL 60439
Phone:(630) 252 0472
Fax: (630) 252 1997
gregor@mcs.anl.gov
Administrative Contact, 40
allocateHost, 37
and, 28
break, 19
cache, 32
channel, 22, 25
channel:from, 21
channel:to, 21
checkpoint, 31
choice, 16
classOf, 35
closeStream, 33
concat, 33
condition, 17
Contact, 40
continue, 19
default, 21, 23
dir:make, 39
dir:remove, 39
discard, 22
echo, 31, 32
element, 21, 22, 24, 25
elementdef, 14
elementList, 32
equals, 28, 29
equalsNumeric, 29
execute, 38
executeElement, 23, 31, 32
executeMain, 35
false, 28
file:chmod, 39
file:contains, 33
file:execute, 32
file:exists, 39
file:isDirectory, 39
file:list, 38
File:read, 33
file:read, 33
file:remove, 38
file:rename, 39
filter, 34
for, 16, 17
future, 12, 16, 22
futureIterator, 13, 16, 22
generateError, 31
getField, 35
global, 21
greaterOrEqual, 30
greaterThan, 30
Grid
Scripting, 3
handler, 36
host, 36, 38
host:hasService, 38
if, 19
ignoreErrors, 30
include, 13, 14, 32
invokeMethod, 34
isDefined, 21
Karajan, 3
Methods, 28
*, 28
+, 28
-, 29
/, 29
¡, 30
¡=, 30
==, 28
¿, 30
¿=, 30
contains, 33
else, 19
executeFile, 32
karajan, 14
readFile, 33
vargs, 21
karajan, 6
lessOrEqual, 30
lessThan, 30
list, 7, 11, 26
list:append, 26
list:butFirst, 27
list:butLast, 27
list:concat, 26
list:first, 26
list:isEmpty, 27
list:last, 26
list:prepend, 26
list:size, 26
map, 27, 32
map:contains, 27
map:delete, 27
map:entry, 27
map:get, 27
map:put, 27
map:size, 27
matches, 34
named, 14
namespace, 13, 14
new, 34, 35
not, 28
null, 35
number, 6, 15
numberFormat, 33
onError, 31
optional, 22, 25
or, 28
outputStream, 33
parallel, 10, 16
parallelChoice, 16
parallelElement, 24, 25
parallelFor, 17
print, 6, 9–11, 31, 32
print(v), 11
product, 28
project, 6, 14, 31
quoted, 22
quotedlist, 15
quotient, 29
Registration, 2
remainder, 29
resources, 36
restartOnError, 30
scheduler, 36
securityContext, 37
securityContextProperty, 37
sequential, 16, 19
service, 37
set, 11, 20, 21
split, 33
sqrt, 29
square, 29
string, 6, 15
subtraction, 29
sum, 28
task:execute, 37
then, 19
transfer, 38
true, 28
uid, 33
unsynchronized, 16
variable, 15
wait, 32
waitForEvent, 35
while, 17, 19
Workflow, 3
Karajan, 3