Developing Origin Applications

The Origin SDK providing a rich set of classes that you can use to develop your own evolutionary computation programs. Your application can build on the functionality provided by the SDK by subclassing and extending Origin classes, or by writing classes that implement the standard interfaces defined in the SDK. Since Origin's behavior is configured at runtime you should normally never have to modify the Origin source code, though you may find it useful to use it as a template for your own work.

The Origin class hierarchy is similar to that provided by ECJ, and the ECJ Tutorials are a good starting point for learning how to develop your own evolutionary computation application. Most ECJ applications will run with little modification under Origin; Converting ECJ Applications to Origin explains this in more detail.

Restrictions on Frontier Grid Tasks

Origin Frontier Grid tasks run in a restricted JVM "sandbox" similar to the sandbox used by Java applets. Code running in the sandbox has extremely limited access to the local system: in particular it cannot access the file system or open network connections. Classes in your application such as fitness evaluators that will run remotely on the grid must follow these rules:

Files cannot be directly accessed. Read input files by calling state.output.getInputStream() to obtain an open input stream instead of opening the File object returned from getFile().
Output must be done using either the Log4J API or via the IOutput object in state.output. Note that remote task output is discarded.
Individual and Fitness classes must implement java.io.Serializable.
Don't call System.exit() to terminate the program in case of fatal error; throw an unchecked exception such as java.lang.Error instead.

Compiling Origin Programs
The Origin SDK and third-party jars in the lib directory must be on your compilation classpath. Don't use different versions of these jars - the included versions are the ones Origin has been tested with.

Classes used in your Origin run must be packaged into jar files, and these jars must be included on the Origin runtime classpath. If you have a lot of code that's only used by the local component of your application you should package this in a separate jar to reduce the amount of code that needs to be sent to the Frontier Compute Engines.

The Sample Application Directory

Origin provides a sample application directory skeleton and ant build files in the sample-application directory. You can copy this directory and use it as a starting point for developing your own applications. The directory contents are:

src/: Your application's java source code.
lib/: Any third-party jars required by your application.
conf/: Configuration files used by your application at runtime.
conf/params/: Origin parameter files for your application.
bin/: Shell scripts and batch files used by your application.
docs/: Documentation for your application.
setup/: Ant build configuration files.

After copying this directory edit setup/setup.xml and change the value of origin.dir (at the top of the file) to the location of the Origin directory.

When you run ant in this directory the Ant build files will compile all source code in the src directory, construct a task and application jar, and copy the jars, libraries, configuration files, and documentation to dist/projectname.

Application Jar Files and Frontier Job Manifest

If your tasks uses additional jars such as third-party components Origin needs to send these jars to the Frontier Grid Server for distribution to the Compute Engines. In most cases Origin will automatically identify these jars and send them, but sometimes Origin needs additional information about the jars required by your application. Origin uses the Job Manifest entry in your application's jar file to locate these jars. If you use the ant build files in the sample application directory this manifest will be created automatically and added to the application jar when it's built.

The Frontier Job Manifest is an XML file listing required jars and their location in the local filesystem. The job manifest has the form:

<?xml version="1.0" encoding="UTF-8"?>
<job-manifest name="appname">
    <executable-element name="jarname" file="jarpath"/>
    <executable-element .../>
	:
</job-manifest>

Each jar has an executable-element element giving the jar a unique name and specifying the jar file pathname. If a pathname is relative and doesn't exist Origin will search the runtime classpath for the jar. Note that you should only include jars required by remote task code: jars used only locally (GUI libraries, database drivers, etc.) should not be listed. The job manifest must be stored as PARABON-META/JOB-MANIFEST in your application's jar file.

If you're using Ant you can use the Frontier SDK jobmanifest Ant task to generate the Frontier job manifest. First define the Frontier SDK Ant task definitions (if your build files include the setup/setup.xml and setup/common.xml files from the Origin source distribution this is done for you):

<taskdef description="Frontier SDK ant tasks"
         resource="com/parabon/client/antlib.xml"
         classpathref="classpath-containing-frontier-sdk.jar"
/>

Generate the manifest like this (you'll need to add it to the application jar file afterwards):

<jobmanifest jobname="application-name"
             destfile="${build.dir}/PARABON-META/JOB-MANIFEST"
             listfiles="true"
>
        <classpath>
            <path refid="jars-used-by-tasks"/>
        </classpath>
        <excludepath>
            <path refid="jars-to-exclude"/>
        </excludepath>
</jobmanifest>

This will generate a manifest containing all jars on the classpath jars-used-by-tasks except for those also on jars-to-exclude. One advantage of using the jobmanifest Ant task is that if you add or remove jars from the compilation classpath in the Ant build file these changes will automatically be reflected in the Frontier job manifest the next time the manifest is generated.

Class and Input File Parameters

When Origin runs a Frontier Grid enabled evolutionary model it scans the parameter database for parameters referencing classes and data files that are required by Origin tasks and packages these classes and files with the task. If a parameter references a class the class' jar file is located and the jar file and any additional jars listed in the jar's Frontier job manifest are included with the Origin task.

When a parameter specifies a file pathname and the file exists Origin converts the file to a Frontier data element and sends the element with the task. The remote task can read the data element by calling getInputStream(). Since the parameter database is scanned only once when Origin starts any changes made locally to the data files during an evolutionary run will not be visible to remote tasks.

Origin uses a parameter naming scheme to identify class and input file parameters. If parametername.slave-class is set to true then parametername is a class parameter, and the jar containing this class will be included with remote tasks. Any parameter with a name ending in .file is treated as a file parameter unless parametername.slave-file is also defined; if slave-file is true parametername is a file parameter regardless of its name, and if it's set to false parametername is not a file parameter. even if it ends in .file.

The standard set of class and file parameters is defined in origin/conf/params/com/parabon/ec/origin.params.

Converting ECJ Applications to Origin

Because Origin is based on ECJ many ECJ applications will run under Origin without modification. There are a few areas where Origin is incompatible with ECJ:

Origin supports only Java 5 and later Java implementations. Code incompatible with Java 5 must be modified.
Origin uses the ec.util.IOutput and ec.util.IParameterDatabase interfaces instead of the ec.util.Output and ec.util.ParameterDatabase classes. Since these interfaces implement the same API as the corresponding classes this change only affects programs that either subclass Output or ParameterDatabase or directly access fields in these classes.
The Origin log message format is different from that used by ECJ and prior announcements are not output when an Origin run is restarted from a checkpoint.
Origin and ECJ checkpoint files are not compatible (you can't restart Origin from an ECJ checkpoint).

Debugging Origin Programs

Debugging an Origin application running on the Frontier grid may seem difficult since you have little visibility into the behavior of grid tasks other than their returned results- or fatal exception messages. These are a some helpful methods you can use to debug your application:

Increase the Log4J logging level as described in The Command Line Program. While remote task logging information is not returned to the Origin application task status information and fatal exceptions are logged at high logging levels.
Verify that your application's evolutionary model is correct by running it locally instead of using the Origin grid models. Local applications can be debugged using the usual methods for program debugging: running the program in a debugger, adding log messages, etc.

Some common errors encountered in Origin applications are described below:

Errors about a class not being loaded from a jar.: This is usually caused when the directory containing the .class file for a class occurs in the classpath before the application jar file. You should include only jar files in the classpath used to run Origin.
Class file version errors from Frontier tasks.: The default class file format for Java 6 and later Java versions is incompatible with the Java 5 JVMs used by some Frontier Compute Engines. When compiling your code you must specify the -source 1.5 and -target 1.5 flags to javac or set the equivalent options in your IDE's project settings (for Eclipse select Project → Properties → Java Compiler and set Compiler Compliance Level to 5.0). The Origin Ant build.xml files set these flags automatically.
"Class not found" errors in Frontier tasks.: This can be caused either because the parameter referencing this class was not identified as a class parameter or because the jar containing the class was not included in the application jar's Frontier Job Manifest. If the class is specified in the parameter file define parametername.slave-class = true for the class' parameter as described in Class and Input File Parameters. If the class is in a separate jar add this jar to the applications Frontier Job Manifest and rebuild the application's jar.
Errors about jar files not being found when processing the Frontier job manifest.: Origin looks for jar files listed in the job manifest first at the pathname given in the manifest, and then if this is a relative pathname searches the classpath for the jar. Check that relative paths are used for the jars in the job manifest and that all listed jars are on the Origin runtime classpath.
"Out of memory" errors.: You can increase the amount of memory available to the local Origin application using the -Xmxmemorysize Origin command line option. All -X and -XX command line options are passed through to the JVM used to run Origin.
If Frontier tasks are failing with out of memory errors reduce the eval.masterproblem.max-data-per-slave or eval.masterproblem.max-jobs-per-slave parameters.
Security exceptions in Frontier tasks.: This indicates that your code - or a third-party library used by your code - is violating the rules described in Restrictions on Frontier Grid Tasks. If the cause of the security violation is not obvious from the logged exception stack trace try running the application locally and tracing its operation using a debugger or profiler.
One less than obvious cause of security violations is caused by parsing an XML document with a DTD reference containing a URL, e.g. DOCTYPE ... SYSTEM "http:some-url.dtd". Some XML parsers will attempt to read this URL when parsing the document, violating the sandbox restrictions on network access. You can often resolve this by parsing the document in non-validating mode.
Different results each time the program is run.: Make sure the parameter seed.0 is set to a constant value rather than time (which seeds the random number generator from the system time). If your fitness evaluation methods perform complex floating-point calculations you may need to use the strictfp modifier to produce consistent results on different JVM implementations.