Makeflow is Copyright (C) 2009 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.
You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool or a Sun Grid Engine batch system, then you can send your jobs there to run. If you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.
Makeflow is part of the Cooperating Computing Tools. You can download the CCTools from this web page, follow the installation instructions, and you are ready to go.
Makeflow attempts to generate all of the target files in a script. It examines all of the rules and determines which rules must run before others. Where possible, it runs commands in parallel to reduce the execution time.
Here is a Makeflow that uses the convert utility to make an animation. It downloads an image from the web, creates four variations of the image, and then combines them back together into an animation. The first and the last task are marked as LOCAL to force them to run on the controlling machine.
CURL=/usr/bin/curl CONVERT=/usr/bin/convert URL=http://www.cse.nd.edu/~ccl/images/capitol.jpg capitol.montage.gif: capitol.jpg capitol.90.jpg capitol.180.jpg capitol.270.jpg capitol.360.jpg LOCAL $CONVERT -delay 10 -loop 0 capitol.jpg capitol.90.jpg capitol.180.jpg capitol.270.jpg capitol.360.jpg capitol.270.jpg capitol.180.jpg capitol.90.jpg capitol.montage.gif capitol.90.jpg: capitol.jpg $CONVERT $CONVERT -swirl 90 capitol.jpg capitol.90.jpg capitol.180.jpg: capitol.jpg $CONVERT $CONVERT -swirl 180 capitol.jpg capitol.180.jpg capitol.270.jpg: capitol.jpg $CONVERT $CONVERT -swirl 270 capitol.jpg capitol.270.jpg capitol.360.jpg: capitol.jpg $CONVERT $CONVERT -swirl 360 capitol.jpg capitol.360.jpg capitol.jpg: $CURL LOCAL $CURL -o capitol.jpg $URLNote that Makeflow differs from Make in a few important ways. Read section 4 below to get all of the details.
% makeflow example.makeflowNote that if you run it a second time, nothing will happen, because all of the files are built:
% makeflow example.makeflow makeflow: nothing left to doUse the -c option to clean everything up before trying it again:
% makeflow -c example.makeflowIf you have access to a batch system running SGE, then you can direct Makeflow to run your jobs there:
% makeflow -T sge example.makeflowOr, if you have a Condor Pool, then you can direct Makeflow to run your jobs there:
% makeflow -T condor example.makeflowTo submit Makeflow as a Condor job that submits more Condor jobs:
% condor_submit_makeflow example.makeflowYou will notice that a workflow can run very slowly if you submit each batch job to SGE or Condor, because it typically takes 30 seconds or so to start each batch job running. To get around this limitation, we provide the Work Queue system. This allows Makeflow to function as a master process that quickly dispatches work to remote worker processes.
To begin, let's assume that you are logged into a machine named barney.nd.edu. start your Makeflow like this:
% makeflow -T wq example.makeflowThen, submit 10 worker processes to Condor like this:
% condor_submit_workers barney.nd.edu 9123 10 Submitting job(s).......... Logging submit event(s).......... 10 job(s) submitted to cluster 298.Or, submit 10 worker processes to SGE like this:
% sge_submit_workers barney.nd.edu 9123 10Or, you can start workers manually on any other machine you can log into:
% work_queue_worker barney.nd.edu 9123Once the workers begin running, Makeflow will dispatch multiple tasks to each one very quickly. If a worker should fail, Makeflow will retry the work elsewhere, so it is safe to submit many workers to an unreliable system.
When the Makeflow completes, your workers will still be available, so you can either run another Makeflow with the same workers, remove them from the batch system, or wait for them to expire. If you do nothing for 15 minutes, they will automatically exit.
Note that condor_submit_workers and sge_submit_workers are simple shell scripts, so you can edit them directly if you would like to change batch options or other details.
# This is an incorrect rule. output.txt: ./mysim.exe -c calib.data -o output.txtHowever, the following is correct, because the rule states all of the files needed to run the simulation. Makeflow will use this information to construct a batch job that consists of mysim.exe and calib.data and uses it to produce output.txt:
# This is a correct rule. output.txt: mysim.exe calib.data ./mysim.exe -c calib.data -o output.txtWhen a regular file is specified as an input file, it means the command relies on the contents of that file. When a directory is specified as an input file, however, it could mean one of two things. First, the command depends on the contents inside the directory. Second, the command relies on the existence of the directory (for example, you just want to add more things into the directory later, it does not matter what's already in it). Makeflow assumes that an input directory indicates that the command relies on the directory's existence.
When using Condor, this string will be added to each submit file. For example, if you want to add Requirements and Rank lines to your Condor submit files, add this to your Makeflow:
BATCH_OPTIONS = Requirements = (Memory>1024)
When using SGE, the string will be added to the qsub options. For example, to specify that jobs should be submitted to the devel queue:
BATCH_OPTIONS = -q devel
% makeflow -D example.makeflow | dot -T gif > example.gif
% makeflow -T wq -p 9567 exmaple.makeflow
% makeflow -T wq -N proj1 -e example.makeflowThis would inform the master to only allow connections from workers who have a preference on "proj1". If the '-e' option is not specified, the master would accept shared workers submitted by anyone. To add a project preference for the worker, add the '-N' option to the worker as well:
% work_queue_worker -N proj1 barney.nd.edu 9123You can even assign mulitple preferred project names to a single worker. The worker would only work for the master whose name is contained in the list of the worker's preferred project names.
To make use of the catalog mode, you will have to have a catalog server to pass information between potential masters and workers. Say you want to run your catalog server on a machine named barney.nd.edu (the default port that the catalog server will be listening on is 9097, you can change it via the '-p' option), do:
barney% catalog_serverNow you have a catalog server listening at barney.nd.edu:9097. To make your masters and workers contact this catalog server, add the '-C hostname:port' option to both the master and worker programs:
% makeflow -T wq -C barney.nd.edu:9097 -N proj1 example.makeflow % work_queue_worker -C barney.nd.edu:9097 -sThe '-C' option turns on the catalog mode. Note that the '-s' option in the above worker command tells the worker to run in a shared worker mode, as opposed to the default exlusive mode which would let the worker only work for masters that it prefers (set by the -N option).
At Notre Dame, we have already had a catalog server runing 24/7. The work queue master and worker program would contact the Notre Dame catalog server by default. Thus, you don't have to specify the catalog server explicitly with the '-C' option unless you have your own catalog server. To turn on the catalog mode in this case you will use the '-a' option, again on both the master and worker programs.
% makeflow -T wq -a -N proj1 example.makeflow % work_queue_worker -a -N proj1 -N proj2 -N proj3Note that this time the worker is not run with the '-s' option. Instead, it has three preferred projects. The worker would ask the catalog server for available masters and only work for a master if its project name matches one of the worker's preferences.