Simulation Wiki Tutorial - Batchsubmit Fundamentals
This tutorial describes how to submit a job to the high performance computing capabilities available at the NEEShub using the batchsubmit command. This tutorial is not intended to be advanced and only describes the general idea of batchsubmit and how to use it. You should view the batchsubmit resource pages for advanced uses of this command. A video of this tutorial is also available at http://youtu.be/6nqffB7ATOU
You must be logged in to NEEShub and have access to the workspace. Basic knowledge about high performance computing and/or environments to submit jobs (i.e. condor) is not required to complete the tutorial.
Step 1 – Upload simulations to your home directory
The examples in the tutorial will be executed from your home directory. The output can be found in the scratch space. The scratch space is a large directory specifically designed to store temporary files from computer simulations. A script keeps enough space for new simulations by deleting files from old simulations. The scratch space is not regularly backed up and should not be used for long term storage.
For this tutorial we will be using an example of a pushover analysis available for download at the OpenSEES wiki (http://opensees.berkeley.edu/wiki/index.php/Pushover_Analysis_of_2-Story_Moment_Frame). Download the .zip file to your local machine, uncompress it, and upload the files to your home directory on the NEEShub.
The recommended tool to upload files is SynchroNEES. For more information refer to SynchroNEES wiki. Use SynchroNEES to create a new directory within your home directory and upload your simulation files there.
If you are unsure of how to upload your files to the home directory, step 1 of the OpenSees tutorial will walk you through the process.
Step 2 – Launch workspace
Search for the “workspace” tool. You can find it at the “Tools” section in the “Tools & Resources” blue menu bar option. You can also simply search for “workspace” in the search box.
Once you have found the workspace tool page press the “Launch” button. If you do not have permission to launch the workspace, please place a ticket by pressing the “Support” button and writing a short message to request access to the workspace.
You can view the directories available in your home directory by typing “ls”. If you want more detailed information about the files in your home directory, you can use the “ls –l” command.
Now, change to the directory where you have uploaded your simulation files using the “cd” command. I uploaded my files to the “pushover” directory so I will use the “cd pushover” command.
Step 3 – Running batchsubmit
Now that we are in the correct directory, we can run the simulation (or better yet, submit the simulation) to one of the high performance computing venues available for NEEShub users. Basically, what is going to happen is that you are going to submit the simulation to a queue. Once it is your turn, your simulation will start running on the appropriate machine. The results will come back to the scratch space when the simulation is completed. The advantage of using this method is that many users can take advantage of the same computational facilities.
You could also send many jobs without waiting for some of the simulations finish. Batchsubmit will take care of scheduling the job and returning the results when done. By default, results are stored in $HOME/scratch/$PROGRAM directory. $PROGRAM is the application that you submitted. If you submitted an OpenSEES job, the results will be found inside $HOME/scratch/opensees. You can change to this directory in the workspace in one single command by using “cd ~/scratch/opensees. You can also access this directory using SynchroNEES.
Note that many of the batchsubmit options are prefixed by 2 consecutive dashes (—). At any time in the workspace, you can enter the command below to view batchsubmit help and see details about the available parameters.
batchsubmit –h | more
We need to specify a minimum of three parameters if we want to use batchsubmit with OpenSEES:
1. On what venue would you like the job to run? Think of the venue as the “service” or “cluster of machines” that you would like your service to run. Some options are “kraken or ranger” (XCEDE cluster), “steele or hansen” (Purdue Cluster) and “osg” (for Open Science Grid).
2. What directory has the executable files of OpenSEES that are specifically compiled for this cluster? This file will be sent to the cluster to run your file
3. What command line needs to be executed to run your simulation?
These three items are specified in the following command line:
batchsubmit —venue osg —infilearg 1 —appdir /apps/openseesbuild/osg OpenSees pushover_concentrated.tcl
This command line must be entered EXACTLY as shown (including capitalization) in order for it to work correctly.
The command line that you would have to used to run the simulation “locally” is “opensees pushover_concentrated.tcl”. Since we have one argument (the name of the tcl file – “pushover_concentrated.tcl”) we need to specify the “—infilearg 1” option. The rest of the parameters are the venue (“—venue”) and the directory of the compiled version of OpenSEES for this particular venue (“—appdir /apps/openseesbuild/osg”). In this particular example we are using the Open Science Grid to run our simulation.
When you successfully submit your job, you will see a message such as the one shown in the figure below. Note the jobXXXX number in the /home/neeshub/rhowell/scratch/opensees/job00095/pushover line. This is the job number under which your simulation results will be stored in the scratch space.
Step 4 – What is the state of my simulation?
To look at the state of your simulation type the “batchstatus” command. This will list the jobs that you have run, the current status of each job, and the batchsubmit command associated with each job. Use “batchstatus | grep keyword” to filter the results on a given keyword.
Step 5 – Locating the job output directory
You can use the workspace to go to the location where the job files are stored. For example, in this particular example my job is “job00095” and the output files are located at “/scratch/rhowell/opensees/job00095”. Use the “cd ..” command to return to your main home directory, and then you can change to your scratch space job directory with the command “cd /scratch/rhowell/opensees/job00095” (Hint: You don’t like typing long path names? – try typing the first few letters and then pressing the “tab” key. It will complete the name for you or give you different options if there are more than one files or directories available). Use the “ls” or “ls –la” command to see the list of files.
Step 6 – Downloading results from simulation
Within the job directory, your output files will be written to the directory with the same name as the directory where your input files were stored in the home directory. For example, my input files were uploaded to the pushover directory. Therefore, the job00095/pushover directory will contain my output files.
You can download the files of your simulation using SynchroNEES