Batchsubmit Tutorial For OpenSEES Day 2012


This tutorial describes how to submit a job to the high performance computing capabilities available at the NEEShub using the batchsubmit command. This tutorial is not intended to be advanced and only describe the general idea of batchsubmit and how to use it. You should view the batchsubmit resource pages for advanced uses of this command. A video of this tutorial is also available at


You must be logged in to NEEShub and have access to the workspace. Basic knowledge about high performance computing and/or environments to submit jobs (i.e. condor) is not required to complete the tutorial.

Step 1 – Upload simulations to your home directory

The examples in the tutorial will be executed from your home directory. The output can be found in the scratch space. The scratch space is a large directory specifically designed to store temporary files from computer simulations. A script keeps enough space for new simulations by deleting files from old simulations. The scratch space is not regularly backed up and should not be used for long term storage.

For this tutorial we will be using an example of a sine wave. Download to your local machine, uncompress it and upload the files to your home directory at /home/neeshub. The recommended tool to upload files is SynchroNEES. For more information refer to SynchroNEES wiki.

Step 2 – Launch workspace

Workspace can be launched from here. If you do not have permission to launch the workspace, please place a ticket by pressing the “Support” button and writing a short message to request access to the workspace.

You can take a look at the directories available in your scratch space by typing “ls”. I like to use the command “ls –l” because it gives me more information about the files in that directory.


Now, change to the directory where you uploaded your simulation files using the “cd” (change directory) command. I uploaded my files to the ““batchsubmit_examples” directory so I will use the “cd batchsubmit_examples” command.

Step 3 – Running batchsubmit

Now that we are in the correct directory, we can run the simulation (or better yet, submit the simulation) to one of the high performance computing venues available for NEEShub users. Basically, what is going to happen is that you are going to submit the file to a queue. Once it is your turn, your simulation will start running on the appropriate machine. The results will come back to the scratch space once it finishes running. The advantage of using this method is that many users can take advantage of the same computational facilities.

You could also send many jobs without waiting for some of the simulations finish. Batchsubmit will take care of scheduling the job and returning the results when done. By default, results are stored in $HOME/scratch/$PROGRAM directory. $PROGRAM is the application that you submitted. If you submitted an OpenSEES job, the results will be found inside $HOME/scratch/opensees. Also, you can change to this directory in one single command by using “cd ~/scratch/opensees.

Note that many of the batchsubmit options are prefixed by 2 consecutive dashes (—).

We need to specify three parameters if we want to use batchsubmit with OpenSEES:

1. On what venue would you like the job to run? Think of the venue as the “service” or “cluster of machines” that you would like your service to run. Some options are “kraken or ranger” (XCEDE cluster), “steele or hansen” (Purdue Cluster), and “osg” (for Open Science Grid).

2. What directory has the executable files of OpenSEES that are specifically compiled for this cluster? This file will be sent to the cluster to run your file.

3. What command line needs to be executed to run your simulation?

Exercise 1: Execute command to get help and see details about the available parameters.
batchsubmit –h | more

Exercise 2: Run OpenSees locally on
batchsubmit —appdir /apps/openseesbuild/osg !OpenSees sine_no_gui.tcl
Examine job.stdout to see the data. Examine job.stderr for errors.

Exercise 3: Change the jobtype to something besides the program name.
batchsubmit —jobtype foobar —appdir /apps/openseesbuild/osg OpenSees sine_no_gui.tcl
Notice the path to the job.

Exercise 4: Change the name of the default jobs to something besides job0000#.
batchsubmit —jbname foobar —appdir /apps/openseesbuild/osg OpenSees sine_no_gui.tcl
Notice the path to the job.

Exercise 5: Start 5 jobs with a wrapper program.

Exercise 6: Run a job on osg. EXAMPLE ONLY – USE FOR LARGE JOBS.
batchsubmit —venue osg —appdir /apps/openseesbuild/osg OpenSees sine_no_gui.tcl

Exercise 7: Run a job on ranger. EXAMPLE ONLY – USE FOR LARGE JOBS.
batchsubmit —venue ranger –ncpus 16 —appdir /apps/openseesbuild/osg OpenSees sine_no_gui.tcl
Request access to High Performance Computing group to use Xsede clusters.
Specify the number of processes with ncpus. Required for ranger.

Step 4 – What is the state of my simulation?

To look at the state of your simulation type the “batchstatus” command. This will list the jobs that you have run, the current status and the directory containing the results of the simulation. Use “batchstatus | grep keyword” to filter the results on a given keyword.


Step 5 – Changing directory to simulation file

You can change your directory to the location where the files are stored. For example, in this particular example my job is “job00025” and the file is located at “/scratch/gemezm/opensees/job00025/batchsubmit_examples”. Therefore, I can change my current directory with “cd /scratch/gemezm/opensees/job00025/batchsubmit_examples” (Hint: You don’t like typing long path names? – try typing the first few letters and then pressing the “tab” key. It will complete the name for you or give you different options if there are more than one files or directories available). Use the “ls” or “ls –la” command to see the list of files and directories.


Step 6 – Downloading results from simulation

You can download the files of your simulation using OpenSEES.

Step 7 – Downloading results from simulation

Create a group to share your results. On the main menu, mouse over Collaborate and select Groups Click the Create User Group link. Complete the form and take note of the required Group ID field.

While on the group page, click the Data Sharing link on the left side of the page. Click the green button to create the data sharing directory for the group. Using SynchroNEES, upload all of your simulation files to “/data/groups/your group/dropbox”


Lastly, invite members to your group. Click the Members link on the left side of the group page. After your peers join the group, they will be able to access the files.


Apple Users

If your workspace does not open, you probably have to update your Java Preferences. To check your Java Preferences, go to Applications->Utilities->Java Preferences. Make sure that you click the checkbox next to “Enable applet plug-in and Web Start applications”.


Created on , Last modified on