NEES - Running OpenSees on OSG Computers
Running OpenSees on OSG from NEEShub
As of July 20th 2011, NEEShub offers the capability to run jobs on the Open Science Grid (OSG) computing infrastructure. The Open Science Grid is a consortium of about 80 universities and national laboratories that work together to form a national cyberinfrastructure. Resources belonging to the OSG institutions are shared across the consortium.
Using OSG computers should allow running many simultaneous jobs without much affecting other NEEShub users. The disadvantage is that if there are just a few jobs they can be slower to start on OSG than on the dedicated NEEShub machines.
In order for jobs to run on OSG they must be explicitly submitted there. An example of how to do this from the command line is given below. Applications that have a GUI could be modified to internally do job submission in a similar way.
There can be many factors to consider when running OpenSees jobs. If you have problems or questions please file a NEES ticket.
Dispatching OpenSees Jobs to OSG from a Workspace
After starting a NEEShub workspace a user can run a script called osg_opensees which does job submission to OSG.
Here is an example of a simple application that comes with opensees2:
$ /apps/bin/osg_opensees /apps/opensees2/current/examples/sine_no_gui/sine_no_gui.tcl osg_opensees START: Transfer directory = /home/neeshub/mslyz/osg_rundirs/run00012 WARNING: 2 files copied from /apps/opensees2/current/examples/sine_no_gui to /home/neeshub/mslyz/osg_rundirs/run00012 including sine_no_gui.tcl submit -v OSGFactory -i /home/neeshub/mslyz/osg_rundirs/run00012 /home/neeshub/mslyz/osg_rundirs/run00012/cmdfile (179.0) Job Submitted at OSGFactory Tue Jul 19 14:48:08 2011 (179.0) Simulation Idle at OSGFactory Tue Jul 19 14:49:13 2011 (179.0) Simulation Running at Fermigridosg1 Tue Jul 19 14:54:49 2011 (179.0) Simulation Done at Fermigridosg1 Tue Jul 19 14:55:44 2011 /apps/bin/osg_opensees: DONE! Results are in /home/neeshub/mslyz/osg_rundirs/run00012/@RUNDIR
When the script is done, the output data appears inside a numbered directory in $HOME/osg_rundirs.
* There are several different ways to transfer data to and from a NEEShub account. If the default 1GB per-account limit is too small, up to 10GB can be available on request.
* osg_opensees automatically transfers to OSG computers any files in the same directory as the OpenSees tcl command script (i.e. sine_no_gui.tcl in the example above), although it does not transfer subdirectories.
* In each job’s environment the JOBCOUNT variable is set to a unique number and can be used to select a different input data set for each run of OpenSees.
* Some of the limitations of the current OSG setup are:
- MPI is not supported.
- There is usually no more than 10GB of disk space available for each individual job.
- There is typically only 1 to 2 GB of RAM available per job.
- Keeping run times to less than 12 hours is recommended. Longer jobs, especially past 16 hours, are likely to not finish successfully.