This page is for building the MPI targets for TIGRE/Globus etc. It will be somewhat rough as it is a work in progress.
Building us_gridcontrol_t
This must be made on bcf. It is not necessary on any other systems, unless there is a new system acting as the web portal.
Any revision update to us_fe_nnls_t_mpi must be in-sync with updates to us_gridcontrol_t
# cd $ULTRASCAN/develop/
# qmake libus
# make -j80
# qmake us_all
# make -j80
# cd us_gridcontrol_t
# make
Building the MPI us_fe_nnls_t_mpi targets
alamo, laredo & bcf
$ ssh to the appropriate system
$ su - tigre
$ cd $ULTRASCAN
$ svn update
alamo, laredo
$ ssh c0-0
$ export QMAKESPEC=linux-g++-mpi
$ cd develop/us_fe_nnls_t_mpi
$ make
bcf
$ export QMAKESPEC=linux-g++-64-mpi
$ make
lonestar
(from bcf)
$ su - apache
$ gsissh lonestar.tacc.utexas.edu
$ cd ultrascan
$ svn update
$ cd develop/us_fe_nnls_t_mpi
$ export QMAKESPEC=linux-icc-64-mpi
$ make
$ cptowork
hlrb
(from bcf)
$ su - apache
$ gsissh -p 2222 a01.hlrb2.lrz-muenchen.de
$ cd ultrascan
$ svn update
$ cd develop/us_fe_nnls_t_mpi
$ make # note - if the Makefile is remade it will need to be manually edited to substitute g++ with mpiCC and gcc with mpicc, also the CFLAG and CXXFLAG -DUSE_MPI needs to added as the qmakespec is system defined
$ cd ~/ultrascan/bin
$ cp us_fe_nnls_t_mpi us_fe_nnls_t
Big Red
$ cat .soft # it should match the following
# # This is the .soft file. # It is used to customize your environment by setting up environment # variables such as PATH and MANPATH. # To learn what can be in this file, use 'man softenv'. # # @bigred # TeraGrid wide basic software suite @teragrid-basic # TeraGrid wide Globus 4 and Grid software suite @globus-4.0 # Platform recommended development software suite # @teragrid-dev +svn #+mpich-mx-ibm-64 #+mpich-mx-1.2.7..1-gcc-4.1.1-64 +mpich-mx-1.2.7..1-gcc-32 #@mpich-g2-1.2.6e-mpich-mx-xlc #+mpich-mx-1.2.7..1-gcc-64 QTDIR=$HOME/qt QWTDIR=$HOME/qwt QWT3DDIR=$HOME/qwt3d ULTRASCAN=$HOME/ultrascan PATH+=$HOME/bin PATH+=$QTDIR/bin LD_LIBRARY_PATH+=/usr/X11R6/lib/:$QTDIR/lib:$QWTDIR/lib QMAKESPEC=linux-g++-32-mpi
$ cat $QTDIR/mkspecs/linux-g++-32-mpi/qmake.conf
# # qmake configuration for linux-g++ # MAKEFILE_GENERATOR = UNIX TEMPLATE = app CONFIG += qt warn_on release incremental link_prl QMAKE_INCREMENTAL_STYLE = sublib QMAKE_CC = mpicc QMAKE_LEX = flex QMAKE_LEXFLAGS = QMAKE_YACC = yacc QMAKE_YACCFLAGS = -d QMAKE_YACCFLAGS_MANGLE = -p $base -b $base QMAKE_YACC_HEADER = $base.tab.h QMAKE_YACC_SOURCE = $base.tab.c QMAKE_CFLAGS = -pipe QMAKE_CFLAGS_DEPS = -M QMAKE_CFLAGS_WARN_ON = -Wall -W QMAKE_CFLAGS_WARN_OFF = -w QMAKE_CFLAGS_RELEASE = -O2 QMAKE_CFLAGS_DEBUG = -g QMAKE_CFLAGS_SHLIB = -fPIC QMAKE_CFLAGS_YACC = -Wno-unused -Wno-parentheses QMAKE_CFLAGS_THREAD = -D_REENTRANT QMAKE_CXX = mpicxx QMAKE_CXXFLAGS = $$QMAKE_CFLAGS QMAKE_CXXFLAGS_DEPS = $$QMAKE_CFLAGS_DEPS QMAKE_CXXFLAGS_WARN_ON = $$QMAKE_CFLAGS_WARN_ON QMAKE_CXXFLAGS_WARN_OFF = $$QMAKE_CFLAGS_WARN_OFF QMAKE_CXXFLAGS_RELEASE = $$QMAKE_CFLAGS_RELEASE QMAKE_CXXFLAGS_DEBUG = $$QMAKE_CFLAGS_DEBUG QMAKE_CXXFLAGS_SHLIB = $$QMAKE_CFLAGS_SHLIB QMAKE_CXXFLAGS_YACC = $$QMAKE_CFLAGS_YACC QMAKE_CXXFLAGS_THREAD = $$QMAKE_CFLAGS_THREAD QMAKE_INCDIR = QMAKE_LIBDIR = QMAKE_INCDIR_X11 = /usr/X11R6/include QMAKE_LIBDIR_X11 = /usr/X11R6/lib QMAKE_INCDIR_QT = $(QTDIR)/include QMAKE_LIBDIR_QT = $(QTDIR)/lib QMAKE_INCDIR_OPENGL = /usr/X11R6/include QMAKE_LIBDIR_OPENGL = /usr/X11R6/lib QMAKE_LINK = mpicxx QMAKE_LINK_SHLIB = mpicxx QMAKE_LFLAGS = QMAKE_LFLAGS_RELEASE = QMAKE_LFLAGS_DEBUG = QMAKE_LFLAGS_SHLIB = -shared QMAKE_LFLAGS_PLUGIN = $$QMAKE_LFLAGS_SHLIB QMAKE_LFLAGS_SONAME = -Wl,-soname, QMAKE_LFLAGS_THREAD = QMAKE_RPATH = -Wl,-rpath, QMAKE_LIBS = QMAKE_LIBS_DYNLOAD = -ldl QMAKE_LIBS_X11 = -lXext -lX11 -lm QMAKE_LIBS_X11SM = -lSM -lICE QMAKE_LIBS_NIS = -lnsl QMAKE_LIBS_QT = -lqt QMAKE_LIBS_QT_THREAD = -lqt-mt QMAKE_LIBS_OPENGL = -lGLU -lGL -lXmu QMAKE_LIBS_OPENGL_QT = -lGL -lXmu QMAKE_LIBS_THREAD = -lpthread QMAKE_MOC = $(QTDIR)/bin/moc QMAKE_UIC = $(QTDIR)/bin/uic QMAKE_AR = ar cqs QMAKE_RANLIB = QMAKE_TAR = tar -cf QMAKE_GZIP = gzip -9f QMAKE_COPY = cp -f QMAKE_COPY_FILE = $(COPY) QMAKE_COPY_DIR = $(COPY) -r QMAKE_MOVE = mv -f QMAKE_DEL_FILE = rm -f QMAKE_DEL_DIR = rmdir QMAKE_STRIP = strip QMAKE_STRIPFLAGS_LIB += --strip-unneeded QMAKE_CHK_DIR_EXISTS = test -d QMAKE_MKDIR = mkdir -p
$ cd $ULTRASCAN/ $ svn update $ cd develop/us_fe_nnls_t_mpi $ make
Creating a new target system
Initial compile
Suppose target system is named target.
Setup ssh access to target system.
from bcf
$ cd ~ftp/pub/linux/
$ scp qt-x11-free-3.3.4.tar.bz2 qwt-4.2.0rc2.tar.bz2 qwtplot3d-0.2.6.tgz user@target:
$ ssh target
$ svn co svn+ssh://username@bcf.uthscsa.edu/ultrascan/trunk ultrascan
extract each of the packages & setup symlinks to ~/qt, ~/qwt, ~/qwt3d on the target system
edit ~/.profile on the target system to include
export QTDIR=~/qt
export QWTDIR=~/qwt
export QWT3DDIR=~/qwt3d
export PATH=~/qt/bin:$PATH
export ULTRASCAN=~/ultrascan
export LD_LIBRARY_PATH=$QTDIR/lib:$QWTDIR/lib:$QWT3DDIR/lib
After editing .profile, logout & in again to make sure the environment has taken.
Now, an appropriate qmake.conf must be setup for this system depending on the preferred compiler.
This is created in ~/qt/mkspecs/new_mkspec_name/qmake.conf
Compile the apps on the target
$ cd ~qt
$ ./configure -thread -platform new_mkspec_name
$ gmake sub-src -j4
$ cd ~/qwt
$ qmake
$ make -j4
$ cd ~/qwt
$ qmake
$ make -j4
$ cd ~/ultrascan/develop/us_fe_nnls_t_mpi
$ export QMAKESPEC=new_mkspec_name
$ qmake
$ make
Test runs should be tried based upon the target systems HPC documentation.
globus integration needs to be tested and setup in us_tigre_job.pl & us_store_tigre_sysstat in bcf:$ULTRASCAN/etc
Test, test & test...
