wiki:MPIBuildNotes

This page is for building the MPI targets for TIGRE/Globus etc. It will be somewhat rough as it is a work in progress.

Building us_gridcontrol_t

This must be made on bcf. It is not necessary on any other systems, unless there is a new system acting as the web portal. Any revision update to us_fe_nnls_t_mpi must be in-sync with updates to us_gridcontrol_t

# cd $ULTRASCAN/develop/
# qmake libus
# make -j80
# qmake us_all
# make -j80
# cd us_gridcontrol_t
# make

Building the MPI us_fe_nnls_t_mpi targets

alamo, laredo & bcf

$ ssh to the appropriate system
$ su - tigre
$ cd $ULTRASCAN
$ svn update

alamo, laredo

$ ssh c0-0
$ export QMAKESPEC=linux-g++-mpi
$ cd develop/us_fe_nnls_t_mpi
$ make

bcf

$ export QMAKESPEC=linux-g++-64-mpi
$ make

lonestar

(from bcf)

$ su - apache
$ gsissh lonestar.tacc.utexas.edu
$ cd ultrascan
$ svn update
$ cd develop/us_fe_nnls_t_mpi
$ export QMAKESPEC=linux-icc-64-mpi
$ make
$ cptowork

hlrb

(from bcf)

$ su - apache
$ gsissh -p 2222 a01.hlrb2.lrz-muenchen.de
$ cd ultrascan
$ svn update
$ cd develop/us_fe_nnls_t_mpi
$ make # note - if the Makefile is remade it will need to be manually edited to substitute g++ with mpiCC and gcc with mpicc, also the CFLAG and CXXFLAG -DUSE_MPI needs to added as the qmakespec is system defined
$ cd ~/ultrascan/bin
$ cp us_fe_nnls_t_mpi us_fe_nnls_t

Big Red

$ cat .soft # it should match the following

#
# This is the .soft file.
# It is used to customize your environment by setting up environment
# variables such as PATH and MANPATH.
# To learn what can be in this file, use 'man softenv'.
#
#
@bigred
# TeraGrid wide basic software suite
@teragrid-basic

# TeraGrid wide Globus 4 and Grid software suite
@globus-4.0

# Platform recommended development software suite
# @teragrid-dev
+svn
#+mpich-mx-ibm-64
#+mpich-mx-1.2.7..1-gcc-4.1.1-64
+mpich-mx-1.2.7..1-gcc-32
#@mpich-g2-1.2.6e-mpich-mx-xlc
#+mpich-mx-1.2.7..1-gcc-64

QTDIR=$HOME/qt
QWTDIR=$HOME/qwt
QWT3DDIR=$HOME/qwt3d
ULTRASCAN=$HOME/ultrascan

PATH+=$HOME/bin
PATH+=$QTDIR/bin
LD_LIBRARY_PATH+=/usr/X11R6/lib/:$QTDIR/lib:$QWTDIR/lib
QMAKESPEC=linux-g++-32-mpi

$ cat $QTDIR/mkspecs/linux-g++-32-mpi/qmake.conf


#
# qmake configuration for linux-g++
#

MAKEFILE_GENERATOR      = UNIX
TEMPLATE                = app
CONFIG                  += qt warn_on release incremental link_prl
QMAKE_INCREMENTAL_STYLE = sublib

QMAKE_CC                = mpicc
QMAKE_LEX               = flex
QMAKE_LEXFLAGS          =
QMAKE_YACC              = yacc
QMAKE_YACCFLAGS         = -d
QMAKE_YACCFLAGS_MANGLE  = -p $base -b $base
QMAKE_YACC_HEADER       = $base.tab.h
QMAKE_YACC_SOURCE       = $base.tab.c
QMAKE_CFLAGS            = -pipe
QMAKE_CFLAGS_DEPS       = -M
QMAKE_CFLAGS_WARN_ON    = -Wall -W
QMAKE_CFLAGS_WARN_OFF   = -w
QMAKE_CFLAGS_RELEASE    = -O2
QMAKE_CFLAGS_DEBUG      = -g
QMAKE_CFLAGS_SHLIB      = -fPIC
QMAKE_CFLAGS_YACC       = -Wno-unused -Wno-parentheses
QMAKE_CFLAGS_THREAD     = -D_REENTRANT

QMAKE_CXX               = mpicxx
QMAKE_CXXFLAGS          = $$QMAKE_CFLAGS
QMAKE_CXXFLAGS_DEPS     = $$QMAKE_CFLAGS_DEPS
QMAKE_CXXFLAGS_WARN_ON  = $$QMAKE_CFLAGS_WARN_ON
QMAKE_CXXFLAGS_WARN_OFF = $$QMAKE_CFLAGS_WARN_OFF
QMAKE_CXXFLAGS_RELEASE  = $$QMAKE_CFLAGS_RELEASE
QMAKE_CXXFLAGS_DEBUG    = $$QMAKE_CFLAGS_DEBUG
QMAKE_CXXFLAGS_SHLIB    = $$QMAKE_CFLAGS_SHLIB
QMAKE_CXXFLAGS_YACC     = $$QMAKE_CFLAGS_YACC
QMAKE_CXXFLAGS_THREAD   = $$QMAKE_CFLAGS_THREAD

QMAKE_INCDIR            =
QMAKE_LIBDIR            =
QMAKE_INCDIR_X11        = /usr/X11R6/include
QMAKE_LIBDIR_X11        = /usr/X11R6/lib
QMAKE_INCDIR_QT         = $(QTDIR)/include
QMAKE_LIBDIR_QT         = $(QTDIR)/lib
QMAKE_INCDIR_OPENGL     = /usr/X11R6/include
QMAKE_LIBDIR_OPENGL     = /usr/X11R6/lib

QMAKE_LINK              = mpicxx
QMAKE_LINK_SHLIB        = mpicxx
QMAKE_LFLAGS            =
QMAKE_LFLAGS_RELEASE    =
QMAKE_LFLAGS_DEBUG      =
QMAKE_LFLAGS_SHLIB      = -shared
QMAKE_LFLAGS_PLUGIN     = $$QMAKE_LFLAGS_SHLIB
QMAKE_LFLAGS_SONAME     = -Wl,-soname,
QMAKE_LFLAGS_THREAD     =
QMAKE_RPATH             = -Wl,-rpath,

QMAKE_LIBS              =
QMAKE_LIBS_DYNLOAD      = -ldl
QMAKE_LIBS_X11          = -lXext -lX11 -lm
QMAKE_LIBS_X11SM        = -lSM -lICE
QMAKE_LIBS_NIS          = -lnsl
QMAKE_LIBS_QT           = -lqt
QMAKE_LIBS_QT_THREAD    = -lqt-mt
QMAKE_LIBS_OPENGL       = -lGLU -lGL -lXmu
QMAKE_LIBS_OPENGL_QT    = -lGL -lXmu
QMAKE_LIBS_THREAD       = -lpthread

QMAKE_MOC               = $(QTDIR)/bin/moc
QMAKE_UIC               = $(QTDIR)/bin/uic

QMAKE_AR                = ar cqs
QMAKE_RANLIB            =

QMAKE_TAR               = tar -cf
QMAKE_GZIP              = gzip -9f

QMAKE_COPY              = cp -f
QMAKE_COPY_FILE         = $(COPY)
QMAKE_COPY_DIR          = $(COPY) -r
QMAKE_MOVE              = mv -f
QMAKE_DEL_FILE          = rm -f
QMAKE_DEL_DIR           = rmdir
QMAKE_STRIP             = strip
QMAKE_STRIPFLAGS_LIB    += --strip-unneeded
QMAKE_CHK_DIR_EXISTS    = test -d
QMAKE_MKDIR             = mkdir -p

$ cd $ULTRASCAN/ $ svn update $ cd develop/us_fe_nnls_t_mpi $ make

Creating a new target system

Initial compile

Suppose target system is named target.
Setup ssh access to target system.
from bcf
$ cd ~ftp/pub/linux/
$ scp qt-x11-free-3.3.4.tar.bz2 qwt-4.2.0rc2.tar.bz2 qwtplot3d-0.2.6.tgz user@target:
$ ssh target
$ svn co svn+ssh://username@bcf.uthscsa.edu/ultrascan/trunk ultrascan

extract each of the packages & setup symlinks to ~/qt, ~/qwt, ~/qwt3d on the target system

edit ~/.profile on the target system to include
export QTDIR=~/qt
export QWTDIR=~/qwt
export QWT3DDIR=~/qwt3d
export PATH=~/qt/bin:$PATH
export ULTRASCAN=~/ultrascan
export LD_LIBRARY_PATH=$QTDIR/lib:$QWTDIR/lib:$QWT3DDIR/lib

After editing .profile, logout & in again to make sure the environment has taken.
Now, an appropriate qmake.conf must be setup for this system depending on the preferred compiler. This is created in ~/qt/mkspecs/new_mkspec_name/qmake.conf

Compile the apps on the target
$ cd ~qt
$ ./configure -thread -platform new_mkspec_name
$ gmake sub-src -j4
$ cd ~/qwt
$ qmake
$ make -j4
$ cd ~/qwt
$ qmake
$ make -j4
$ cd ~/ultrascan/develop/us_fe_nnls_t_mpi
$ export QMAKESPEC=new_mkspec_name
$ qmake
$ make

Test runs should be tried based upon the target systems HPC documentation.
globus integration needs to be tested and setup in us_tigre_job.pl & us_store_tigre_sysstat in bcf:$ULTRASCAN/etc
Test, test & test...

Last modified 9 years ago Last modified on Jul 24, 2009 8:26:19 PM