Globus Installation in the Astrogrid-D Standard (GACSI)

(AstroGrid-D / AIP, as of: 02.01.2007. Based on Globus-Quickstart)

[top]
1. System Requreiments and Preparation

This guide is intended for the Astrogrid-D-Standard Installation with Scientific Linux (Redhat Enterprise Unix, 4.2 or higher). Essential is Root-access incl. sudo: for the most part the installation is done as user root. System administration experience would be useful, but is not required to complete the installation successfully using this guide.
At many places in this document a text editor is required. Here vi is used for examples. Other alternatives are nedit (graphical Editor) or pico and nano.

Next the Guide to required Unix Software and Updates should be followed, which can be found for easier reference in a separate document. It should be ensured that the required Software versions are correctly installed and run.

The following table gives the links to the Source-archives for (Globus, iODBC) and the Globus-Helper Package, which need to be downloaded, preferably to /work1:

Source code Version Buid directory Target binary directory

Globus toolkit - Full Toolkit Source Download gt4.0.5-all-source-installer.tar.bz2 /work1/build/ /usr/local/globus/gtk/ [link]

iODBC source files 3.5 /work1/build/ ~globus/iodbc/

Astrogrid-D Globus Helper package globus-helper-v1.2.tgz - ~globus/globus-helper/

SGAS

sgas-2.0-installer.tar.gz

/usr/local/globus/sgas-2.0

The installation can be done using a virtual machine (VMware 5.5 or higher recommended). Recommendations for the firewall settings are given in D-Grid at: Configuration of static Firewalls (pdf) (76KB).

The IP-Number and Fully Qualified Domain-Name (FQDN) need to be entered correctly into /etc/hosts, e. g.:

more /etc/host 141.33.4.98 cashmere.aip.de

The relevant IP address can be found using ifconfig. If the FQDN is not present, /etc/hosts should be edited and a line added.
If the machine name is not listed in the DNS, then it must be explicitly entered into /etc/hosts on the Grid resources from which it will be accessed.

In the following examples at different places Names and Version numbers are used which are correct at the time of creating this document. During installation these should be replaced if required. In this guide a Standard directory structure is used, which is listed below:

Home directory for user Globus: /work1/globus/ [= "~globus"]
Build: /work1/build/
$GLOBUS_LOCATION: /usr/local/globus/gtk [=link to /work1/globus/gtk403]
Certificates: /etc/grid-security/, /root/AstroCert/ and ~/.globus (for the User Certificates)

This structure is used in the following (especially Chapter 4) and is implemented in the scripts belonging to "globus-helper" but it does not have to be adhered to as long as the scripts are adjusted accordingly.

[top]
2. Set up user `globus`

As user root, the user "globus" with group "globus" is set up. The home directory may be /work1/globus; the login-shell should be bash. The id's can be chosen (here: 10001).

/usr/sbin/groupadd -g 10001 globus /usr/sbin/useradd -u 10001 -g globus -s /bin/bash -d /work1/globus globus

Then the directory for the Globus Toolkit is set up and a soft link created:

cd ~globus mkdir gt403 mkdir -p /usr/local/globus chown globus:globus gt403 /usr/local/globus
ln -s ~globus/gt403 /usr/local/globus/gtk

In later upgrades to new version this link is updated, but the configuration data, .bashrc and $PATH variables remain the same. Finally the globus-helper package is copied into the directory and the update directory created:

wget -q -O - http://www.gac-grid.net/project-products/Software/globus-helper-v1.2.tgz | tar xfz - ln -s globus-helper-v1.2 globus-helper mkdir globus-update

[top]
3. iODBC installation

The relevant directories are created the source files unpacked:

mkdir -p /work1/build chown globus:globus /work1/build mv /work1/libiodbc-3.52.2.tar.gz /work1/build cd /work1/build tar xvzf libiodbc-3.52.2.tar.gz cp ~globus/globus-helper/globus-install/iodbc.cfg /work1/build/libiodbc-3.52.2/ cd /work1/build/libiodbc-3.52.2

The file iodbc.cfg contains the correct Globus settings. If other paths are to be used these must be set in iodbc.cfg. The file contains the following call to configure:

./configure --prefix=/work1/globus/iodbc --disable-gtktest --with-pthreads --disable-gui \ --with-iodbc-inidir=/work1/globus/iodbc/etc

sh -x iodbc.cfg make

Now iodbc can be installed (as user root):

make install

[top]
4. Setting the PATH and Globus defaults

It is important to have the directory-variables for the Java Software Development Kit (JDK), ant and Tomcat set correctly. These are not Globus specific and often follow from the package installation (see the " Guide to Required Unix Software and Updates"). It is also useful, if not already in place, to create symbolic links from a standard location to the actual binaries in the particular directory versions e.g.:

mkdir -p /usr/local/jdk ln -s /opt/jsdk-xx/ant-1.6.5/ /usr/local/jdk/ant

Similar links should be created for JSDK - /usr/local/jdk/jsdk and Tomcat to /usr/local/jdk/tomcat.
Then these standard directories can be put into the standard path:

cat ~globus/globus-helper/griduser/etc_profile.add >> /etc/profile

The contents of globus_defs/griduser/etc_profile.add is then:

JDK="/usr/local/jdk"
JAVA_HOME="$JDK/jsdk"
CATALINA_HOME="$JDK/tomcat"
ANT_HOME="$JDK/ant"
J_BIN="$ANT_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$CATALINA_HOME/bin"
PATH="$J_BIN:$PATH"
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC JAVA_HOME ANT_HOME CATALINA_HOME

Duplicate 'export' commands should be removed so that the path does not become to long. The given syntax is valid for bash. For csh and tcsh the relevant command for setting variables 'setenv' should be used.

After the standard directories the Globus-specific variables for the users 'root' and 'globus' are set (and respectively for additional users). For this purpose the globus-helper/griduser/grid-env.sh is appended to the .bashrc (for csh/tcsh: globus-helper/griduser/grid-env.csh):

cat ~globus/globus-helper/griduser/grid-env.sh >> ~globus/.bashrc cat ~globus/globus-helper/griduser/grid-env.sh >> /root/.bashrc

In the grid-env.sh the target pointed to by the symbolic link is set as the main Globus directory. The standard content is:

GLOBUS_LOCATION=/usr/local/globus/gtk GLOBUS_PATH=$GLOBUS_LOCATION/sbin PATH=$GLOBUS_PATH:$PATH GLOBUS_TCP_PORT_RANGE=20000,25000 export PATH GLOBUS_LOCATION GLOBUS_TCP_PORT_RANGE source $GLOBUS_LOCATION/etc/globus-user-env.sh

[top]
5. Configuring and Compiling the Globus Toolkit

Change to user globus now to avoid permissions issues. If the source is unpacked as user root then run the following command afterwards in the expanded directory : 'chown -R globus *'. Because some scripts have not yet been copied for the Globus Toolkit there is an error message on switching to user globus - 'su globus':
/usr/local/globus/gtk/etc/globus-user-env.sh: No such file or directory.
This can be ignored.

cd /work1/build/ chmod 777 * su - globus # blue letters indicate user globus cd /work1/build tar xvfj ../gt4.0.3-all-source-installer.tar.bz2 cd gt4.0.3-all-source-installer cp ~globus/globus-helper/globus-install/gt4.cfg .

The file gt4.cfg should be edited as follows.

the variable GLOBUS_LOCATION is to specify the absolute, real path (no symbolic links!) of the directory in which Globus is to be installed e.g. GLOBUS_LOCATION=/work1/globus/gtk403
For older machines with 32-bit processers the 32-bit Support is set using the configure option --with-flavor="gcc32dbg" (in the 64-bit script).
For PBS Support (compute-clusters) the configure option --enable-wsgram-pbs Home-Variable (e. g. PBS_HOME=/work1/PBS) must be set, otherwise this should be commented out PBS = "Portable Batch System" for compute-clusters.
ANT_HOME and JAVA_HOME have to be defined correctly and exported and the their bin directories must be in the $PATH.
it should be verified that the configure options (given in the last line ./configure ...) are not separated by a blank line or a commented line as the following options would be ignored.

Now the Makefile can be generated:

ls $ANT_HOME $JAVA_HOME # Test sh -x gt4.cfg

Warnings should not be ignored - instead the causes need to be resolved. Also attempts with additional compiler flags (-j3) were so far unsuccessful.

Only after a configuration without Warnings is the Makefile to be called:

make 2>&1 | tee /work1/build/globus_install.log

Because 'make' calls 'ant' it is not certain that all errors in the make dependencies are correctly reported in the case paths are not set correctly. If "make" fails later due to Java errors and the problem is resolved then before starting a new 'make' remove the entire directory-tree under /work1/globus/gtk . 'Make' also stops if a version-number of the required program is insufficient (see the Guide to Required Unix Software and Updates) e.g. for libz.so (zlib).

If the compile is successful (on current systems this only take 2 hours) the program is then installed:

make install

A successful install finishes with the message "..Done". If this does not complete normally the options should be checked and the compilation restarted. If it is not present in the make script the last call is to 'gpt-postinstall' as a final check and completion of the installation.

[top]
6. Linking in the Globus Advisories

The Globus Advisories (i.e. the updates and patch-fixes for a particular version number) can be downloaded into the directory "~globus/globus-updates/". Here it needs to be checked which parts of Globus are installed e.g. no 'rls' updates reduces the effort. The linking in can be done using the script update_globus.pl from the globus-helper package:

cd ~globus/globus-updates cp ~globus/globus-helper/globus-install/update_globus.pl . ./update_globus.pl -d . export GPT_LOCATION=$GLOBUS_LOCATION gpt-postinstall

For full updates between two version numbers there is a separate guide in the section on Globus Installation on the AstroGrid-D website.

[top]
7. Installation of the Service and Host Certificates

First as user globus the Globus-default Certificate is generated.

$GLOBUS_LOCATION/setup/globus/setup-simple-ca -noint

To complete the user root calls the command printed from output of the above script. ( The GLOBUS_LOCATION variable must be set) e.g. :

$GLOBUS_LOCATION/setup/globus_simple_ca_{CA-HASH}_setup/setup-gsi

These default certificates are not required for working in the AstroGrid and are for backup only. They will be replaced by the certificates from the Root-CA at the Forschungszentrum Karlsruhe. The script which does this work, installs the public FZK-Certificates and configures the Organisational Unit, to which the machine belongs. This is passed using the option -i. The list of all the OU's registered in the Astrogrid-D is given in http://grid.fzk.de/ca/RA.html

cd /etc/grid-security tar cfz orig.tgz * cd ~globus/globus-helper/security/configrootCA/ ./ConfigureFZK-CA.pl -i {AEI|AIP| ...} -g

If the Certificate Authority used is DFN instead of FZK, check with the local Registration Authority in case there are any differences. The Certificate Request can now be generated and sent. The identifying domain-name is required in the request e.g. "nonexistent.aip.de".

mkdir -p /root/AstroCert ~globus/globus-helper/security/service/generate_service_req.pl -h nonexistent.aip.de -d /root/AstroCert/

the submitted request for "nonexistent.aip.de_hostcert_request.pem" can be sent as an attachment to the locally responsible Registration Authority ( in your own Institute) or using

cat /root/AstroCert/nonexistent.aip.de_hostcert_request.pem | mail

The locally responsible RA can be found in the RA-List of VOMRS. The request will be signed and passed on. It is also possible in principle to submit the request ones-self but this is very unusual. It can take a few days until the email-response with the signed and valid certificate arrives. Some of the steps from the following chapters can already be completed without the certificate.
When the signed certificate is available, it can be installed:

mkdir -p /root/AstroCert cp certificates_from_mail /root/AstroCert/ cd /root/AstroCert/ ~globus/globus-helper/security/service/install_service_cert.pl -h nonexistent.aip.de

The last script moves and renames the certificates to /etc/grid-security/. The Container-certificate is copied from the Host-Certificate. After installation using 'ls -l /etc/grid-security/*.pem' check that the certificates have the correct permissions (owner 'root' , instead of 'globus' is also possible):

  -r--------  1 globus globus containerkey.pem
  -rw-r--r--  1 globus globus containercert.pem
  -rw-r--r--  1 root   root   hostcert.pem
  -r--------  1 root   root   hostkey.pem

Finally the public-key from the DFN Certification Authority is imported, which is used by the Astrogrid-D:

tar xfz ~globus/globus-helper/security/configrootCA/dfncerts.tgz --directory /etc/grid-security/certificates

For a daily verification that User-Certificates are valid, the program fetch-crl has to be installed. It updates the list of Certificate Authority revoked Certificates (Certificate Revocation List, CRL). The program has been adopted by the 'Astrogrid-D-Standard-Installation' and is installed using:

cd ~globus/globus-helper/security/astro-fetch-crl/ ./install_fetch-crl.pl

The script enters a daily update of the CRL into the cron-tables.
Gridfile-Management is handled further down in the section on User-Creation.
The process for getting a User-Certificate, with which a user can access the Grid is described in the GAC-Userguide.

[top]
8. Setup for `gsiftp` and `gsissh`

The required gsiftp ports have to be entered into /etc/services using an editor such e.g. vi /etc/services. the relevant ports are entered as new lines:

gsigatekeeper     2119/tcp

...

gsiftp            2811/tcp

...

Registering the services to xinitd.d is done using the below script :

cd ~globus/globus-helper/xinetd.d ./install_xinet-conf.pl

The Globus standard port for gsissh is the ssh port 22. if the usual ssh daemon is still to be used, then the gsissh daemon needs to be set to a different port. The port 2222 is recommended as more-or-less standard.

The service is added to init.d :

ln -s $GLOBUS_LOCATION/sbin/SXXsshd /etc/init.d/gsisshd /sbin/chkconfig --add gsisshd

To reconfigure the daemon,

the entry $GLOBUS_LOCATION/etc/ssh/sshd_config for "Port 22" is to be commented out (if not already commented out) and the new entry " Port 2222 " should be entered.
edit the file $GLOBUS_LOCATION/etc/ssh/ssh_config and the change to "Port 2222". It is also necessary to chmod 664 $GLOBUS_LOCATION/etc/ssh/ssh_config.
The new service is entered by editing the /etc/services file with the new line " gsissh 2222/tcp ".

Then the daemon is started to test the configuration:

/etc/init.d/gsisshd start

Anyone who is registered using grid-proxy-init (or my-proxy-logon) in the Grid and is listed in the grid-mapfile on the relevant machines, can then acess the Grid-Resources without a further password request. Otherwise gsissh behaves like ssh and asks for a local password. In case the Client and Server use different ports, use the option -p in the call e. g. gsissh -p 22 noexist.aip.de.

As a last step, the reaction speed of gsiftp connections is vastly improved if we configure iptables to avoid a certain timeout. The following line is to be executed as root:

iptables -A OUTPUT -p tcp -m tcp --dport 113 --tcp-flags SYN,RST,ACK SYN -j REJECT --reject-with tcp-reset

[top]
9. Configuring the database for Globus File Transfer (RFT-Database)

PostgreSQL is used for the database. For installation and checking PostgreSQL, see the System Requirements.

The database creation for Globus Reliable File Transfer (RFT) and access to it is done as user postgres, using an existing SQL script.

sudo su - postgres postgres> createdb rftDatabase CREATE DATABASE postgres> psql -d rftDatabase -f /usr/local/globus/gtk/share/globus_wsrf_rft/rft_schema.sql 6: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "requestid_pkey" for table "requestid" ... CREATE INDEX postgres> psql -d rftDatabase -c "create role globus with superuser login encrypted password 'foo'" postgres CREATE ROLE postgres> exit

If the postgresql.conf file does not exist, e,g. if postgresql had never been started, then this is created as follows: /etc/init.d/postgresql start.

If you are using PostgreSQL 8.0 or greater, to allow local access over TCP/IP sockets, the first line of Connection Settings in
/var/lib/pgsql/data/postgresql.conf has to be changed:

# - Connection Settings - listen_addresses = '*' ...

In any case, the /var/lib/pgsql/data/pg_hba.conf is updated to contain the local host.

# TYPE	DATABASE	USER	IP-ADDRESS	IP-MASK		METHOD
host	rftDatabase	globus	<IP-Number>	255.255.255.255	md5
...

In $GLOBUS_LOCATION/etc/globus_wsrf_rft/jndi-config.xml set the parameters:

<parameter> <name>userName</name> <value>globus</value> </parameter> <parameter> <name>password</name> <value>foo</value> </parameter> ...

Then restart the database with sudo /etc/init.d/postgresql restart! Test: psql -d rftDatabase -c " \d" globus shows the database tables.

[top]
10. Starting the Java Web-Services Core Container and `gsissh`

To start the container the Certification process must be completed. The Web-Service of the Globus Container can then be added to the init.d directory:

cd ~globus/globus-helper/init.d/ ./install_init-conf.pl -e

The service will then be started automatically on system start on init level 3. This can be tested using:

/etc/init.d/globus start

(this includes the call to "$GLOBUS_LOCATION/sbin/globus-start-container-detached -p 8443" ). The startup can take several seconds.

To check the successful startup and for debugging the following commands are available:

/etc/init.d/globus status more $GLOBUS_LOCATION/var/container.log ps aux | grep java # is the Java container running? $GLOBUS_LOCATION/bin/globus-start-container -debug -p 8443

[top]
11. Creating the Grid Users

The registered Astrogrid-D users are selected from the VORMS database and are entered in the grid-mapfile. This is handled by the script "ManageLocalGridUser.pl". The configuration file "conf/manage-grid-user.conf" has to contain the absolute path for the particular Globus installation. Be sure to adapt the configuration for your environment. For security reasons to do with 'sudo' no symbolic links are allowed in the file, only the absolute path of the original directory e.g. "/work1/globus/gtk403/". The script also expects the file

/root/AstroCert/VOMRS/volist_secrets

to be present. Copy the default file from ~globus/globus-helper/manage-griduser/, and ask your local VOMRS authority for the correct user and password fields.

cd ~globus/globus-helper/manage-griduser/ ManageLocalGridUser.pl -c conf/manage-grid-user.conf

The enabled set of users and groups are entered in the "sudoers" file, so that the WS-GRAM allows the user globus password-free access to all the grid-user accounts. The required lines are created by "ManageLocalGridUser.pl" and can be found in the files "sudoers-alias" and "sudoers-command". They can be copied into the sudoers files using copy & paste:

cat sudoers-* xterm -e visudo

Note the difference between the unix-user 'globus' as which the toolkit runs, and the Globus users. The latter uses its certificate to identify itself and to execute Globus programs.

[top]
12. Ganglia and Cluster Integration

The Globus-Helper package contains a script for configuring Ganglia with Globus which should be run:

sh -x $GLOBUS_LOCATION/globus-helper/globus-install/ganglia.cfg

For the Ganglia installation there is a separate guide, which is currently not integrated : Ganglia-MDS4.

To configure Ganglia, changes have to be made to the file
/usr/local/globus/gtk/etc/globus_wsrf_mds_usefulrp/gluerp.xml
The existing "defaultProvider" should be commented out and replaced with:

<defaultProvider>java org.globus.mds.usefulrp.glue.GangliaElementProducer</defaultProvider>

The standard installation continues in the Globus sbin directory. The file /work1/globus/ganglia-3.0.2/gmond/gmond.init is edited and the line GMOND=/usr/sbin/gmond replaced with GMOND=/usr/local/globus/ganglia/sbin/gmond, before being copied to /etc/init.d/.

For MDS the file /usr/local/globus/gtk/etc/globus_wsrf_core/server-config.wsdd is edited. the following lines are added to the section "<globalConfiguration>" :

<parameter name="logicalHost" value="Nonexistent.aip.de"/> <parameter name="publishHostName" value="true"/>

Of course 'Nonexistent.aip.de' is to be replaced with the correct DNS name.

For the MDS upload in the file /usr/local/globus/gtk/etc/globus_wsrf_mds_index/hierarchy.xml the section "<upstream>" is to be commented out and replace with:

https://astrogrid-mds.aip.de:8443/wsrf/services/DefaultIndexService

[top]
13. SGAS - SweGrid Accounting System

Part of the SGAS software functionality is used for host monitoring.
As user globus:

tar xzf sgas-2.0-installer.tar.gz cd sgas-2.0-installer ./install.sh /usr/local/globus/sgas-2.0 cp -p /usr/local/globus/sgas-2.0/lib/sgas*.jar $GLOBUS_LOCATION/lib

Add the contents of /usr/local/globus/sgas-2.0/etc/wsgram.conf as child elements of the <service> element belonging to the ManagedJobFactoryService GRAM service configuration in $GLOBUS_LOCATION/etc/gram-service/server-config.wsdd.

Add to the crontab of the user globus:

# # Move SGAS Usage Records to Stellaris Database # LOGDIR = /usr/local/globus/sgas-2.0/var/log/jarm URATTS = xmlns:urwg="http://www.gridforum.org/2003/ur-wg xmlns:grddl="http://www.w3.org/2003/g/data-view#" grddl:transformation="http://www.gac-grid.org/project-products/Software/xml2rdf/xml2rdf21.xsl" URHOST = http://is.astrogrid-d.org/files/hosts/ # * * * * * for UR in `ls $LOGDIR/urs/* 2>/dev/null`; do sed -i "2s!\(!\1 $URATTS>!" $UR && curl $URHOST${HOSTNAME-other}/urs/ -sT $UR && rm $UR; done; rm $LOGDIR/* 2>/dev/null

Restart the globus container with sudo /etc/init.d/globus restart!

Globus Installation in the Astrogrid-D Standard (GACSI)

Contents:

[top] 1. System Requreiments and Preparation

[top] 2. Set up user globus

[top] 3. iODBC installation

[top] 4. Setting the PATH and Globus defaults

[top] 5. Configuring and Compiling the Globus Toolkit

[top] 6. Linking in the Globus Advisories

[top] 7. Installation of the Service and Host Certificates

[top] 8. Setup for gsiftp and gsissh

[top] 9. Configuring the database for Globus File Transfer (RFT-Database)

[top] 10. Starting the Java Web-Services Core Container and gsissh

[top] 11. Creating the Grid Users

[top] 12. Ganglia and Cluster Integration

[top] 13. SGAS - SweGrid Accounting System