Appendices

Contents

  1. Appendix A : Necessary external software
    1. Windows specific
    2. Installing and configuring a JMS broker
      1. Obtaining a JMS broker
      2. Installing the JMS broker
      3. Configuring the JMS broker
      4. Starting and stopping JMS
        1. How to empty queues
        2. How to allocate additional JMS broker memory
    3. Installing and configuring FTP
      1. Starting and stopping a Proftpd server
  2. Appendix B : Starting automatically
    1. Linux
    2. Windows
  3. Easy Installation of NetarchiveSuite
    1. Examples of deploy configuration files
    2. A running HW/SW setup example from June 2009

Appendix A : Necessary external software

edit

The NetarchiveSuite is developed and tested with Sun Java SE (Standard Edition) JDK version 1.6.0_07. In any case a Java 1.6+ JDK will be necessary to compile and run the NetarchiveSuite, and we recommend that all applications use the same JDK.

The following external software is required for running the applications

Windows specific

Some application requires the Unix command sort, but they should be able to run under Windows if Cygwin is installed. This should only affect the ViewerProxy and the IndexServer.

Installing and configuring a JMS broker

The software have been tested with the free JMS broker from Sun "Open Message Queue 4.1", and the commercial JMSBroker "Sun MQ 3.6 Enterprise Edition".

Obtaining a JMS broker

Sun's Open Message Queue can be obtained from the following site: https://mq.dev.java.net/downloads.html Go to the section named "Legacy Versions", and click on the Linux link in the subsection "Open MQ 4.1 Binary Downloads". This will give you a jar-file named "mq4_1-binary-Linux_X86-XXXXXXXX.jar". (We have no reason to suppose that NetarchiveSuite will have problems with newer versions but these are still untested with our software.)

Note: We only support installation on the Linux platform here. However, you may want to install your JMS broker on a different platform. Binary versions are available at the site for: Solaris Sparc, Solaris x86, Linux (x86), Windows (x86). If you want to build a binary for another platform, the source can be downloaded from the download-page.

Installing the JMS broker

Select Linux server where you want to install JMS broker, and select an installation directory. Then log on the linux server as root, and do the following: 

   export INSTALLATION_DIR=/path/to/installationdir
   cd $INSTALLATION_DIR
   unzip mq4_1-binary-Linux_X86-XXXXXXXX.jar
   chmod +x ./mq/bin/imqbrokerd
   ./mq/bin/imqbrokerd -reset store -tty (tests that the broker can start - CTLR-C to stop)

Check that it starts, and that the last message is
"Broker <localhost>:7676 ready" We are now ready to configure the JMS broker.

Configuring the JMS broker

  • Edit the file $INSTALLATION_DIR/mq/etc/imqenv.conf to set IMQ_DEFAULT_JAVAHOME to a JDK1.5.0.

  • Changing the number of the listening port number 7676 is done by editing the line

  • imq.portmapper.port=7676

  • in the file

  • $INSTALLATION_DIR/mq/lib/props/broker/default.properties

  • Set max listeners any given queue to 20. You need to make sure, that the following line

  • imq.autocreate.queue.maxNumActiveConsumers=20

  • is present and not commented out in the file

  • $INSTALLATION_DIR/mq/var/instances/imqbroker/props/config.properties

  • (increase the number 20 if you have more than that number of applications of the same kind on the same bitarchive replica, for instance more than 20 bitarchiveapplications)
  • Set max producers to 100. You add the following line

  • imq.autocreate.destination.maxNumProducers=100

  • in the file

  • $INSTALLATION_DIR/mq/var/instances/imqbroker/props/config.properties 

  • If you get an error like this:

  • Producer can not be added to destination PROD_COMMON_MONITOR [Queue], limit of 100 producers would be exceeded 

  • in the JMS broker log, you need to increase this value.

Starting and stopping JMS

The broker is started directly in this way: 

   $INSTALLATION_DIR/mq/bin/imqbrokerd -reset store -tty &

The sysadmin would maybe like to start the broker on machine startup by inserting the statement above into the /etc/rc.d/rc.local

The broker is stopped in this way: 

logon on machine as root
find processid for the broker (ps auxw | grep imqbrokerd)
kill -9 $IMQ_PROCESSID

Alternatively press Crtl-c, if the terminal where the broker was started, is still available

You can test that JMS broker is alive by telnetting to its port, where it will give some technical information in reply: 

[svc@udvikling kb-dev-adm-001.kb.dk]$ telnet localhost 7676
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
101 imqbroker 4.1
portmapper tcp PORTMAPPER 7676 [sessionid=1729683678303517696]
cluster_discovery tcp CLUSTER_DISCOVERY 46760
jmxrmi rmi JMX 0 [url=service:jmx:rmi://udvikling.kb.dk/stub/rO0...Hg=]
admin tcp ADMIN 46763
jms tcp NORMAL 46762
cluster tcp CLUSTER 46764
.
Connection closed by foreign host.

To run JMS client applications, include the following jar files in the classpath : 

   $INSTALLATION_DIR/mq/lib/jms.jar  $INSTALLATION_DIR/mq/lib/imq.jar

Create a passfile named '.imq_passfile' (used when emptying JMS queues): 

   imq.imqcmd.password=REPLACE_WITH_PASSWORD

How to empty queues

log on as root to the server, where the JMS broker is installed. The following assumes that the JMS environmentName is PROD, and that JMS password file resides in ~root/.imq_passfile: 

export JMS_ENV=PROD
export MQ_HOME=/usr/local
# imqcmd using -u admin -passfile ~/.imq_passfile
$MQ_HOME/bin/imqcmd list dst -t q -u admin -passfile ~/.imq_passfile | grep ^${JMS_ENV}_ | cut -f1 -d\ |xargs -r -n 1 $MQ_HOME/bin/imqcmd destroy dst -t q -u admin -passfile ~/.imq_passfile -f -n
$MQ_HOME/bin/imqcmd list dst -t t -u admin -passfile ~/.imq_passfile | grep ^${JMS_ENV}_ | cut -f1 -d\ |xargs -r -n 1 $MQ_HOME/bin/imqcmd destroy dst -t t -u admin -passfile
~/.imq_passfile -f -n"

How to allocate additional JMS broker memory

export MQ_HOME=/usr/local
$MQ_HOME/mq/bin/imqbrokerd -vmargs "-Xms256m -Xmx512m" -reset store -tty & 
#which adds min 256Mb and max 512MB heap space

Installing and configuring FTP

If you decide to use FTPRemote for file transfer in the NetarchiveSuite, you need to install and start one or more FTP servers, before you begin the installation of the NetarchiveSuite. Any brand of FTP-servers will probably do, but we have good experience with Proftpd.

You can download Proftpd from http://www.proftpd.org/. We are using version 1.2.10, but any recent non-beta version will probably do.

The text below shows part of the proftpd.conf needed by NetarchiveSuite. Other parameters in proftpd.conf may be left with their default values. 

# Port 21 is the standard FTP port.
Port                            21
# Umask 022 is a good standard umask to prevent new dirs and files
# from being group and world writable.
Umask                           022
# To prevent DoS attacks, set the maximum number of child processes
# to 30.  If you need to allow more than 30 concurrent connections
# at once, simply increase this value.  Note that this ONLY works
# in standalone mode, in inetd mode you should use an inetd server
# that allows you to limit maximum number of processes per service
# (such as xinetd).
MaxInstances                    30
# Set the user and group under which the server will run.
User                            nobody
#Group                          nogroup
Group                           nobody
# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
#DefaultRoot ~
# Normally, we want files to be overwriteable.
## This is necessary to allow the append operation
AllowOverwrite          on
AllowStoreRestart on
# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
  DenyAll
</Limit>
# This enables or disables the PAM authentication module.
# The default is 'on'.
#AuthPAM off

If you want to have the FTP-server use a specific directory for uploading files, e.g. ~/ftp, you can use add the configuration 

DefaultChdir ~/ftp

If the ~/ftp does not exist, the server will fallback to the "~".

Starting and stopping a Proftpd server

Log as root on to the server, where Proftpd is installed, and the following command will start the FTP-server 

/usr/local/sbin/proftpd

The following will kill the FTP-server. 

killall -9 proftpd

Appendix B : Starting automatically

edit

This manual contains the description about how to make the applications start automatically when the operating system is starting.

Currently, when a computer is rebooted, the applications has to be started manually. This descripes how to make the operating systems start the applications during startup.

Linux

Log in as administrator. Create following script in '/etc/init.d/' (the name of the script will be refered to as netarkiv): 

# chkconfig: 345 80 20
# description: netarkiv

[ -x /home/USERNAME/ENV_NAME/conf/startall.sh ]  || exit 0
case $1 in
    start)
        su - netarkiv -c 'ENV_NAME/conf/startall.sh'
        ;;
    stop)
        su - netarkiv -c 'ENV_NAME/conf/killall.sh'
        ;;
    *)
        echo "Usage: $0 { start | stop }"
        exit 1
esac

Where USERNAME is the name of the user for the installation, and ENV_NAME is the environment name for NetarchiveSuite (defined in the configuration file).

The following command has to be run for the netarkiv script to be run during start-up and shut-down of Linux: 

chkconfig --add netarkiv

The script can also be run manually, by the commands: 

service netarkiv stop
service netarkiv start

Windows

This is an example of how to make Windows 2003 Server automatically call a script during start-up.

The first thing to do is to create a script for restarting the application. This has to be done, since it might not have closed correctly last time (e.g. power-failure, spontanious reboot, etc.). Therefore the kill scripts has to be run before the applications can be started again.

Create a restart script (refered to as restart.bat) with the following content: 

 cmd /c killall.bat
 cmd /c startall.bat

Create the service.

  • Install Microsoft Resource Kit Windows 2003 Server.

  • Run the program RkTools.exe, and install with standard settings.

  • Open a Command Prompt, and go the to directory where the Resource Kit has been installed (e.g. C:\Program Files\Windows Resource Kits\Tools).

  • Install a service with the following command Instsrv <ServiceName> <path to resource kit>\srvany.exe (e.g. Instsrv BitApp "C:\Program Files\Windows Resource Kits\Tools\srvany.exe").

  • Open the registration database wit regedit, and find the service through the path HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\<SercviceName>.

  • Make sure that the start value is 2 (starting automatically).

  • Create a new 'Key' called Parameters.

  • In this 'Key' create a new 'String Value' called Application, which contains the complete path to the bat-script (e.g. c:\users\USERNAME\ENV_NAME\conf\restart.bat).

  • Also within the 'Key' create another 'String Value' called AppDirectory, which should contain a path to the directory where the bat-script is placed (e.g. c:\users\USERNAME\ENV_NAME\conf).

Now the application should automatically start during Windows startup.

Easy Installation of NetarchiveSuite

RunNetarchiveSuite.sh

deploy_example_one_machine.xml

The first script is a simple script for doing all the steps during deployment. It takes a NetarchiveSuite package ('.zip'), a configuration file (the second file), and a temporary installation directory as arguments (in the given order).

In the configuration file all the applications are placed on one machine (e.g. the current machine, localhost). This gives the same kind of instance as the QuickStart, though this is configured and installed through the Deploy module. If run directly it is installed and run from the directory /home/test/USER.

Below, you find other deploy examples. ( They have to be modfied to your environment)

E.g. 

#stop the Quickstart
cd /home/test/netarchive/scripts/simple_harvest/killhard.sh
cd /home/test/netarchive
bash RunNetarchiveSuite.sh NetarchiveSuite.zip deploy_example_one_machine.xml USER/
#if you have not setup your ssh keygen correctly, you need to login some times before the installation finish successfully

The script creates a "USER" folder in e.g. /home/test , which contains e.g. methods for starting and stopping NetarchiveSuite and starts the whole NetarchiveSuite.

Examples of deploy configuration files

In the following are two examples of configuration files for deploy. The first two requires adaptation to your own system before use.

deploy_example.xml

The instance with two replicas divided over two physical locations. Each physical locations contain several machines. Bitarchive machines, harvester machine and viewerproxy machine. Only one physical location has an administator machine, which contains the GUI application, the Bitarchive monitors and the arc repository.

deploy_example_single.xml

This is the instance with only one replica and one physical location. It is very close to the first example, just with one replica removed.

A running HW/SW setup example from June 2009

Installation Manual 3.8/AppendixC/HW SW KB SB prod june 2009

Installation Manual 3.8/Appendices (last edited 2010-08-16 10:24:47 by localhost)