NetarchiveSuite
Download and installation
For a quick start, we have prepared a bash shell script that starts all the necessary components on one machine. We will use this script throughout this quickstart manual to allow you to get a feel for what the system can do and how it works without having to deal with issues of distributing to other servers.
Base system required
For the quick startup, NetarchiveSuite requires
- a Linux system.
- Note that for the quickstart, you must be able to run a browser on the machine that you run the system on - this is an artifact of the quickstart system and is not the case in the full system. Root access is not required.
- Running Ant application - Java based build tool like make
- Sun Java SE (Standard Edition) JDK version 1.6.0_19 (or later) running on the Linux system.
- Newer versions of Sun Java 1.6 will probably work, but have not been tested. The latest download version of Sun Java 6 SE is "JDK 6 Update 24" (03 February 2011).
To check that you have the right version of Java do the following
- start a terminal login to the linux system as a ordinary user
- check java version is version 1.6.0_19 (or higher) by writing:
$ java -version
- you should then see something like
linux>java -version java version "1.6.0_19" Java(TM) SE Runtime Environment (build 1.6.0_19-b04) Java HotSpot(TM) Server VM (build 16.2-b04, mixed mode)
Downloading
Download of the newest release is described here
- start a terminal login to the linux system as a ordinary user in a bash shell
make a directory for the download e.g. directory ~/netarchive
$ mkdir ~/netarchive
- start a web browser, e.g. Firefox
follow the registration and download instructions on ["Get NetarchiveSuite"] and save the download file to the directory you created earlier
there should now be a NetarchiveSuite*.zip file in the installation directory, e.g. ~/netarchive
Note: Instead of downloading a NetarchiveSuite.zip you can also build itself from the SVN trunk:
$ svn checkout --username developername https://gforge.statsbiblioteket.dk/svn/netarchivesuite/trunk
$ cd trunk
$ ant releasezipball
JMS
NetarchiveSuite uses JMS for inter-process communication. JMS is the Java Messaging Service, which provides asynchronous communication between processes. You do not need any knowledge of JMS to use NetarchiveSuite. However you need to make sure that there are not already JMS brokers running on your system using PORT 7676.
Currently only the open-source version of Sun's JMS implementation is supported, since some functionality of other implementations does not match our assumptions well.
To download and install it, do the following:
open this link in a browser window http://mq.dev.java.net/downloads.html
- click the Linux Link under version 4.4 Binary Downloads to download a file openmq4_4-installer-Linux_X86.zip (or later version)
save the download file to the created directory you created earlier e.g. ~/netarchive
- go to the directory
$ cd ~/netarchive
- unpack the zip file (this creates a directory openmq4_4-installer), and run the X-Windows installer. The installer ask you to choose an install-home (choose ~/netarchive/MessageQueue), and a JDK.
$ unzip openmq4_4-installer-Linux_X86.zip
$ cd openmq4_4-installer; ./installer
- Set necessary environment variables: IMQ_HOME, IMQ_VARHOME, IMQ_ETCHOME)
$ export IMQ_HOME=$HOME/netarchive/MessageQueue/mq; export IMQ_VARHOME=$HOME/netarchive/MessageQueue/var/mq; export IMQ_ETCHOME=$HOME/netarchive/MessageQueue/etc/mq
- run imqbroker in order to create settings file
$ chmod +x ./MessageQueue/mq/bin/imqbrokerd
$./MessageQueue/mq/bin/imqbrokerd
check that imqbrokerd starts and that the last message is "Broker <localhost>:7676 ready"
stop the imqbroker by pressing control-C
- edit settings to allow for enough listeners to a queue by doing
edit ~/netarchive/MessageQueue/var/mq/instances/imqbroker/props/config.properties
- uncomment and specify count=20 for listeners by changing line
"# imq.autocreate.queue.maxNumActiveConsumers"
->
" imq.autocreate.queue.maxNumActiveConsumers=20"
To start it, do the following:
$ cd ~/netarchive
$./MessageQueue/mq/bin/imqbrokerd &
Configuration
Assuming a releasezipball of NetarchiveSuite NetarchiveSuite*.zip is available in the directory ~/netarchive, you must do the following to configure the NetarchiveSuite for your system:
- Download following attached files to ~/netarchive:
The first script is a simple script for doing all the steps during deployment. It takes a NetarchiveSuite package ('.zip'), a configuration file (the second file), and a temporary installation directory as arguments (in the given order). The different ports used by the application for communication are included in the deploy_standalone_example.xml file.
In the configuration file all the applications are placed on one machine (e.g. the current machine, localhost). When the installation script is run from the indicated deploy directory ~/netarchive/$DEPLOY_DIR and it will be installed into the localhost:/home/test/$INSTALL_DIR directory (using ssh). It assumes that user 'test' already exists. Remember to check, that a Sun JVM is in the path for the test (instead of GNU java compiler, that is default with some Linux'es.). If you already have a Quickstart installation the existing bitarchive, database and admin.data files will be untouched. You must explicit remove any previous installation, if you want a clean empty installation.
We will use 'deploy' as the $DEPLOY_DIR and 'Quickstart' as the $INSTALL_DIR in the rest of the guide. Remember to edit the deploy_standalone_example.xml, so the setting environmentName is set to Quickstart. The default value is USER.
cd ~/netarchive bash RunNetarchiveSuite.sh NetarchiveSuite.zip deploy_standalone_example.xml deploy/ #if you have not setup your ssh keygen correctly, you need to login some times before the installation finish successfully. You must also have permission to ssh and scp to localhost ( try e.g "ssh localhost" and "scp somefile localhost:")
The script creates a deployment folder named "Quickstart" in e.g. /home/test/Quickstart, which contains methods for starting and stopping NetarchiveSuite, and starts the whole NetarchiveSuite. The files to run the installation will be placed in the ~/netarchive/deploy dir
start a web browser by e.g. $ firefox Note that it is important that the browser is started on the same machine as the simple harvest script is run on
- setup the browser to proxy on port 8070 and exclude localhost and the hostname (used by the Heritrix GUI) e.g. in firefox:
Choose in the firefox toolbar: Edit->Preferences->Advanced->Network->Settings Checkmark: Manual Proxy Configuration and add: Proxy: localhost Port: 8070 No Proxy for: localhost, kb-test-way-001.kb.dk
Write following url in the started browser http://localhost:8074/HarvestDefinition
- You can now see the webinterface in the browser
- You can now create, run and browse according to the following or the User Manual
- if you want to stop and start the entire NAS system, then
cd /home/test/Quickstart ./killall_NATIONAL_LIBRARY.sh ./startall_NATIONAL_LIBRARY.sh
- If you want to try other deploy examples, then go to "Examples of deploy configuration files" in the Installation Manual