Differences between revisions 1 and 37 (spanning 36 versions)
⇤ ← Revision 1 as of 2009-05-26 12:16:29 →
Size: 7543
Editor: KaareChristiansen
Comment: Created by the PackagePages action.
← Revision 37 as of 2010-08-16 10:24:29 → ⇥
Size: 8544
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
[[Action(edit)]] <<Action(edit)>>
Line 4: Line 4:
For a quick start, we have prepared a bash script that starts all the necessary components on one machine. We will use this script throughout this quickstart manual to allow you to get a feel for what the system can do and how it works without having to deal with issues of distributing to other servers. For a quick start, we have prepared a bash shell script that starts all the necessary components on one machine. We will use this script throughout this quickstart manual to allow you to get a feel for what the system can do and how it works without having to deal with issues of distributing to other servers.
Line 13: Line 13:
 . Newer versions of Java may work, but have not been tested. Older versions of Java will '''not''' work correctly. The latest download version of Sun Java 6 SE is "JDK 6 Update 13" (19 May 2009).  . Newer versions of Java may work, but have not been tested. Older versions of Java will '''not''' work correctly. The latest download version of Sun Java 6 SE is "JDK 6 Update 16" (01 September 2009).
Line 19: Line 19:
 . you should then see  . you should then see something like
Line 27: Line 27:
Download and unzip of the newest release is described here Download of the newest release is described here
Line 33: Line 33:
 * follow the registration and download instructions on ["Get NetarchiveSuite"] and save the download file to the directory you created earlier
 * go to the directory
 . {{{ $ cd ~/netarchive }}}
 * unzip the binary package
 . {{{ $ unzip NetarchiveSuite*.zip }}}
 * if downloading the trunk:		  * follow the registration and download instructions on [[Get NetarchiveSuite]] and save the download file to the directory you created earlier
 * there should now be a {{{NetarchiveSuite*.zip}}} file in the installation directory, e.g. {{{ ~/netarchive }}}
Note: Instead of downloading a !NetarchiveSuite.zip you can also build itself from the SVN trunk:

 . {{{ $ svn checkout --username developername https://gforge.statsbiblioteket.dk/svn/netarchivesuite/trunk }}}
Line 40: Line 39:
 . {{{ $ ant warfiles }}}  . {{{ $ ant releasezipball }}}
Line 44: Line 43:
## But the JMS application listen to port 7676 so this must be available before starting the simple harvest program. ## But the JMS application listen to port 7676 so this port must be available before starting the quickstart system.
Line 67: Line 66:
 * stop imqbroker by pressing {{{ control-C }}}  * stop the imqbroker by pressing {{{ control-C }}}
Line 74: Line 73:
To start it, do the following:

 . {{{ $ cd ~/netarchive }}}
 . {{{ $./mq/bin/imqbrokerd &}}}
When you are finished using the quickstart system, you kill the jmsbroker by:

 . Find processid for the broker (ps auxw | grep imqbrokerd)
 . export IMQ_PROCESSSID=foundprocessid
 . {{{ kill -9 $IMQ_PROCESSID }}}
Line 75: Line 83:
Assuming the !NetarchiveSuite files were unzipped to the directory {{{~/netarchive}}} as described above, you must do the following to configure the !NetarchiveSuite for your system: The Quick Start guide assumes you have a username "test" on the machine and will be installing NetarchiveSuite from /home/test/netarchive/
Line 77: Line 85:
For the simple harvest setup, the startup script needs to know a few paths. You can either do this by exporting the environment variables in your shell or by changing the harvest script. Assuming a releasezipball of !NetarchiveSuite {{{NetarchiveSuite*.zip }}} is available in the directory {{{~/netarchive}}}, you must do the following to configure the !NetarchiveSuite for your system:
Line 79: Line 87:
The following describes how to set the environment variables:  * Download following attached files to e.g. /home/test/netarchive:
[[http://netarchive.dk/suite/Installation_Manual_3.10/AppendixC?action=AttachFile&do=get&target=RunNetarchiveSuite.sh|RunNetarchiveSuite.sh]]
Line 81: Line 90:
 * set `JAVA` to point to your java installation directory, e.g. `/usr/java/jdk1.6.0_07`
 . {{{ $ export JAVA=/usr/java/jdk1.6.0_07 }}}
 * set `JAVA_HOME` to point to your java installation directory, e.g. `/usr/java/jdk1.6.0_07`
 . {{{ $ export JAVA_HOME=/usr/java/jdk1.6.0_07 }}}
 * set `IMQ_JAVAHOME` to point to your java installation directory, e.g. `//usr/java/jdk1.6.0_07`
 . {{{ $ export IMQ_JAVAHOME=/usr/java/jdk1.6.0_07 }}}
 * set `IMQ` to the full path of the executable of the Sun JMS broker, e.g. {{{~/netarchive/mq/bin/imqbrokerd}}}
 . {{{ $ export IMQ=~/netarchive/mq/bin/imqbrokerd }}}
Note that these environment variables need to be set every time you log in, if you wish to use the quickstart scripts. You can also edit the file {{{harvest.sh}}} and set the variables there, if you wish the settings to be persistent.		 [[http://netarchive.dk/suite/Installation_Manual_3.10/AppendixC?action=AttachFile&do=get&target=deploy_example_one_machine.xml|deploy_example_one_machine.xml]]
Line 91: Line 92:
=== Starting the system ===
'''Note:''' Starting the script clears all data from previous runs. The clearing of previous data is disabled by setting environment variable KEEPDATA: {{{ export KEEPDATA=1 }}}.		 The first script is a simple script for doing all the steps during deployment. It takes a !NetarchiveSuite package ('.zip'), a configuration file (the second file), and a temporary installation directory as arguments (in the given order).
Line 94: Line 94:
To start the program do the following: In the configuration file all the applications are placed on one machine (e.g. the current machine, ~+{{{localhost}}}+~). If run directly it is started/stopped from the directory ~+{{{/home/test/netarchive/USER}}}+~ and installed/run in e.g. localhost:/home/test/USER. It assumes, that you want to run this as user 'test'. So you need to have 'test' user on the current machine. If installation user is different from the 'test' user, remember to check, that a Sun JVM is in the path (instead of GNU java compiler, that is default with some Linux'es.). If you allready have a USER installation, then remember also, that the existing bitarchive, database and admin.data files will be untouched. You must explicit remove any previous USER installation, if you want a clean empty installation.
Line 96: Line 96:
 * start a terminal login to the linux system as a ordinary user in a {{{bash}}} shell
 * set the environment variables `JAVA`, `IMQ`, and `IMQ_JAVAHOME` as described above, if they are not set already.
 * go to the simple harvest directory and run program
 . {{{$ cd ~/netarchive/scripts/simple_harvest}}}
 . {{{$ ./harvest.sh}}}
 . note that it will start several xterm windows, one for each application running (including JMS). All the xterms will display the logs of their applications (see figure below) -- you can follow how they start up and when they are ready to use.
 * if running on trunk, make sure to have generated jar files first
 . {{{$ ant jarfiles }}}
 * wait until the xterm with the title {{{ HarvestDefinition }}} has stopped writing messages. It should include towards the end one saying {{{ "Scheduler running every 60 seconds" }}}
A typical set of windows looks like this:

 . attachment:Screenshot-1.png
 * start a web browser by e.g. {{{ $ mozilla }}} Note that it is important that the browser is started on the same machine as the simple harvest script is run on.
 * write url in the started browser http://localhost:8073/HarvestDefinition
 . You can now see the webinterface in the browser
=== Stopping the system ===
When you are done experimenting, you can use the scripts described below to stop the programs. After that, it should be possible to restart from scratch by following the instructions described above under section "Starting simple_harvest version".

Do the following:		 E.g. (you should use "USER" as the installation name to make things easy)
Line 117: Line 99:
$ cd ~/netarchive/scripts/simple_harvest
$ ./killhard.sh		 cd /home/test/netarchive
bash RunNetarchiveSuite.sh NetarchiveSuite.zip deploy_example_one_machine.xml USER
#if you have not setup your ssh keygen correctly, you need to login some times before the installation finish successfully. You must also have permission to ssh and scp to localhost ( try e.g "ssh localhost" and "scp somefile localhost:")
Line 120: Line 103:
The script creates a deployment folder named "USER" in e.g. /home/test/netarchive, which contains methods for starting and stopping NetarchiveSuite, and starts the whole NetarchiveSuite. It deploys the installation locally to /home/test/USER.

It is also necessary to have a ftp server running with permission for this user (username:test & password: test123 - to be changed in the deploy-file: deploy_example_one_machine.xml under "<remoteFile>" (before running the build-script)

 * start a web browser by e.g. {{{ $ mozilla }}} Note that it is important that the browser is started on the same machine as the simple harvest script is run on
 * setup the browser to proxy on port 8070 and exclude localhost e.g. in mozilla:
{{{
Choose in the mozilla toolbar:
Edit->Preferences->Advanced->Proxies
Checkmark:
Manual Proxy Configuration
and add:
Proxy: localhost
Port: 8070
No Proxy for: localhost
}}}
 * Write following url in the started browser http://localhost:8074/HarvestDefinition
 * You can now see the webinterface in the browser
 . You can now create, run and browse according to the following or the User Manual
 * if you want to stop and start the entire NAS system, then
{{{
cd /home/test/netarchive/USER
./killall_NATIONAL_LIBRARY.sh
./startall_NATIONAL_LIBRARY.sh
}}}
 * If you want to try other deploy examples, then go to "Examples of deploy configuration files" in the Installation Manual

Download and installation

edit

For a quick start, we have prepared a bash shell script that starts all the necessary components on one machine. We will use this script throughout this quickstart manual to allow you to get a feel for what the system can do and how it works without having to deal with issues of distributing to other servers.

Base system required

For the quick startup, NetarchiveSuite requires

  • a Linux system.
  • Note that for the quickstart, you must be able to run a browser on the machine that you run the system on - this is an artifact of the quickstart system and is not the case in the full system. Root access is not required.
  • Running Ant application - Java based build tool like make
  • Sun Java SE (Standard Edition) JDK version 1.6.0_07 running on the Linux system.

  • Newer versions of Java may work, but have not been tested. Older versions of Java will not work correctly. The latest download version of Sun Java 6 SE is "JDK 6 Update 16" (01 September 2009).

To check that you have the right version of Java do the following

  • start a terminal login to the linux system as a ordinary user
  • check java version is version 1.6.0_07 (or higher) by writing:

  • $ java -version

  • you should then see something like 
  • linux>java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b06)
Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)

Downloading

Download of the newest release is described here

  • start a terminal login to the linux system as a ordinary user in a bash shell

  • make a directory for the download e.g. directory  ~/netarchive 

  •  $ mkdir ~/netarchive 

  • start a web browser, e.g. Firefox

  • follow the registration and download instructions on Get NetarchiveSuite and save the download file to the directory you created earlier

  • there should now be a NetarchiveSuite*.zip file in the installation directory, e.g.  ~/netarchive 

Note: Instead of downloading a NetarchiveSuite.zip you can also build itself from the SVN trunk:

  •  $ svn checkout --username developername https://gforge.statsbiblioteket.dk/svn/netarchivesuite/trunk 

  •  $ cd trunk 

  •  $ ant releasezipball 

JMS

NetarchiveSuite uses JMS for inter-process communication. JMS is the Java Messaging Service, which provides asynchronous communication between processes. You do not need any knowledge of JMS to use NetarchiveSuite. However you need to make sure that there are not already JMS brokers running on your system using PORT 8100.

Currently only the open-source version of Sun's JMS implementation is supported, since some functionality of other implementations does not match our assumptions well.

To download and install it, do the following:

  • open this link in a browser window https://mq.dev.java.net/downloads.html

  • click the Linux Link under version 4.1 Binary Downloads to download a file mq4_1-binary-Linux_X86-20070816.jar (or later)

  • save the download file to the created directory you created earlier e.g.  ~/netarchive 

  • go to the directory

  •  $ cd ~/netarchive 

  • unpack the jar file (this creates a directory mq and three files with licensing information)

  •  $ jar xvf mq4_1-binary-Linux_X86-20070816.jar 

  • run imqbroker in order to create settings file

  •  $ chmod +x ./mq/bin/imqbrokerd 

  •  $./mq/bin/imqbrokerd 

  • For this to work, it may be necessary to set the IMQ_JAVAHOME environment variable to point to your java installation directory, e.g. /usr/java/jdk1.6.0_07:  $ export IMQ_JAVAHOME=/usr/java/jdk1.6.0_07 

  • check that  imqbrokerd  starts and that the last message is  "Broker <localhost>:7676 ready" 

  • stop the imqbroker by pressing  control-C 

  • edit settings to allow for enough listeners to a queue by doing

    • edit  ~/netarchive/mq/var/instances/imqbroker/props/config.properties 

    • uncomment and specify count=20 for listeners by changing line

    • "#            imq.autocreate.queue.maxNumActiveConsumers" 

    • ->

    • "            imq.autocreate.queue.maxNumActiveConsumers=20"

To start it, do the following:

  •  $ cd ~/netarchive 

  •  $./mq/bin/imqbrokerd &

When you are finished using the quickstart system, you kill the jmsbroker by:

  • Find processid for the broker (ps auxw | grep imqbrokerd)
  • export IMQ_PROCESSSID=foundprocessid

  •  kill -9 $IMQ_PROCESSID 

Configuration

The Quick Start guide assumes you have a username "test" on the machine and will be installing NetarchiveSuite from /home/test/netarchive/

Assuming a releasezipball of NetarchiveSuite NetarchiveSuite*.zip  is available in the directory ~/netarchive, you must do the following to configure the NetarchiveSuite for your system:

  • Download following attached files to e.g. /home/test/netarchive:

RunNetarchiveSuite.sh

deploy_example_one_machine.xml

The first script is a simple script for doing all the steps during deployment. It takes a NetarchiveSuite package ('.zip'), a configuration file (the second file), and a temporary installation directory as arguments (in the given order).

In the configuration file all the applications are placed on one machine (e.g. the current machine, localhost). If run directly it is started/stopped from the directory /home/test/netarchive/USER and installed/run in e.g. localhost:/home/test/USER. It assumes, that you want to run this as user 'test'. So you need to have 'test' user on the current machine. If installation user is different from the 'test' user, remember to check, that a Sun JVM is in the path (instead of GNU java compiler, that is default with some Linux'es.). If you allready have a USER installation, then remember also, that the existing bitarchive, database and admin.data files will be untouched. You must explicit remove any previous USER installation, if you want a clean empty installation.

E.g. (you should use "USER" as the installation name to make things easy) 

cd /home/test/netarchive
bash RunNetarchiveSuite.sh NetarchiveSuite.zip deploy_example_one_machine.xml USER
#if you have not setup your ssh keygen correctly, you need to login some times before the installation finish successfully. You must also have permission to ssh and scp to localhost ( try e.g "ssh localhost" and "scp somefile localhost:")

The script creates a deployment folder named "USER" in e.g. /home/test/netarchive, which contains methods for starting and stopping NetarchiveSuite, and starts the whole NetarchiveSuite. It deploys the installation locally to /home/test/USER.

It is also necessary to have a ftp server running with permission for this user (username:test & password: test123 - to be changed in the deploy-file: deploy_example_one_machine.xml under "<remoteFile>" (before running the build-script)

  • start a web browser by e.g.  $ mozilla  Note that it is important that the browser is started on the same machine as the simple harvest script is run on

  • setup the browser to proxy on port 8070 and exclude localhost e.g. in mozilla: 

Choose in the mozilla toolbar:
Edit->Preferences->Advanced->Proxies
Checkmark:
Manual Proxy Configuration
and add:
Proxy: localhost
Port: 8070
No Proxy for: localhost

  • Write following url in the started browser http://localhost:8074/HarvestDefinition

  • You can now see the webinterface in the browser
  • You can now create, run and browse according to the following or the User Manual
  • if you want to stop and start the entire NAS system, then 

cd /home/test/netarchive/USER
./killall_NATIONAL_LIBRARY.sh
./startall_NATIONAL_LIBRARY.sh
  • If you want to try other deploy examples, then go to "Examples of deploy configuration files" in the Installation Manual

Quick Start Manual 3.10/Download and installation (last edited 2010-08-16 10:24:29 by localhost)