== MOVED TO CONFIGURATION MANUAL ==

[[Action(edit)]]

=== Configure notifications ===

!NetarchiveSuite can send notifications of serious system warnings or failures to the system-owner by email.  This is implemented using the Notifications plugin, see Appendix C.
Several settings in the settings.xml can be changed for this to work:

The setting {{{settings.common.notifications.receiver}}} (recipient of notifications), {{{settings.common.notifications.sender}}} (the official sender of the email, and receiver of any bounces), and {{{settings.common.mail.server}}} (the proper mail-server to use): 

{{{
<settings>
  <common>
    <notifications>
       <!-- Which class to instantiate to handle error notifications -->
       <class>dk.netarkivet.common.utils.EmailNotifications</class>
       <!-- The receiver of emails -->
       <receiver>example@netarkivet.dk</receiver>
       <!-- The stated sender of emails  (and receiver of bounces)-->
       <sender>example@netarkivet.dk</sender>
    </notifications>
    <!-- Settings for sending email. Currently mail is only used for email notifications. -->
    <mail>
      <!-- The email server to use -->
      <server>examplesmtpserver.netarkivet.dk</server>
    </mail>
  </common>
</settings>
}}}

Alternatively, the class {{{dk.netarkivet.common.utils.PrintNotifications}}} can be used. This will simply print the notifications to stderr on the terminal.

{{{
<settings>
  <common>
    <notifications>
      <!-- Which class to instantiate to handle error notifications -->
      <class>dk.netarkivet.common.utils.PrintNotifications</class>
    </notifications>
  </common>
</settings>
}}}

=== Select a file datatransfer method ===

You can currently choose between FTP, HTTP, or HTTPS as the filetransfer method.  The HTTP transfer method uses only a single copy per transfer, while the FTP method first copies the file to an FTP server and then copies it from there to the receiving side.  Additionally, the HTTP transfer method reverts to simple filesystem copying whenever possible to optimize transfer speeds.  However, to use HTTP transfers you must have ports open into most machines, which some may consider a security risk. The HTTPS transfer method meets this problem by having the HTTP communication secured and encrypted. To use the HTTPS transfer method you will need to generate a certificate that is needed to contact the embedded HTTPS server.

The FTP method requires one or more FTP-servers installed. (See Appendix A for further details). 
The XML below is an example settings.xml, in which you have to replace serverName, userName, userPassword with proper values.  This must be set for all applications to use FTP remote files.  

{{{
<settings>
  <common>
    <remoteFile>
      <!-- The class to use for RemoteFile objects. -->
      <class>dk.netarkivet.common.distribute.FTPRemoteFile</class>
      <!-- The default FTP-server used -->
      <serverName>hostname</serverName>
      <!-- The FTP-server port used -->
      <serverPort>21</serverPort>
      <!-- The FTP username -->
      <userName>exampleusername</userName>
      <!-- The FTP password -->
      <userPassword>examplepassword</userPassword>
      <!-- The number of times FTPRemoteFile should try before giving up
           a copyTo operation. We augment FTP with checksum checks. -->
      <retries>3</retries>
    </remoteFile>
  </common>
</settings>
}}}

It is possible to use more than one FTP server, but each application can only use one.  The FTP server that is used for a particular transfer is determined by the application that is sending a file.  If you want to use more than one FTP-server, you must use different settings-files, or define the serverName and possibly also the userName and userPassword when starting the applications on the commandline with 
{{{   -Dsettings.common.remoteFile.serverName=FTP-server1 
   -Dsettings.common.remoteFile.userName=ftpUser 
   -Dsettings.common.remoteFile.userPassword=ftpPassword}}}

Using HTTP as filetransfer method, you need to reserve a HTTP port on each machine per application.  You can set this port on an application level on the commandline: 
{{{-Dsettings.common.remoteFile.port=5442}}}

The following XML shows the the corresponding syntax in the {{{settings.xml}}} file:
{{{
<settings>
  <common>
    <remoteFile>
      <!-- The class to use for RemoteFile objects. -->
      <class>dk.netarkivet.common.distribute.HTTPRemoteFile</class>
      <!-- Port for embedded HTTP server -->
      <port>5442</port>
    </remoteFile>
  </common>
</settings>
}}}

Using the HTTPS file transfer method, you first need to generate a certificate that is used for communication. You can do this with the {{{keytool}}} application distributed with Java 5.

Run the following command:

{{{
keytool -alias NetarchiveSuite -keystore keystore -genkey
}}}

It should the respond with the following:

{{{
Enter keystore password:  
}}}

Enter the password for the keystore.

The keytool will now prompt you for the following information
{{{
What is your first and last name?
  [Unknown]:  
What is the name of your organizational unit?
  [Unknown]:  
What is the name of your organization?
  [Unknown]:  
What is the name of your City or Locality?
  [Unknown]:  
What is the name of your State or Province?
  [Unknown]:  
What is the two-letter country code for this unit?
  [Unknown]:  
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
  [no]:  
}}}

answer all the questions, and end with "yes".

Finally you will be asked for the certificate password.

{{{
Enter key password for <NetarchiveSuite>
        (RETURN if same as keystore password): 
}}}

Answer with a password for the certificate.

You now how a file called {{{keystore}}} which contains a certificate. This keystore needs to be available for all !NetarchiveSuite applications, and referenced from settings as the following example shows:

{{{
<settings>
  <common>
    <remoteFile xsi:type="httpsremotefile">
      <!-- The class to use for RemoteFile objects. -->
      <class>dk.netarkivet.common.distribute.HTTPSRemoteFile</class>
      <!-- The port for the remote file transfers -->
      <port>8300</port>
      <!-- The keystore -->
      <certificateKeyStore>path/to/keystore</certificateKeyStore>
      <!-- The keystore passwd -->
      <certificateKeyStorePassword>testpass</certificateKeyStorePassword>
      <!-- The key password-->
      <certificatePassword>testpass2</certificatePassword>
    </remoteFile>
  </common>
</settings>
}}}

To keep your environment secure, you should make sure that the keystore and settings file ''only'' are readable for the user running the application.

=== Configure job-generation ===

The scheduling takes place every one minute, unless the previous scheduling is not finished yet. The scheduling interval cannot be changed.
Scheduling amounts to searching for active harvestdefinitions, that is ready to have jobs generated, and subsequently submitted for harvesting.
The job-generation procedure are governed by a set of settings prefixed by ''settings.harvester.scheduler.''. These settings rule how large your crawljobs are going to be, and how long time they will take to complete.
Note that harvestdefinitions consist of at least one !DomainConfiguration, (containing a Heritrix setup, and a seed-list), and that there are two kinds: Snapshot Harvestdefinitions, and Selective Harvestdefinitions.

During scheduling, each harvest is split into a number of ''crawl jobs''.  This is done to keep Heritrix from using too much memory and to avoid that particularly slow or large domains cause harvests to take longer than necessary.  In the job splitting part of the scheduling, the scheduler partitions a large number of !DomainConfigurations into several crawljobs.  Each crawljob can have only one Heritrix setup, so !DomainConfigurations with different Heritrix setups will be split into different crawljobs.  Additionally, a number of parameters influence what configurations are put into which jobs, attempting to create jobs that cover a reasonable amount of domains of similar sizes.

If you don't want to have the harvests split into multiple jobs, you just need to set each of {{{settings.harvester.scheduler.jobs.maxRelativeSizeDifference}}}, {{{settings.harvester.scheduler.jobs.minAbsoluteSizeDifference}}}, {{{settings.harvester.scheduler.jobs.maxTotalSize}}}, and {{{settings.harvester.scheduler.configChunkSize}}} to a large number, such as MAX_LONG.  Initially, we suggest you don't change these parameters, as the way the work together is subtle. Harvests will always be split in different jobs, though, if they are based on different order.xml templates, or if different harvest limits need to be enforced.

'''settings.harvester.scheduler.errorFactorPrevResult''': Used when calculating expected size of a harvest of some domain during the job-creation process for snapshot harvests. This defines the factor by which we maximally allow domains that have previously been harvested to increase in size, compared to the value we estimate the domain to be. In other words, it defines how conservative our estimates are.  The default value is 10, meaning that the maximum number of bytes harvested is as most 10 times as great as the value we use as expected size.

'''settings.harvester.scheduler.errorFactorBestGuess''': Used when calculating expected size of a harvest of some domain during job-creation process for a snapshot Harvests.  This defines the factor by which we maximally allow domains that have previously been incompletely harvested or not harvested at all to increase in size, compared to the value we estimate the domain to be. In other words, it defines how conservative our estimates are.  The default value is 20, meaning that the maximum number of bytes harvested is as most 20 times as great as the value we use as expected size. This is probably an unreasonable number, it should be reset to 2 for most installations.

'''settings.harvester.scheduler.expectedAverageBytesPerObject''': How many bytes the average object is expected to be on domains where we don't know any better.  This number should grow over time, as of end of 2005 empirical data shows 38000. Default is 38000.

'''settings.harvester.scheduler.maxDomainSize''': Initial guess of #objects in an unknown domain. Default value is 5000

'''settings.harvester.scheduler.jobs.maxRelativeSizeDifference''': The maximum allowed relative difference in expected number of objects retrieved in a single job definition.  Set to MAX_LONG for no splitting.

'''settings.harvester.scheduler.jobs.minAbsoluteSizeDifference''': Size differences for jobs below this threshold are ignored, regardless of the limits for the relative size difference. Set to MAX_LONG for no splitting. Default value is 2000.

'''settings.harvester.scheduler.jobs.maxTotalSize''': When this limit is exceeded no more configurations may be added to a job. Set to MAX_LONG for no splitting. Default value is 2000000

'''settings.harvester.scheduler.configChunkSize''': How many domain configurations we will process in one go before making jobs out of them.  This amount of domains will be stored in memory at the same time.  Set to MAX_LONG for no job splitting. The default value is 10000.

MAX_LONG refers to the number 2^63-1 or 9223372036854775807.

=== Configure Domain Granularity ===

The !NetarchiveSuite software is bound to the concept of Domains, where a Domain is defined as
{{{
"domainname"."tld"
}}}

This concept is useful for grouping harvests with regard to specific domains.

It can be configured what is considered a TLD by changing the settings files. The settings file currently distributed with the !NetarchiveSuite software will list all country-level top-level-domains as "tld"s like ".dk", ".se" and ".no". However, as a proof of concept, for ".uk"-domains, there is listed the pseudo-top-level-domains ".co.uk", ".gov.uk", ".edu.uk" and some more.

Currently, only grouping by domain suffix is supported. A feature request is open for making the domain splitting pluggable. See [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1072 Feature Request 1072].

=== Configure Heritrix process ===

Each harvester runs an instance of Heritrix for each harvest job being executed. It is possible to get access to the Heritrix web user interface for purposes of pausing or stopping a job, examining details of an ongoing harvest or even, if necessary, change an ongoing harvest.  Note that some changes to harvests, especially those that change the scope and limits, may confuse the harvest definition system.  We suggest using the Heritrix UI only for examination and pausing/terminating jobs.

Each harvest ''application'' running requires two ports, one for the user interface and one for JMX.  The JMX port is set by the `settings.harvester.harvesting.heritrix.jmxPort` setting, and does not need to be open to other machines. The user interface port is set by the `settings.harvester.harvesting.heritrix.guiPort` setting, and should be open to the machines that the user interface should be accessible from.  Make sure to have different ports for each harvest application if you're running more than one on a machine.  Otherwise, your harvest jobs will fail when two harvest applications happen to try to run at the same time -- an error that could go unnoticed for a while, but which is more likely to happen exactly in critical situations where more harvesters are needed.

The Heritrix user interface is accessible through a browser using the port specified, e.g. http://my.harvester.machine:8090, and entering the administrator name and password set in the `settings.harvester.harvesting.heritrix.adminName` and `settings.harvester.harvesting.heritrix.adminPassword` settings.  It is also possible to use JConsole to access the JMX interface of the Heritrix process, using one of the passwords in the `conf/jmxremote.password` file.

The final setting for the Heritrix processes is the amount of heap space each process is allowed to use.  Since Heritrix uses a significant amount of heap space for seen URLs and other stuff, it is advisable to keep the `settings.harvester.harvesting.heritrix.heapSize` setting at at least its default setting of 1.5G if there is enough memory in the machine for this (remember to factor in the number of harvesters running on the machine -- swapping will slow the crawl down ''significantly'').

=== Configure web page look ===

The look of the web pages can be changed by changing files in the webpages directory. 
The files are distributed in war-files, which are simply zip-files. They can be unpacked to customize styles, and repacked afterwards using zip. Each of the five war files under webpages corresponds to one section of the web site, as seen in the left-hand menu.  The two PNG files transparent_logo.png and transparent_menu_logo.png are used on the front page and atop the left-hand menu, respectively.  They can be altered to suite your whim, but the width of transparent_menu_logo.png should not be increased so much that the menu becomes overly wide.  The color scheme for each section is set in the netarkivet.css file for that section and can be changed to suit your whim, though we recommend changing them all at the same time to provide a uniform look.

=== Configure security ===

Security in !NetarchiveSuite is mainly defined in the `conf/security.policy` file.  This file controls two main configurations: Which classes are allowed to do anything (core classes), and which classes are only allowed to read the files in the bit archive (third-party batch classes).

To enable the use of the security policy, you will need to launch your applications with the command line options {{{-Djava.security.manager}}} and {{{-Djava.security.policy=conf/security.policy}}}.

==== Core classes ====

For the core classes, we need to identify all the classes that can be involved.  The default security.policy file assumes that the program is started from the root of the distribution.  If that is not the case, the codeBase entries must be changed to match.  The following classes should be included:

 * The dk.netarkivet.* jar files and supporting jar files, located in the `lib` directory.  By default, all files in this directory and its subdirectories are included by the statement

{{{
grant codeBase "file:lib/-" {
  permission java.security.AllPermission;
};
}}}

 * The heritrix jar files and supporting jar files for it, usually located in the `lib/heritrix/lib` directory.  By default, these are included by the above, but in the [:Quick_Start_Manual:QuickStart] system, they are in a separate directory under `scripts/simple_harvest` (this place is included in the default security.policy file).

 * The standard Java classes, which by default are included by the statement

{{{
grant codeBase "file:${java.home}/-" {
  permission java.security.AllPermission;
};
}}}

 * The classes compiled by JSP as part of the web interface.  These classes only exist on the machine(s) that run a web interface, and are found in the directory specified by the `settings.common.tempDir` setting.  The default security file contains entries that assume this directory is `tests/commontempdir`.  Note that an entry is required for each section of the web site:

{{{
grant codeBase "file:tests/commontempdir/Status/jsp/-" {
  permission java.security.AllPermission;
};
}}}

If you change the `settings.common.tempDir` setting, you will need to change this entry, too, or the web pages won't work.

==== Third-party classes ====

The default security.policy file includes settings that allow third-party batch jobs to read the bitarchives set up for the [:Quick_Start_Manual:QuickStart] system.  In a real installation, the bitarchive machines must specify which directories should be accessible and set up permissions for these.  The default setup is:

{{{
grant {
  permission java.util.PropertyPermission "settings.archive.bitarchive.thisLocation", "read";
  permission java.io.FilePermission "${user.home}/netarchive/scripts/simple_harvest/bitarchive1/filedir/*", "read";
  permission java.io.FilePermission "${user.home}/netarchive/scripts/simple_harvest/bitarchive2/filedir/*", "read";
};
}}}

Notice how these permissions are not granted to a specific codebase, but the permissions given are very restrictive: The classes can read files in two explicitly stated directories, and can query for the value of the `settings.archive.bitarchive.thisLocation` setting -- all other settings are off-limits, as is reading and writing other files, including temporary files.  If you wish to allow third-party batch jobs to do more, think twice first -- loopholes can be subtle.