NetarchiveSuite
Wayback Configuration
This section describes the configuration of the applications responsible for the continuous indexing of files in a NetarchiveSuite arcrepository. In addition, there is a plugin which enables an arcrepository to be accessed by an instance of wayback. This is described in the Additional Tools Manual, along with various batch jobs which may be of use to anyone wishing to index an arcrepository without using the applications described here.
Basic Concepts for the Indexer/Aggregator
There are two applications responsible for indexing an arcrepository. The WaybackIndexerApplication checks a repository for any new files and issues batch jobs to index each new file individually. These unsorted index files are deposited in a local folder. The AggregatorApplication sorts and merges these index files and then merges the result into the existing index files being used by your wayback instance. These applications may be configured and deployed using the NetarchiveSuite Deploy Tool.
WaybackIndexerApplication
This application uses a database to maintain a list of all files in a repository and information as to whether or not they have been archived. It uses a set of worker threads to issue batch jobs to index any new files found. The application behaviour is that any arcfile which contains the string "metadata" in its name is assumed to be a metadata file and is indexed with a tool that searches for deduplication records. Any other file is simple indexed as an arcfile.
The default settings for this application are
<settings>
<wayback>
<hibernate>
<c3p0>
<acquire_increment>1</acquire_increment>
<idle_test_period>100</idle_test_period>
<max_size>100</max_size>
<max_statements>100</max_statements>
<min_size>10</min_size>
<timeout>100</timeout>
</c3p0>
<connection_url>jdbc:derby:derbyDB/wayback_indexer_db;create=true</connection_url>
<db_driver_class>org.apache.derby.jdbc.ClientDriver</db_driver_class>
<use_reflection_optimizer>false</use_reflection_optimizer>
<transaction_factory>org.hibernate.transaction.JDBCTransactionFactory</transaction_factory>
<dialect>org.hibernate.dialect.DerbyDialect</dialect>
<show_sql>true</show_sql>
<format_sql>true</format_sql>
<hbm2ddl_auto>update</hbm2ddl_auto>
<user></user>
<password></password>
</hibernate>
<indexer>
<replicaId>ONE</replicaId>
<final_batch_output_dir>batchOutputDir</final_batch_output_dir>
<temp_batch_output_dir>tempdir</temp_batch_output_dir>
<maxFailedAttempts>3</maxFailedAttempts>
<producerDelay>0</producerDelay>
<producerInterval>86400000</producerInterval>
<consumerThreads>5</consumerThreads>
<initialFiles></initialFiles>
</indexer>
</wayback>
</settings>As can be seen, the application uses a hibernate object-relational mapping layer to communicate with a relational database. Thus it should be possible to plug in any RDBMS simply by changing the hibernate settings. The code has only been tested with DerbyDB and postgresql. The hibernate settings are not described in any more detail here as they are fully documented in the hibernate documentation at http://www.hibernate.org.
The NetarchiveSuite-specific settings are as follows:
dk.netarkivet.wayback.settings.indexer.replicaId: The Id of the replica to be used for indexing. Since indexing is a relatively intensive operation, it is useful to be able to specify which replica is used by the indexer.
dk.netarkivet.wayback.settings.indexer.final_batch_output_dir: The directory where the unsorted index files are stored.
dk.netarkivet.wayback.settings.indexer.temp_batch_output_dir: A directory in which the output from partially finished batch jobs can be written.
dk.netarkivet.wayback.settings.indexer.maxFailedAttempts: The maximum number of failures allowed per file before the indexer permanently gives up attempting to index a given file. At present there is no way, other than manipulating the database, to retry indexing a file once it has reached this limit.
dk.netarkivet.wayback.settings.indexer.producerDelay: The delay in milliseconds after the system start before the indexing process begins.
dk.netarkivet.wayback.settings.indexer.producerInterval: The interval (in milliseconds) between successive reads of the latest filelist from the repository. The value of this parameter is a compromise between updating the index as quickly as possible and overburdening the repository with heavy-duty FileListBatchJobs.
dk.netarkivet.wayback.settings.indexer.consumerThreads: The number of simultaneous indexing threads to be started and hence the maximum number of indexing batch jobs to be run simultaneously.
dk.netarkivet.wayback.settings.indexer.initialFiles: the path to a file containing a list of files in the archive which the indexer should not archive. This can be used when deploying the indexer to a legacy system to ensure that archive files already indexed are not reindexed at unnecessary computational expense.
AggregatorApplication
The aggregator takes all files found in the indexer's output directory, sorts them, and merges them into an existing index file. The unix sort command is used so this application runs only in unix-like systems. The aggregator uses a rollover system to deal with the growth of index files. At any given time, the active index files will consist of a list such as
wayback_intermediate.index wayback.index wayback.index.1 wayback.index.2 wayback.index.3 (etc.)
Whenever the aggregator runs (the interval between aggregator runs is determined by the parameter dk.netarkivet.wayback.settings.aggregator.aggregationInterval}} in milliseconds) the new index files are sorted and merged into {{{wayback_intermediate.index. If this file is now larger than dk.netarkivet.wayback.settings.aggregator.maxIntermediateIndexFileSize (in KB) then this file is merged into wayback.index. If this would cause wayback.index to grow to larger than dk.netarkivet.wayback.settings.aggregator.maxMainIndexFileSize then the filenames for the main index files are rolled over - wayback.index is renamed to wayback.index.1 etc., and a new wayback.index is started. Note that at present the names of these index files are hard-coded.
In addition to the settings described above, the aggregator also uses
dk.netarkivet.wayback.settings.aggregator.indexFileInputDir: the directory where the unsorted index files are to be found, ie the output directory of the indexer
dk.netarkivet.wayback.settings.aggregator.indexFileOutputDir: the directory containing all the final active index files
dk.netarkivet.wayback.settings.aggregator.tempAggregatorDir: a temporary workspace directory. This directory should have storage space at least equal to maxMainIndexFileSize and should ideally be on the same file system as indexFileOutputDir.