• AssignmentHarvester1

Harvester assigment 1: Upgrade to Heritrix release 3

This assignment contains the subtasks needed to upgrade NetarchiveSuite to use Heritrix 3.0.0+ Note that Internet Archive, the maintainer of Heritrix more or less deprecates the use of Heritrix 2.0+ Therefore we will bypass the 2.0 branch of Heritrix.

Tasks

Task A

Download binary copy of Heritrix, and update the contents of lib/heritrix, lib/heritrix-dependencies, and make the appropiate changes to our build.xml

The deduplication-X-Y-Z.jar (if we're still using this module for deduplication) needs to be recompiled with the new Heritrix.

Task B

All heritrix templates bundled with NetarchiveSuite needs to be migrated to the new format.

This task includes the writing of extensive documentation on how to migrate.

Beware that already generated jobs that are in one of the states (submitted, started or failed) will not be resubmittable. So it is a good idea to minimise this number, for instance by disabling all active harvestdefinitions.