• IssuesFromNs97

Review (NS-97): FR 1691 (supersedes NS-95)

Objectives

1017:
[FROM NICOLAS @ BNF]
------------------------------------------------------------------------
FR 1691 Configure which Heritrix reports to include in metadata ARC file
------------------------------------------------------------------------
Three new setting properties have been added:
- settings.harvester.harvesting.metadata.heritrixFilePattern is a java pattern that allows to filter which files in the crawl dir (not recursively) to include in the lmetadata ARC.
- settings.harvester.harvesting.metadata.reportFilePattern is also a java pattern that controls which subset of the files selected by heritrixFilePattern are to be considered as report files. All the other files will be considered as setup files.
- settings.harvester.harvesting.metadata.logFilePattern is a third java pattern that controls which files in the logs subdirectory of the crawldir are to be added as log files to the metadata ARC.
NOTE: FR 1691 also addresses Bug 808 (Should store crawl-manifest.txt)
This bug also suggests storing the conf/heritrix_properties, but why not take all files in conf  and modules.

Total Time Used (Coding,Documentation,Review):

Time use (Coding,Documentation,Review)
Nicolas: 2 MD
SVC:  0.5 MD

General comments:

Comments on file 'trunk/src/dk/netarkivet/harvester/HarvesterSettings.java', revision 1017

Comments on file 'trunk/src/dk/netarkivet/harvester/harvesting/HarvestDocumentation.java', revision 1017

Comments on file 'trunk/src/dk/netarkivet/harvester/harvesting/MetadataFile.java', revision 1019