Review (NS-87): FR1678: Improved indexing for wayback

Author	Colin
Moderator	Colin
State	Closed

Objectives

See https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1678
and http://netarchive.dk/suite/ImprovedIndexing
The implemented code includes:
A batch job to extract wayback cdx indexes from archive arc files
A batch job to extract wayback cdx indexes from deduplication metadata records
An application to extract wayback cdx indexes from deduplication crawl logs
 + associated helper methods

Summary

follow-up: csr

Total Time Used (Coding,Documentation,Review):

CSR:4 MD
SVC:0.5 MD

General comments:

Description	Classification	Status
On several of the files you need to set the SVN "svn:keywords" property with value=URL Revision Author Date Id	Cosmetic	OK

Comments on file 'trunk/src/dk/netarkivet/wayback/DeduplicateToCDXApplication.java', revision 995

Lines	Description	Classification	Status
28	blank line	NA	NOTOK
50	Missing argument validation	Cosmetic	NOTOK
55	explicitly close stream	Minor	NOTOK
70	[spelling] wyaback => wayback	Cosmetic	NOTOK

Comments on file 'trunk/src/dk/netarkivet/wayback/batch/ExtractWaybackCDXBatchJob.java', revision 995

Lines	Description	Classification	Status
43-44	We don't include that kind of information in the NetarchiveSuite javadoc	Cosmetic	OK
52-53	Missing javadoc	Cosmetic	OK
55	Missing javadoc	Cosmetic	OK
64	missing javadoc	Cosmetic	OK
84	javadoc	Cosmetic	OK

Comments on file 'trunk/src/dk/netarkivet/wayback/batch/ExtractDeduplicateCDXBatchJob.java', revision 995

Lines	Description	Classification	Status
General	Remove underscores in variablenames. This violates our coding style	Cosmetic	OK
30	Unnecessary blank lines in import block	NA	OK
42	Remove unused line	Cosmetic	OK
48	Missing javadoc	Cosmetic	OK
53	missing javadoc	NA	OK
63	missing javadoc	Cosmetic	OK

Comments on file 'trunk/src/dk/netarkivet/wayback/batch/UrlCanonicalizerFactory.java', revision 995

Lines	Description	Classification	Status
31	Missing period in first sentence of javadoc	Cosmetic	OK
47	use try/catch on securityexception	Cosmetic	OK

Comments on file 'trunk/src/dk/netarkivet/wayback/batch/DeduplicateToCDXAdapter.java', revision 995

Lines	Description	Classification	Status
General	Remove underscores in variablenames. This violates our coding style	Cosmetic	OK
General	Divide lines longer than 80 characters into two lines.	Cosmetic	OK
41	Missing class javadoc	NA	OK
48-58	Missing javadoc	Cosmetic	OK
60	Missing javadoc	Cosmetic	OK
62	Missing javadoc	Cosmetic	OK
66	Missing javadoc, and missing validation of argument 'line'	Cosmetic	OK
67	Make a constant for the "duplicate:" string	Cosmetic	OK
108	Missing javadoc and argument validation	Cosmetic	OK

Comments on file 'trunk/src/dk/netarkivet/wayback/batch/DeduplicateToCDXAdapterInterface.java', revision 995

Lines	Description	Classification	Status
30	Missing period in first sentence of javadoc	Cosmetic	OK
37	What type of canonicalization is done on the target url	Cosmetic	OK
46	Replace "dedup lines" with "lines containing deduplication information" or similar	Cosmetic	TOK
47	Missing period in first sentence of javadoc	Cosmetic	OK

Comments on file 'trunk/src/dk/netarkivet/wayback/WaybackSettings.java', revision 995

Lines	Description	Classification	Status
General	Missing svn svn:keywords property with value=URL Revision Author Date Id	Cosmetic	OK
1	File headers/copyright missing	Cosmetic	OK
25	Missing javadoc	Cosmetic	OK