We use a patched version of the 0.3.0-20061218 beta version of the deduplicator . The patches fixes the following issues in the NetarchiveSuite:

• [https://gforge.statsbiblioteket.dk/tracker/?aid=1062 Bug 1062] Indexserver skips a lot of lines due to threading problem with SimpleDateFormat

• [https://gforge.statsbiblioteket.dk/tracker/?aid=1078 Bug 1078] DeDuplikator index too large

Known bugs in 0.3.0-20061218b:

• [https://gforge.statsbiblioteket.dk/tracker/?aid=1248] Bug 1248 NPE in deduplicator-0.3.0-20061218b.jar

Downloads

[attachment:deduplicator-0.3.0-20061218a.diff Patch against Deduplicator 0.3.0-20061218]

[attachment:Deduplicator-0.3.0-20061218b.diff Patch against Deduplicator 0.3.0-20061218a]

[attachment:Deduplicator-0.3.0-20061218b-src.zip Patched sourcecode Deduplicator 0.3.0-20061218b-src.zip]

[attachment:Deduplicator-0.3.0-20061218a-bin.zip Patched binary Deduplicator 0.3.0-20061218a-bin.zip]

[attachment:Deduplicator-0.3.0-20061218b-bin.zip Patched binary Deduplicator 0.3.0-20061218b-bin.zip]

Note that the deduplicator must be compiled with the same version of heritrix as the NetarchiveSuite uses, or the deduplicator will fail to work during runtime.