== Estimate time for Improving deduplication: 14 MD ==
== Degree of uncertainty: B ==
== Responsible: SVC ==

== Improving deduplication ==

=== Introduction ===

Generating deduplicating indices is currently error-prone and time-consuming.
It is however my belief, that the OOM bugs in the batch monitoring software may limit the
errors until now seen while generating indices.

Task 1: Try to retrieve data from other bitarchive replica if the local bitarchive replica is unresponsive

Refer to FR XXXXX.

Task 2: Optimize the fetching process.

To generate the deduplication indices, it needs to fetch a lot of metadata-records, specifically the records containing the crawl-logs, and cdx-entries.

Currently, our software for each job-id included in the dedup index, launches two batch-jobs, one for the cdxes, and one for the crawllog. This could maybe be reduced to one job, that maybe also does some of the processing locally on the storage-machines.

Task 3: Optimize the indexing process.

Currently, part of indexing process is a merging of the information found in the CDX-file with the information found in the crawllog.
Investigate if this merging of information could be done as part of the post-processing, or as a pre-processing step, if this information is absent in the metadata-records.