Describe It38CheckHarvestNotDeduplicated here.

start a web browser by e.g. $ mozilla Note that it is important that the browser is started on the same machine as the RunNetarchiveSuite.sh is run on

setup the browser to proxy on port 8070 and exclude localhost e.g. in mozilla:

Choose in the mozilla toolbar: Edit->Preferences->Advanced->Proxies Checkmark: Manual Proxy Configuration and add: Proxy: localhost Port: 8070 No Proxy for: localhost Write following url in the started browser http://localhost:8074/HarvestDefinition

If you are a netarchive tester:

Forward the access port and set browser to be local forward port

Do the following on kb-prod-udv-001.kb.dk as a test user:

Write the following in a prompt <verbatim> ssh -g -N -L$PORT:kb-test-acs-001.kb.dk:$PORT kb-test-acs-001.kb.dk </verbatim> Note that the shell will hang after this command. The ssh must be killed after the test (by <CTRL>C)

Set browser (Mozilla, Internet Explorer or <nop>FireFox) to be local forward port.

Mozilla: Go to =preferences -> advanced.proxies= and activate 'manual proxy-settings', 'HTTP-proxy:' is set to "kb-prod-udv-001.kb.dk" and the belonging 'Port:' is set to the test port (807?). 'No proxy for' is set to "kb-test-adm-001.kb.dk"

Internet Explorer: Go to =Tools -> Internet options= choose =Connections= and click =LAN settings=. Mark "Use proxy with …" and insert "kb-prod-udv-001.kb.dk" in 'Address' with 'Port:' and set the test port (807?). Click 'Advanced' and insert =kb-test-adm-001.kb.dk= under 'Exception' ('Do not use proxy for …')

<nop>FireFox 2: Go to =General->Settings=, choose =Connection= and click =Settings=. Mark 'Manual proxy configuration:' and insert "kb-prod-udv-001.kb.dk" in 'HTTP Proxy' and set 'Port:' to the test port (807?). Mark ‘Use for all protocols’ and insert text “=kb-test-adm-001.kb.dk=” in 'No proxy for'

<nop>FireFox 3: Go to =Tools->Settings=, choose =Advanced->Network= and click =Settings...=. Mark 'Manual proxy configuration:' and insert "kb-prod-udv-001.kb.dk" in 'HTTP Proxy' and set 'Port:' to the test port (807?). Mark ‘Use this profor all protocols’ and insert text “=kb-test-adm-001.kb.dk, kb-prod-udv-001.kb.dk=” in 'No proxy for'

Verificer, at metadata arc filen ikke rummer en processor-report for deduplicator som i følgende report:

metadata://netarkivet.dk/crawl/reports/processors-report.txt?heritrixVersion=1.14.3&harvestid=2&jobid=6 130.225.27.140 2009091611 3022 text/plain 1846 Processors report - 200909161130

Job being crawled: forsider_plus_1niveau Number of Processors: 16 NOTE: Some processors may not return a report! Processor: org.archive.crawler.fetcher.FetchHTTP

Function: Fetch HTTP URIs CrawlURIs handled: 140 Recovery retries: 0 Processor: org.archive.crawler.extractor.ExtractorHTTP

Function: Extracts URIs from HTTP response headers CrawlURIs handled: 140 Links extracted: 0 Processor: org.archive.crawler.extractor.ExtractorHTML

Function: Link extraction on HTML documents CrawlURIs handled: 51 Links extracted: 443 Processor: org.archive.crawler.extractor.ExtractorCSS

Function: Link extraction on Cascading Style Sheets (.css) CrawlURIs handled: 0 Links extracted: 0 Processor: org.archive.crawler.extractor.ExtractorJS

Function: Link extraction on JavaScript code CrawlURIs handled: 0 Links extracted: 0

Processor: org.archive.crawler.extractor.ExtractorSWF

Function: Link extraction on Shockwave Flash documents (.swf) CrawlURIs handled: 0 Links extracted: 0 Processor: is.hi.bok.digest.DeDuplicator

Function: Abort processing of duplicate records - Lookup by url in use Total handled: 88 Duplicates found: 0 0.0% Bytes total: 6391852 (6.1 MB) Bytes discarded: 0 (0 0.0% New (no hits): 88 Exact hits: 0 Equivalent hits: 0 Timestamp predicts: (Where exact URL existed in the index) Change correctly: 0 Change falsly: 0 Non-change correct:0 Non-change falsly: 0 Missing timpestamp:0 [Host] [total] [duplicates] [bytes] [bytes discarded] [new] [exact] [equiv] [change correct] [change falsly] [non-change co rrect] [non-change falsly] [no timestamp] sejr.kb.dk 88 0 6391852 0 88 0 0 0 0 0 0 0