Check Global Crawler traps

Choose 'Definitions' -> 'Global Crawler Traps' and click 'Edit'

Type a name e.g. crawlertraps1

Save the file http://kb-prod-udv-001.kb.dk/cvsweb/cvsweb.cgi/~checkout~/projects/webarkivering/documents/internal/crawlertrapsCollection.txt?rev=1.1;content-type=text%2Fplain on your Desktop as crawlertraps.txt and upload the file.

Check that all crawler traps are uploaded.

The number of crawlertraps uploaded may vary, as duplicates are removed during the upload to the database.

You can verify that the only difference between the uploaded and downloaded list is duplicates in the former list:

download the uploaded list to your local harddisk as crawlertraps.downloaded.txt
sort -u crawlertraps.downloaded.txt > crawlertraps.downloaded.txt.sorted
sort -u crawlertraps.txt > crawlertraps.txt.sorted

diff crawlertraps.downloaded.txt.sorted crawlertraps.txt.sorted
The output should be empty.